How to implement a message queue over Redis?

Why Redis for queuing?

I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or RabbitMQ. With RabbitMQ we've had many problems - the client libraries are very poor and buggy and we'd like not to invest too many developer-hours into fixing them, a few problems with the server management console, etc. And, for the time being at least, we're not grasping for milliseconds or seriously pushing performance, so as long as a system has an architecture that supports a queue intelligently we are probably in good shape.

Okay, so that's the background. Essentially I have a very classic, simple queue model - several producers producing work and several consumers consuming work, and both producers and consumers need to be able to scale intelligently. It turns out a naive PUBSUB doesn't work, since I don't want all subscribers to consume work, I just want one subscriber to receive the work. At first pass, it looks to me like BRPOPLPUSH is an intelligent design.

Can we use BRPOPLPUSH?

The basic design with BRPOPLPUSH is you have one work queue and a progress queue. When a consumer receives work it atomically pushes the item into the progress queue, and when it completes the work it LREM's it. This prevents blackholing of work if clients die and makes monitoring pretty effortless - for instance we can tell if there is a problem causing consumers to take a long time to perform tasks, in addition to telling if there is a large volume of tasks.

It ensures

• work is delivered to exactly one consumer
• work winds up in a progress queue, so it can't blackhole if a consumer

The drawbacks

• It seems rather strange to me that the best design I've found doesn't actually use PUBSUB since this seems to be what most blog posts about queuing over Redis focus on. So I feel like I'm missing something obvious. The only way I see to use PUBSUB without consuming tasks twice is to simply push a notification that work has arrived, which consumers can then non-blocking-ly RPOPLPUSH.
• It's impossible to request more than one work item at a time, which seems to be a performance problem. Not a huge one for our situation, but it rather obviously says this operation was not designed for high throughput or this situation
• In short: am I missing anything stupid?

Also adding node.js tag, because that's the language I'm mostly dealing with. Node may offer some simplifications in implementing, given its single-threaded and nonblocking nature, but furthermore I'm using the node-redis library and solutions should or can be sensitive to its strengths and weaknesses as well.

Asked by user53263 (1 )
Jul 12, 2013 at 14:29