A very detailed article about creating a message queue (nb: not Pub/Sub) using Redis:

One particular combination has recently allowed us to implement a moderately (read: good enough) reliable message delivery mechanism for multiple consumers consuming messages at their own pace.

Redis allowed for all of the implementation characteristics we were looking for, namely:

• Concurrency: Because all operations in Redis are atomic, supporting concurrency without too much of a hassle is straightforward.
• Persistence: Configured properly, we can ensure persistence of our queues to disk using one of the supported Redis persistence strategies.
• Lightweight: Using Redis from any language/platform is extremely simple and provisioning it / maintaining it on a production server is dead easy.

I’m always saying to use the right tool for the job, but maintaining the simplicity of a system architecture is an art too.

Original title and link: Reliable Delivery Message Queues With Redis (NoSQL database©myNoSQL)

via: http://davidmarquis.posterous.com/reliable-delivery-message-queues-with-redis

DataSift moves from Redis PubSub to Kafka and 0mq :

Kafka is still a young project, but it’s maturing fast, and we’re confident enough to use it in production (as a matter of fact, we’ve been using it for months now) in front of our HBase cluster and to collect monitoring events sent from all our internal services. We chose Kafka especially for its persistent storage (which is essentially a partitioned binary log), but we plan to do some analytics via its support for Hadoop soon. And its distributed nature (coordination beetween consumers and brokers is done via Zookeeper) makes it very appealing too.

Original title and link: DataSift PubSub: From Redis to Kafka and 0mq (NoSQL database©myNoSQL)

via: http://www.alberton.info/kafka_apache_incubator.html

I’m hearing quite often lately of Redis PUB/SUB replacing real queuing systems. Here is an example application:

There’s also a screencast and the code available on GitHub:

Original title and link: Building Publish / Subscribe Apps with Tropo and Redis (NoSQL databases © myNoSQL)

via: http://blog.tropo.com/2011/04/04/building-publish-subscribe-apps-with-tropo-and-redis/

Another open source project from LinkedIn:

Kafka is a distributed publish-subscribe messaging system. It is designed to support the following:

• Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
• High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
• Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
• Support for parallel data load into Hadoop.

LinkedIn has open sourced a couple of exciting projects, but they haven’t been able to get enough attention and grow so far a community around these.

Original title and link: Kafka: LinkedIn’s Distributed Publish/Subscribe Messaging System (NoSQL databases © myNoSQL)

via: http://sna-projects.com/kafka/

Debasish Ghosh integrates Akka, a Scala implementation of the ☞ actor model, with Redis PUB/SUB support:

Redis (the version on the trunk) offers publish/subscribe based messaging. This is quite a big feature compared to the typical data structure oriented services that it had been offering so far. This also opens up lots of possibilities to use Redis as a messaging engine of a different kind.

via: http://debasishg.blogspot.com/2010/04/pubsub-with-redis-and-akka-actors.html