NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter



pubsub: All content tagged as pubsub in NoSQL databases and polyglot persistence

Reliable Delivery Message Queues With Redis

A very detailed article about creating a message queue (nb: not Pub/Sub) using Redis:

One particular combination has recently allowed us to implement a moderately (read: good enough) reliable message delivery mechanism for multiple consumers consuming messages at their own pace.

Redis allowed for all of the implementation characteristics we were looking for, namely:

  • Concurrency: Because all operations in Redis are atomic, supporting concurrency without too much of a hassle is straightforward.
  • Persistence: Configured properly, we can ensure persistence of our queues to disk using one of the supported Redis persistence strategies.
  • Lightweight: Using Redis from any language/platform is extremely simple and provisioning it / maintaining it on a production server is dead easy.

I’m always saying to use the right tool for the job, but maintaining the simplicity of a system architecture is an art too.

Original title and link: Reliable Delivery Message Queues With Redis (NoSQL database©myNoSQL)


DataSift PubSub: From Redis to Kafka and 0mq

DataSift moves from Redis PubSub to Kafka and 0mq :

Kafka is still a young project, but it’s maturing fast, and we’re confident enough to use it in production (as a matter of fact, we’ve been using it for months now) in front of our HBase cluster and to collect monitoring events sent from all our internal services. We chose Kafka especially for its persistent storage (which is essentially a partitioned binary log), but we plan to do some analytics via its support for Hadoop soon. And its distributed nature (coordination beetween consumers and brokers is done via Zookeeper) makes it very appealing too.

Original title and link: DataSift PubSub: From Redis to Kafka and 0mq (NoSQL database©myNoSQL)


Building Publish / Subscribe Apps with Tropo and Redis

I’m hearing quite often lately of Redis PUB/SUB replacing real queuing systems. Here is an example application:

Building Publish / Subscribe Apps with Tropo and Redis

There’s also a screencast and the code available on GitHub:

Original title and link: Building Publish / Subscribe Apps with Tropo and Redis (NoSQL databases © myNoSQL)


Kafka: LinkedIn's Distributed Publish/Subscribe Messaging System

Another open source project from LinkedIn:

Kafka is a distributed publish-subscribe messaging system. It is designed to support the following:

  • Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
  • High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
  • Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
  • Support for parallel data load into Hadoop.

LinkedIn has open sourced a couple of exciting projects, but they haven’t been able to get enough attention and grow so far a community around these.

Original title and link: Kafka: LinkedIn’s Distributed Publish/Subscribe Messaging System (NoSQL databases © myNoSQL)


Redis, Scala and PubSub

Debasish Ghosh integrates Akka, a Scala implementation of the ☞ actor model, with Redis PUB/SUB support:

Redis (the version on the trunk) offers publish/subscribe based messaging. This is quite a big feature compared to the typical data structure oriented services that it had been offering so far. This also opens up lots of possibilities to use Redis as a messaging engine of a different kind.