NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter



Pub/Sub: All content tagged as Pub/Sub in NoSQL databases and polyglot persistence

MongoDB Pub/Sub With Capped Collections

Rick Copeland designs a MongoDB Pub/Sub system based on:

  • MongoDB’s capped collections,
  • tailable data-awaiting cursors,
  • sequences (using find_and_modify()),
  • a “poorly documented option” of capped collections: oplog_replay1.

If you’ve been following this blog for any length of time, you know that my NoSQL database of choice is MongoDB. One thing that MongoDB isn’t known for, however, is building a publish / subscribe system. Redis, on the other hand, is known for having a high-bandwith, low-latency pub/sub protocol. One thing I’ve always wondered is whether I can build a similar system atop MongoDB’s capped collections, and if so, what the performance would be. Read on to find out how it turned out…

The solution is definitely ingenious and it could probably work for systems with not so many requirements for their pub/sub. It’s definitely a good excercise in combining some interesting features of MongoDB (I like the capped collections and the tailable data-awaiting cursors).

✚ I’m wondering if the behavior of the tailable data-awaiting cursors is the one of the non-blocking polls.

  1. I don’t really understand how this works. 

Original title and link: MongoDB Pub/Sub With Capped Collections (NoSQL database©myNoSQL)


Visualizing Wikipedia Update Stream With Redis Pub/Sub and Node.js

Fascinating story about the volume of edits happening on Wikipedia and how using Redis Pub/Sub and node.js helped building some visualization tools—wikistream and wikipulse—for the activity stream:

For both of these apps I needed to log into the wikimedia IRC server, listen on ~30 different channels, push all the updates through some code that helped visualize the data in some way, and then get this data out to the browser. I had heard good things about node for high concurrency network programming from several people. I ran across a node library called that reported to make it easy to stream updates from the server to the client, in a browser independent way, using a variety of transport protocols. Instinctively it felt like the pub/sub model would also be handy for connecting up the IRC updates with the webapp. I had been wanting to play around with the pub/sub features in redis for some time, and since there is a nice redis library for node I decided to give it a try.

Original title and link: Visualizing Wikipedia Update Stream With Redis Pub/Sub and Node.js (NoSQL database©myNoSQL)


Watch System Logs in Real Time with Redis Pub/Sub

Little trick to get access to your logs:

This is a little python log handler that lets you stream your log data over a redis pub/sub channel, so you can monitor your system in real time from any redis client.[…] I am also interested in mapping the python logging names ( to redis channels that would enable you to listen to arbitrary applications, or subscribe to a collection using the redis wildcard subscriptions.

On GitHub.

Original title and link: Watch System Logs in Real Time with Redis Pub/Sub (NoSQL databases © myNoSQL)


Redis: A technique for Verifying Propagation of Writes

Interesting idea on how to emulate in Redis MongoDB’s db.runCommand( { getlasterror : 1 , w : 2 } )[1] to verify propagation of writes:

Have client establish two connections - one to master Redis and one to slave Redis.

Now when client wants to write KEY to both master and slave:

  1. Client SUBSCRIBEs to  KEY on slave
  2. Client writes KEY (e.g. set foo bar) to master
  3. Client publishes KEY to master

Assuming redis-master will serialize the updates:

  1. Master will replicate KEY to slave
  2. Master will propagate PUBLISH KEY to slave

Finally client:

  1. Gets KEY event on SUBSCRIBE!

You can probably replace Pub and Subs with RPUSH and BLPOP respectively.

Couple of comments:

  1. using PUB/SUB will require an async client that will need to perform coordination to conclude the propagation was successful
  2. using RPUSH and BLPOP would allow a sync client, that can also set up timeouts

Original title and link: Redis: A technique for Verifying Propagation of Writes (NoSQL databases © myNoSQL)