Tornado, the non-blocking web server and tools open sourced by FriendFeed before their acquisition, seems to get some NoSQL activity. While Django is leading the way in the Python world, judging by the NoSQL projects happening around Node.js, one could say that Tornado, with its non-blocking architecture, may be an interesting alternative.

Thomas Pelletier has ☞ a blog post about a simple websocket + Tornado + Redis Pub/Sub protocol integration:

The principle is very simple: when your user loads the page, she is automatically added to a list of “listeners”. An independent thread is running: it listens for messages from Redis with the subscribe command, and send a message through Websocket to every registered ”listener”. In this example, the user can send a message to herself with a simple AJAX-powered form, which calls a view with a payload (the message), and the view publish it via the publish command of Redis.

This is basically a web chat! If you want to have fun, you can then add a roster, with a presence system, authentication etc…

There’s also a ☞ GitHub project called Trombi:

Trombi is an asynchronous CouchDB client for Tornado.

And I’m pretty sure there are other projects I’ve missed (but you can leave a comment to add them to the list).

Original title and link for this post: Tornado Sees Some NoSQL Activity (published on the NoSQL blog: myNoSQL)

I’m not really sure how I’ve missed the release of Riak 0.12.0 last week. The ☞ release notes are listing over 30 enhancements and bug fixes. Another important update coming with Riak 0.12.0 is the improved failure recovery:

Riak now uses a new and improved mechanism for determining whether a node is fully online and ready to participate in Riak operations. This is especially important in failure recovery situations, as it allows the storage backend to complete a full integrity check and repair process.

Riak is getting closer to celebrate 1 year and there were more than 12 releases during this year. That’s a nice pace for a project!

Meanwhile, Riak has got a library in Go named ☞ Goriak whose API is described ☞ here, but also a Javascript library for node.js^[1]: ☞ riak-js.

The main point behind Grant Goodale’s presentation is that MongoDB and Node.js are great together:

• both understand Javascript and JSON
• both are fast

Alex Payne (ex-Twitter, BankSimple) has a very interesting post about Node.js:

If you look at who’s flocking to Node, it’s largely web developers who have been working in dynamic languages with what we could politely call limited performance characteristics. Adding Node to their architectures means that these developers have gone from having essentially no concurrency story and very constrained runtime performance to having some semi-sane concurrency story – one rigidly enforced by the Node framework – running on a virtual machine with comparatively respectable performance. They slice off a painful bit of their application that’s suited to asynchrony, rewrite it in Node, and move on.

That’s awesome. That kind of outcome definitely meets Node’s secondary stated goal of “less-than-expert programmers” being “able to develop fast systems”. However, it has very little to do with scaling in the larger, more widely-understood sense of the term.

— al3x.net

nStore:

A simple in-process key/value document store for node.js. nStore uses a safe append-only data format for quick inserts, updates, and deletes. Also a index of all documents and their exact location on the disk is stored in in memory for fast reads of any document. This append-only file format means that you can do online backups of the datastore using simple tools like rsync. The file is always in a consistent state.

nStore can be found on ☞ GitHub

☞ A simple notice about a short scheduled GitHub downtime resulted in a great dialog about how to perform zero downtime Redis upgrades. The following approaches were suggested:

1. using Redis replication (i.e. SLAVE OF) to move existing data from the old Redis to the new Redis and haproxy to transparently pass client requests to the Redis instance (initially to the existing version and once replication completed to the new one)

   1. Leave the old redis version running on, say, redis:6379.
   2. Install and start a new redis on redis:6380 with a different dump file location.
   3. Execute SLAVE OF redis 6379 against redis:6380. Wait for first SYNC to complete.
   4. echo "enable server redis/redis-6380" | socat stdio unix-connect:/var/run/haproxy/admin.sock
   5. echo "disable server redis/redis-6379" | socat stdio unix-connect:/var/run/haproxy/admin.sock
   6. Execute SLAVE OF no one on redis:6380.
   7. Execute SHUTDOWN on redis:6379.

2. using virtual IPs instead of haproxy

A more generic idea shared in the comment thread was to architect the solution so that the overall system would continue to work even if some subsystems are temporarily down. If you think in terms of the CAP theorem, this idea could be translated as a system that is AP at all times.

While writing this post I remembered reading about a ☞ solution based on node.js. node.js would be used to proxy requests until it is notified that the back system goes down. It would temporarily hold incoming connections until it would be notified again that the back system is resuming.

Any other ideas?

While most of Francisco Treacy’s (@frank06) “An Introduction to node.js and Riak” presentation is focusing on the advantages of event-based architectures, it also shows how to integrate node.js and Riak using ☞ riak-js, a node.js library for Riak that takes advantage of the friendly HTTP-based Riak protocol

There are a couple of other interesting things that can be learned from this slide deck. For example the cost of I/O:

simply described afterwards:

In other words, reaching RAM is like going from here to the Red Light District. Accessing the network is like going to the moon.

Update: thanks to a comment on this post, here is what Googler Jeff Dean presented on the cost of I/O:

But as Frank mentions, there are some risks while working with cutting-edge technologies:

• Cutting-edge technologies are not bug-free
• Riak still has some rough edges (some in terms of performance)
• node.js is approaching its first stable version
• asynchronous JS code can get “boomerang-shaped”

Last week, I have mentioned a post by Gleicon Moraes on ☞ building an URL shortener using MongoDB and alternatively Redis.

There is also a ☞ GitHub project by Sean Cribbs implementing the same scenario using Riak and Ruby-based Sinatra.

Update: Jan Lehnardt was quick to point me to a CouchDB-based URL shortner on ☞ GitHub.

Update 2: Mathias Meyer shared with us ☞ Relink: a solution built on top of Redis with Sinatra

Update 3: Aaron pointed out ☞ little, another solution using Redis and Node.js

Update 4: Frank Denis has just pushed live ☞ the code of another URL shortener ☞ http://sk.tl, built using Tokyo Cabinet and Sinatra. Thanks Frank!

I’m pretty sure there are many more such projects so please post a link to the project in the comment section and I’ll update the post.

1. Chris Storm: Getting started with node.js and CouchDB ☞ part 1 and ☞ part 2. You cannot accuse him for having fun! ¶
2. blog.isabel-drost.de: ☞ Bob Schulze on Tips and patterns with HBase ¶
3. Dennis Forbes: ☞ Fighting The NoSQL Mindset, Though This Isn’t an anti-NoSQL Piece. ¶

   I am still not sure the post deserved linking, but it generated too much noise around. Personally I’m in complete agreement with ☞ @coda:

   My tests, with data sets 2-3 orders of magnitude smaller than yours, zero concurrent writers, and a single reader, indicate you suck.

We’ve been covering tons of Redis usecases, not to mention this amazing list of ideas. Lately, it looks like there is a new emerging usecase that Redis can be proud of: queues.

Now if that already sounds interesting then I guess you could just take a look at QR, a Python ☞ GitHub hosted project that makes it easy to create queues, stacks and deques on top of Redis. For some help on using it you could check Ted Nyman’s posts on ☞ queues and ☞ deques and stacks. Another option would be to head to Resque, a Ruby ☞ GitHub hosted library for creating and processing jobs using Redis queues.

Anyway, if you don’t have yet an idea on how this can be useful, then I hope these following posts will wet your appetite. David Czarnecki’s ☞ article covers a very simple Redis-based queue scenario: inter-application communication (basically the two apps will get an easy way to pass from one to another any kind of messages). If this is still not enough, then Paull Gross’s ☞ post is introducing you to a web proxy built using node.js and Redis queues for high availability.

Last, but not least, I should emphasize the fact that what sets aside Redis as a good tool for this sort of things is not the fact that Redis is a extremely fast, persistent key-value store, but rather Redis native support for ☞ data structures like lists, sets and ordered sets and a set of specific ☞ commands to deal with these.