Anything I’d write would just steal from your time to read and think about the email Joseph Blomstedt posted to the Riak list.

Original title and link: Eventual and Strong Consistency, Sloppy and Strict Quorums, and Other Lessons and Thoughts on Distributed Systems (NoSQL database©myNoSQL)

Last night, Mark Phillips^[1] sent me the following message:

Yesterday a Riak community member, Joseph Blumstedt released a pair of repos: riak_zab and riak_zab_example. In short, riak_zab is an extension for riak_core that provides totally ordered atomic broadcast capabilities.

If your first reaction was something along the line “so what?”, then you are not alone. Anyway I hope I’ve done my homework and now I understand why this is both interesting and important.

ZooKeeper is a coordination service created by Yahoo! to allow large scale applications to perform coordination tasks such as leader election, status propagation, and rendezvous. Embedded into ZooKeeper is a totally ordered broadcast protocol: Zab^[2].

ZooKeeper makes the following requirements on the Zab broadcast protocol:

Reliable delivery

If a message, m, is delivered by one server, then it will be eventually delivered by all correct servers.

Total order

If message a is delivered before message b by one server, then every server that delivers a and b delivers a before b.

Causal order

If message a causally precedes message b and both messages are delivered, then a must be ordered before b.

Prefix property:

If m is the last message delivered for a leader L, any message proposed before m by L must also be delivered

Putting it together:

riak_zab is an extension for riak_core that provides totally ordered atomic broadcast capabilities. This is accomplished through a pure Erlang implementation of Zab, the Zookeeper Atomic Broadcast protocol invented by Yahoo! Research.

Basically this means that many operations that were typically hard to do well using the Dynamo eventual consistency model which Riak implements with riak_core are now theoretically possible. In case you’d like to read more about riak_core I’d suggest these posts Building blocks of Dynamo-like distributed systems, riak_core: building distributed applications without shared state, and Where to start with riak_core .

Indeed there are other Dynamo aspects like synchronization, scaling, ring rebalancing, handoff that riak_zab had to address and its approach is documented on the project page

Technically, riak_zab isn’t focused on message delivery. Rather, riak_zab provides consistent replicated state.

So why is this important? While parts of an application can deal with eventual consistency, there are typically a few aspects of an app where stronger consistency is needed for various reasons. A few such examples:

• Distributed Counters — just ask Digg people how they implemented distributed counters in Cassandra or check how Twitter built a ZooKeeper and Cassandra based solution for realtime analytics .
• Ordered Messaging
• Compare-and-Swap Operations
• Set Operations
• Sub-document updates
• Multi-key transactions

riak_zab will theoretically allow users to reap the availability of the standard eventually consistent model Riak provides and opt in to stronger consistency for the parts of the app that require it. In other words, it expands the number of CAP trade offs that a developer can make beyond the R, W and N model and makes possible functionality that relies on consistency guarantees.

After connecting all these dots, I’ve finally got it (nb: thanks Mark for sending this out!). And it definitely sounds like an exciting addition to riak_core. The caveat here is that even if this code has been tested in an academic environment the project is still an alpha version. Hopefully other members of the Riak community will pick it up and together with Basho guys will make it production ready.

Original title and link: riak_zab: ZooKeeper’s Zab Protocol for Riak (NoSQL databases © myNoSQL)

Basho guys have started to talk about their experience on building Riak, the Dynamo-like distributed key-value store and the common building blocks of distributed systems.

Justin Sheehy interviewed by Sadek Drobi over ☞ InfoQ.com:

Even just the Dynamo specific parts are very dramatic in differences. There have been a number of Dynamo-like systems developed over the past few years, each of which has had to design and implement large portions of even just the Dynamo-like sections on their own. Because Dynamo tells you what some very good design decisions are but it doesn’t show you how to implement the system. Even just the Dynamo portion you have to do a lot of design work, just to implement that.

Justin on choosing Erlang for implementing Riak:

There was a really natural choice because especially when you look at the Dynamo model, where they talk about all these operations where to get a value you’ll send messages to multiple other parties, then you’ll wait through various phases for responses of different classes to come back and the basic building blocks to do that kind of messaging and to do that kind of more complex state machine are there for you out of the box for you in Erlang.

Kevin Smith promises a series of posts covering the details of ☞ riak_core, the refactored core of the Riak system that can be used for building Dynamo-like distributed systems:

Distributed systems are complex and some of that complexity shows in the amount of features available in riak_core. Rather than dive deeply into code, I’m going to separate the features into broad categories and give an overview of each.

The ☞ first part covers aspects like:

• node liveness & membership (note it is interesting to note that improvements to the failure recovery mechanism is part of the latest Riak release
• partitioning & distributing work
• cluster state

Definitely a series I’ll keep an eye on as I’m pretty sure there are many things to be learned from their experience. (shameless plug) If you happen to be in the Bay area in November, come check the NoSQL track at QCon where, even if not yet published yet, among others, Andy Gross, VP of engineering at Basho, will be speaking about how to build Dynamo style systems using Riak’s core.

Building Blocks of Dynamo-like Distributed Systems originally posted on the NoSQL blog: myNoSQL

This is becoming a habit of the NOSQL summer meeting in ☞ Tokyo:

They did it before for the Google BigTable paper. If you speak Japanese and happen to be around in Tokyo I’d say you shouldn’t miss such an event.

Note: In case you missed it, you can get all NoSQL papers using this little “hack”

Funny video from Basho guys (physically) demonstrating what’s happening with your data when a server crashes

Next time, send the mugs to people and use real servers for such demos!

This is what I expect to become a great article series on the Amazon Dynamo paper. The author, Will Larson, suggests a different path of understanding the inner workings of this system:

I decided that a good way to record the ideas (as well as solidify them in my mind) was to go through the process of writing a distributed key-value store, and then incrementally add the enhancements discussed in the Dynamo paper. By the end of this series we’ll have re-implemented most of the interesting ideas from Dynamo in a distributed Erlang system.

He sounds really excited to have to deal with the concepts introduced in the Dynamo paper: consistent hashing, merkle trees, vector clocks, gossip protocols, sloppy quorums) and so far has published the first part: Hands On Review of the Dynamo Paper and the 2nd Durable Writes & Consistent Reads.