Kristina Chodorow¹ shared a good list of tests to put a MongoDB cluster through:

Here are some exercises to battle-test your MongoDB instance before going into production. You’ll need a Database Master (aka DM) to make bad things happen to your MongoDB install and one or more players to try to fix it.

Netflix is using a series of tools that perform similar tests against their Cassandra clusters. With a small twist: they are run against the production clusters.

Original title and link: Battle-Test Your MongoDB Cluster (NoSQL database©myNoSQL)

via: http://architects.dzone.com/articles/databases-and-dragons-battle

Great intro to ZooKeeper and the problems it can help solve by Sean Mackrory:

Done wrong, distributed software can be the most difficult to debug. Done right, however, it can allow you to process more data, more reliably and in less time. Using Apache ZooKeeper allows you to confidently reason about the state of your data, and coordinate your cluster the right way. You’ve seen how easy it is to get a ZooKeeper server up and running. (In fact, if you’re a CDH user, you may already have an ensemble running!) Think about how ZooKeeper could help you build more robust systems

Leaving aside for a second the main topic of the post, another important lesson here is that the NIH syndrom in distributed systems is very expensive.

Original title and link: Using Apache ZooKeeper to Build Distributed Apps (And Why) (NoSQL database©myNoSQL)

via: http://blog.cloudera.com/blog/2013/02/how-to-use-apache-zookeeper-to-build-distributed-apps-and-why/

Learning from Peter Bailis:

While the CAP Theorem is fairly well understood, the relationship between CAP and ACID transactions is not. If we consider the current lack of highly available systems providing arbitrary multi-object operations with ACID-like semantics, it appears that CAP and transactions are incompatible. This is partly due to the historical design of distributed database systems, which typically chose consistency over high availability. Standard database techniques like two-phase locking and multi-version concurrency control do not typically perform well in the event of partial failure, and the master-based (i.e., master-per-shard) and overlapping quorum-based techniques often adopted by many distributed database designs are similarly unavailable if users are partitioned from the anointed primary copies.

There’s also a paper (PDF) authored by Peter Bailis, Alan Fekete, Ali Ghodsi, Joseph m. Hellerstein, Ion Stoica. These names should tell you something.

Original title and link: Introducing Highly Available Transactions: The Relationship Between CAP and ACID Transactions (NoSQL database©myNoSQL)

via: http://www.bailis.org/blog/hat-not-cap-introducing-highly-available-transactions/

Emin Gün Sirer:

So, MongoDB is broken, and not just superficially so; it’s broken by design. If you’re relying on its fault tolerance,a you’re probably using it wrong.

He’s not the first to write about some of the critical issues in MongoDB. Every once in a while, there has been at least one post detailing one of these. But none of them stopped MongoDB’s adoption. Why? My only explanations are that:

1. these posts are not reaching enough people;
2. MongoDB is used mostly for simple scenarios where the occurance of such errors can be ignored
3. those using MongoDB for more complicated scenarios have developed internal, application specific workarounds

Original title and link: MongoDB Fault Tolerance - Broken by Design (NoSQL database©myNoSQL)

via: http://hackingdistributed.com/2013/01/29/mongo-ft/

Jeff Hodges:

Below is a list of some lessons I’ve learned as a distributed systems engineer that are worth being told to a new engineer. Some are subtle, and some are surprising, but none are controversial. This list is for the new distributed systems engineer to guide their thinking about the field they are taking on. It’s not comprehensive, but it’s a good beginning.

Go through this list. Twice. Then for every item ask “why is this” at least 3 times. It won’t make you a better system engineer, but you’ll understand better what people developing or operating large scale systems are facing.

Original title and link: Notes on Distributed Systems for Young Bloods (NoSQL database©myNoSQL)

via: http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/

While in conference mode¹ I’m like a sponge, I’m almost no good at putting all my chaotic notes in a format that is usable to anyone else.

Alex Siculars has done a great job writing down his thoughts about Basho’s fantastic Ricon 2012 and linking to his post makes me feel less guilty for not being able to post mines—I’m learning to get better for the next events:

Chatter by conference attendees left me convinced that Ricon was a success. Ricon was-well executed, well-attended and actually interesting. But more importantly, it was relevant. For those of us at the conference, we actually work in this space. We are interested in the ongoing development of distributed solutions to a number of problems. The conference delivered on creating a space that brought us together to share solutions and learn about continuing advancements. For a new conference to have a successful maiden voyage is no small feat in my book. I, for one, am looking forward to the next one.

My only contribution to Alex Sicular’s great recap is to provide some links to the talks his blog post refers to:

Joe Hellerstein: Programming Principles for a Distributed Era

The PDF can be downloaded from here

Eric Brewer: Advancing Distributed Systems

Russel Brown and Sean Cribbs: Data Structures in Riak

Bryan Fink: Riak Pipe: Distributed Processing System

Ryan Zezeski: Yokozuna: Riak + Solr

More presentation slides can be found on the official Ricon 2012 site.

Original title and link: Alex Sicular’s Recap of Ricon 2012, a Distributed Systems Conference for Developers (NoSQL database©myNoSQL)

via: http://siculars.posterous.com/ricon-2012-recap

Great post by Peter Bailis looking at how some systems are reducing tail latency by distributing reads across nodes:

Open-source Dynamo-style stores have different answers. Apache Cassandra originally sent reads to all replicas, but CASSANDRA-930 and CASSANDRA-982 changed this: one commenter argued that “in IO overloaded situations” it was better to send read requests only to the minimum number of replicas. By default, Cassandra now sends reads to the minimum number of replicas 90% of the time and to all replicas 10% of the time, primarily for consistency purposes. (Surprisingly, the relevant JIRA issues don’t even mention the latency impact.) LinkedIn’s Voldemort also uses a send-to-minimum strategy (and has evidently done so since it was open-sourced). In contrast, Basho Riak chooses the “true” Dynamo-style send-to-all read policy.

Original title and link: Doing Redundant Work to Speed Up Distributed Queries (NoSQL database©myNoSQL)

via: http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/

The Amazon post about the service disruption that happened late last month provides an interesting description of the behavior of the Amazon EC2 and EBS metadata datastores:

The EC2 and EBS APIs are implemented on multi-Availability Zone replicated datastores. These datastores are used to store metadata for resources such as instances, volumes, and snapshots. To protect against datastore corruption, currently when the primary copy loses power, the system automatically flips to a read-only mode in the other Availability Zones until power is restored to the affected Availability Zone or until we determine it is safe to promote another copy to primary.

Original title and link: The Behavior of EC2/EBS Metadata Replicated Datastore (NoSQL database©myNoSQL)

via: http://aws.amazon.com/message/67457/