NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter



ACID: All content tagged as ACID in NoSQL databases and polyglot persistence

Enterprise-class NoSQL

What is distinctive about an enterprise-class NoSQL database is its support for additional enterprise-scale application requirements, namely: ACID (atomic, consistent, isolated, and durable) transactions, government-grade security and elasticity, as well as automatic failover.

What is distinctive about an enterprise-class NoSQL database is what my company is selling.

If that would be true, I doubt we would have no any other databases around considering MarkLogic’ age and perfect fit.

Snarky comments aside, the enterprise requirements are so complicated, numerous, political and sometime non-technical, that I don’t think anyone would ever be able to come up with a definition or (even if extremely long) checklist of what’s enterprise-grade.

Original title and link: Enterprise-class NoSQL (NoSQL database©myNoSQL)


NoSQL meets Bitcoin and brings down two exchanges

Most of Emin Gün Sirer’s posts end up linked here, as I usually enjoy the way he combines a real-life story with something technical, all that ending with a pitch for HyperDex.

The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB. And the situation would not have been any different if we had used Cassandra or Riak. All of these first-generation NoSQL datastores were early because they are easy to build. When the datastore does not provide any tangible guarantees besides “best effort,” building it is simple. Any masters student in a top school can build an eventually consistent datastore over a weekend, and students in our courses at Cornell routinely do. What they don’t do is go from door to door in the valley, peddling the resulting code as if it could or should be deployed.

Unfortunately in this case, the jump from the real problem, which was caused only by the pure incompetence, to declaring “first-generation NoSQL databases” as being bad and pitching HyperDex’s features is both too quick and incorrect1.

  1. 1) ACID guarantees wouldn’t have solved the issue; 2) All 3 NoSQL databases mentioned, actually offer a solution for this particular scenario. 

Original title and link: NoSQL meets Bitcoin and brings down two exchanges (NoSQL database©myNoSQL)


Scalable Atomic Visibility with RAMP Transactions

We’ve developed three new algorithms—called Read Atomic Multi-Partition (RAMP) Transactions—for ensuring atomic visibility in partitioned (sharded) databases: either all of a transaction’s updates are observed, or none are.

Still digesting Peter Bailis’s post and the accompanying Scalable Atomic Visibility with RAMP Transactions paper.

Original title and link: Scalable Atomic Visibility with RAMP Transactions (NoSQL database©myNoSQL)


Non-blocking transactional atomicity

Peter Bailis:

tl;dr: You can perform non-blocking multi-object atomic reads and writes across arbitrary data partitions via some simple multi-versioning and by storing metadata regarding related items.

Without the time to go through all the details of the algorithm proposed by Peter Bailis and the various scenarios of a distributed system where the algorithm would have to work, my head was cycling between:

  1. could this actually be expanded to a read/write scenario? at what costs?
  2. isn’t this a form of a (weaker) XA implementation?

Luckly, Peter Bailis is already answering some of these questions in his post1:

If you’re a distributed systems or database weenie like me, you may be curious how NBTA related to well-known problems like two-phase commit.

In case you are familiar with XA, you could start reading the post with the “So what just happened?” section and then dive into the details of the algorithm and possible extensions.

  1. Thanks Peter for stopping my head spin! 

Original title and link: Non-blocking transactional atomicity (NoSQL database©myNoSQL)


Banks and the Ethernal Consistency Example or What trumps consistency

Todd Hoff extracts and expands on some thoughts about BASE vs ACID from Eric Brewer’s NoSQL: Past, Present, Future published on InfoQ:

Consistency it turns out is not the Holy Grail. What trumps consistency is:

  • Auditing
  • Risk Management
  • Availability

But the cornerstone of the availability vs consistency conversation is:

Availability correlates with revenue and consistency generally does not.

✚ Over time Michael Stonebraker has been the most prominent supporter of exactly the opposite argument.

✚ Remember Emin Gün Sirer’s The NoSQL Partition Tolerance Myth? He used the bank example too.

Original title and link: Banks and the Ethernal Consistency Example or What trumps consistency (NoSQL database©myNoSQL)


MongoDB Transactions With TokuDB's Fractal Tree Indexes Engine

Interesting new direction of TokuDB pushing their storage engine based on Fractal Tree Indexes to MongoDB:

Running MongoDB with Fractal Tree Indexes (used today in the MySQL storage engine TokuDB) is fully transactional. Each statement is transactional. If an update is to modify ten rows, then either all rows are modified, or none are. Queries use multi-versioning concurrency control (MVCC) to return results from a snapshot of the system, thereby not being affected by write operations that may happen concurrently.

Original title and link: MongoDB Transactions With TokuDB’s Fractal Tree Indexes Engine (NoSQL database©myNoSQL)


Introducing Highly Available Transactions: The Relationship Between CAP and ACID Transactions

Learning from Peter Bailis:

While the CAP Theorem is fairly well understood, the relationship between CAP and ACID transactions is not. If we consider the current lack of highly available systems providing arbitrary multi-object operations with ACID-like semantics, it appears that CAP and transactions are incompatible. This is partly due to the historical design of distributed database systems, which typically chose consistency over high availability. Standard database techniques like two-phase locking and multi-version concurrency control do not typically perform well in the event of partial failure, and the master-based (i.e., master-per-shard) and overlapping quorum-based techniques often adopted by many distributed database designs are similarly unavailable if users are partitioned from the anointed primary copies.

There’s also a paper (PDF) authored by Peter Bailis, Alan Fekete, Ali Ghodsi, Joseph m. Hellerstein, Ion Stoica. These names should tell you something.

Original title and link: Introducing Highly Available Transactions: The Relationship Between CAP and ACID Transactions (NoSQL database©myNoSQL)


Cloud Computing Lets Us Rethink How We Use Data

But not everything we do in a database needs guaranteed transactional consistency.

Imagine you are charged with designing a system to collect data on temperature, air flow and electricity use in a building every few minutes from hundreds of locations. The system will be used to make the building more energy efficient. Now imagine you lose a few data points every day.  The cause isn’t important but it could be a glitch with a sensor, a dropped packet, or an incomplete write operation in the database.

Do you care?

It depends from what angle I’m looking at this question. If I’m the producer of the sensor, I do care if it has a glitch. If I’m a network administrator I do care there are dropped packets. And if I am a database system I do care if I’m dropping write operations. And I also have to tell whoever is using me if I am able to receive operations—am I available when I’m needed?

Original title and link: Cloud Computing Lets Us Rethink How We Use Data (NoSQL database©myNoSQL)


ACID in HBase: Row Level Operations Explained. Plus Something New

Lars Hofhansl:

HBase employs a kind of MVCC. And HBase has no mixed read/write transactions. […] When a write transaction (a set of puts or deletes) starts it retrieves the next highest transaction number. In HBase this is called a WriteNumber. When a read transaction (a Scan or Get) starts it retrieves the transaction number of the last committed transaction. HBase calls this the ReadPoint.

Understanding the behavior of read and write operations in HBase is definitely useful. Learning that an upcoming HBase version will support atomic multi operations (HBASE-3584) and even multi-row local transactions (HBASE-5229) is priceless.

For HBase atomic multi-operations:

 Delete d = new Delete(ROW);
 Put p = new Put(ROW);
 AtomicRowMutation arm = new AtomicRowMutation(ROW);

and HBase multi-row local transactions is implemented as mutateRowsWithLocks method in HRegion and can be used by coprocessors only (no client API).

Original title and link: ACID in HBase: Row Level Operations Explained. Plus Something New (NoSQL database©myNoSQL)


WhySQL: MySQL/InnoDB ACID Guarantees for Evernote

Dave Engberg has published on the Evernote Techblog a post explaining why the Atomicity, Consistency, and Durability characteristics of a single replicated MySQL/InnoDB deployment are essential to the way Evernote operates.

While it’s difficult to argue about a technical decision with so little details available, I still wanted to point out a couple of things:

  1. Atomicity: most of the NoSQL databases offer atomic operation at the level of a single record. For distributed systems that do not want to rely on 2PC, it is the multi-row atomic operations that are not supported.

    The example presented in the post does not require multi-row transactions, but rather guaranteed client operation ordering. This is achievable in most NoSQL databases.

  2. Consistency: the post talks about data consistency from the perspective of data integrity guarantees through usage of foreign keys.

    In the world of NoSQL similar behavior could be achieved by different data modeling solutions. Using Cassandra as an example for the notebook deletion scenario, one could store all the notes of a notebook in a single Cassandra row, thus making the delete operation safe.

    It’s also worth mentioning that many of the eventually consistent NoSQL databases offer different consistent read and write operations.

  3. Durability: with just a few known exceptions, most NoSQL databases offer strong durability guarantees.

In conclusion, based only on the few details of the post, one could easily argument that a NoSQL database would fit the bill. But most of the time the reality behind is much different, making technical decisions a tad more complicated.

Original title and link: WhySQL: MySQL/InnoDB ACID Guarantees for Evernote (NoSQL database©myNoSQL)


Couchbase Server 2.0 Durability and Write Performance

Matt Ingenthron in a forum thread:

There is quite a bit of work ongoing to optimize some of these paths, and there are some features coming to allow you to specify that you want to block until a change is durable at either the replication or the disk IO level. I believe use that internally to the server for prioritization as well.

Right now we write things as fast as we can and we constantly scan.

I think I’ve seen this before. And I thought Couchbase Server 2.0 will be using CouchDB durable persistence engine. Couchbase Server 2.0 is still in developer preview so there’s time for this to change. But some clarifications would be welcome.

Original title and link: Couchbase Server 2.0 Durability and Write Performance (NoSQL database©myNoSQL)

Redis-Based MOM: Redis for Processing Payments

Santosh Kumar:

As you work more with Redis you soon start finding yourself building out workflows, i.e. small pieces of code that talk to each other via Redis. For someone familiar with a Service Oriented approach to building systems this should feel like deja-vu. Except, instead of using a protocol (HTTP, TCP, UDP, AMQP, ZeroMQ) we are going back to CS101 using a good old queue datastructure.

I think instead of workflows and Service Oriented, the right term is message services or MOM. Indeed, Redis’ blocking queues, the corresponding commands, and PUB/SUB support provides one with the basic building blocks of message services. But these are useful only if you don’t need specialized solutions (RabbitMQ, ActiveMQ, etc.) which will provide solutions for more complicated scenarios.

Later in the post, Kumar mentions implementing transactions for the scenario he picked, payment processing, using Redis’ multi-exec. If things haven’t changed radically lately, I’d underline the fact that Redis MULTI/EXEC/DISCARD is just a batched serialized executor and not ACID transactions.

Original title and link: Redis-Based MOM: Redis for Processing Payments (NoSQL database©myNoSQL)