NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter



LevelDB: All content tagged as LevelDB in NoSQL databases and polyglot persistence

MySQL is a great Open Source project. How about open source NoSQL databases?

In a post titled Some myths on Open Source, the way I see it, Anders Karlsson writes about MySQL:

As far as code, adoption and reaching out to create an SQL-based RDBMS that anyone can afford, MySQL / MariaDB has been immensely successful. But as an Open Source project, something being developed together with the community where everyone work on their end with their skills to create a great combined piece of work, MySQL has failed. This is sad, but on the other hand I’m not so sure that it would have as much influence and as wide adoption if the project would have been a “clean” Open Source project.

The article offers a very black-and-white perspective on open source versus commercial code. But that’s not why I’m linking to it.

The above paragraph made me think about how many of the most popular open source NoSQL databases would die without the companies (or people) that created them.

Here’s my list: MongoDB, Riak, Neo4j, Redis, Couchbase, etc. And I could continue for quite a while considering how many there are out there: RavenDB, RethinkDB, Voldemort, Tokyo, Titan.

Actually if you reverse the question, the list would get extremely short: Cassandra, CouchDB (still struggling though), HBase. All these were at some point driven by community. Probably the only special case could be LevelDB.

✚ As a follow up to Anders Karlsson post, Robert Hodges posted The Scale-Out Blog: Why I Love Open Source.

Original title and link: MySQL is a great Open Source project. How about open source NoSQL databases? (NoSQL database©myNoSQL)


Quick Intro to LevelDB

If you haven’t heard of LevelDB before or you forgot some of the details, read Rod Vagg’s short post that will give you an overview of the basics and internals. Particularly interesting is the file organization in LevelDB (which also gives it the name).

Original title and link: Quick Intro to LevelDB (NoSQL database©myNoSQL)


Improvements and Benchmarks for LevelDB in Riak 1.2

Basho team started to investigate and optimize LevelDB, one of the supported storage engine for Riak and the engine for Riak 2i, and the results are already impressive:

  • reduced stalls (from 10-90s every 3-5min to 10-30s every 2h)
  • increased throughput (from 400 ops/s to 2000 ops/s)
  • a better solution for dealing with an infinite loop during compaction against a corrupted data block
  • LevelDB bloom filter for quickly identifying keys that don’t exist in the data store

The original posts also shows some charts of the throughput and maximum latency measured in Level 1.1 vs Level 1.2.

Original title and link: Improvements and Benchmarks for LevelDB in Riak 1.2 (NoSQL database©myNoSQL)


LevelDB: SSTable and Log Structured Storage

Ilya Grigorik digs into LevelDB’s SSTable and log structured storage1:

If Protocol Buffers is the lingua franca of individual data record at Google, then the Sorted String Table (SSTable) is one of the most popular outputs for storing, processing, and exchanging datasets. As the name itself implies, an SSTable is a simple abstraction to efficiently store large numbers of key-value pairs while optimizing for high throughput, sequential read/write workloads.

Even if not very talked about, LevelDB is making notable contributions to the NoSQL space: the leveled compaction strategy in Cassandra 1.0 is based on LevelDB and Riak ships with LevelDB since 1.0.

Original title and link: LevelDB: SSTable and Log Structured Storage (NoSQL database©myNoSQL)


LevelDB Might Be a Great Fit for MongoDB

Mark Callaghan:

MongoDB doesn’t need multi-statement transactions. Both are limited by 1-writer or N-reader concurrency, but writes to database files are much faster with LevelDB because it doesn’t do update in place. So LevelDB doesn’t lose as much performance for IO-bound workloads by doing 1-writer or N-readers and my guess is that this could make MongoDB much better at supporting IO-bound workloads.

If getting LevelDB as a storage engine for MySQL is doubtful and I wouldn’t hold my breath until MongoDB would be using it, keep in mind that Riak might actually use LevelDB at some point.

Update: Justin Sheehy (CTO, Basho) has kindly (again) corrected me: LevelDB is supported and ships with Riak since the release of Riak 1.0. Plus it is the backend for the new secondary indexes, as it was correctly mentioned in the comments by Philip Cristiano.

Original title and link: LevelDB Might Be a Great Fit for MongoDB (NoSQL database©myNoSQL)


LevelDB and Kyoto Cabinet Benchmark

I’ve been pretty excited about Google’s LevelDB, not to mention there are some really old tanks already in the battle field like BDB, Tokyo Cabinet (Kyoto Cabinet as new one), HamsterDB etc. Fortunately I’ve already worked with Kyoto Cabinet and when I looked at the benchmarks I was totally blown away.

His benchmark results are radically different than the ones published in the LevelDB benchmark.

Original title and link: LevelDB and Kyoto Cabinet Benchmark (NoSQL database©myNoSQL)


LevelDB: Google’s Fast Persistent Key-Value Store Library

Google open sourced a while ago LevelDB , a C++ library that provides an ordered mapping key-value storage. LevelDB performance convinced Basho guys to experiment with adding LevelDB as a storage engine for Riak. And there’s also a benchmark comparing LevelDB with SQLite and Kyoto Cabinet.

The LevelDB project lists the following key features:

  • Keys and values are arbitrary byte arrays.
  • Data is stored sorted by key.
  • Callers can provide a custom comparison function to override the sort order.
  • The basic operations are Put(key,value), Get(key), Delete(key).
  • Multiple changes can be made in one atomic batch.
  • Users can create a transient snapshot to get a consistent view of data.
  • Forward and backward iteration is supported over the data.
  • Data is automatically compressed using the Snappy compression library.
  • External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.
  • Detailed documentation about how to use the library is included with the source code.

You can check out also the old thread on Hacker News about LevelDB..

Original title and link: LevelDB: Google’s Fast Persistent Key-Value Store Library (NoSQL database©myNoSQL)

Riak Getting LevelDB as Storage Engine

After Innostore and Bitcask, Basho guys are currently experimenting with integrating Google’s LevelDB as a storage engine for Riak. Preliminary results are looking promising:

For most Riak users, Bitcask is the obvious right storage engine to use. It provides low latency, solid predictability, is robust in the face of crashes, and is friendly from a filesystem backup point of view. However, it has one notable limitation: total RAM use depends linearly (though via a small constant) on the total number of objects stored. For this reason, Riak users that need to store billions of entries per machine sometimes use Innostore (our wrapper around embedded InnoDB) as their storage engine instead. InnoDB is a robust and well-known storage engine, and uses a more traditional design than Bitcask which allows it to tolerate a higher maximum number of items stored on a given host.

It appears that LevelDB may become a preferred choice for Riak users whose data set has massive numbers of keys and therefore is a poor match with Bitcask’s model. Performance aside, it compares favorably to InnoDB on other issues such as permissive license and operational usability.

Original title and link: Riak Getting LevelDB as Storage Engine (NoSQL database©myNoSQL)