benchmark: All content tagged as benchmark in NoSQL databases and polyglot persistence
I usually don’t trust vendor benchmarks, but these Riak benchmarks look pretty much inline with Mozilla’s Riak benchmark. What is even more impressive is that these results were from running Riak on virtualized machines (the Joyent SmartMachines).
Watch it for youself. Slides can be downloaded from ☞ here
I ran three micro-benchmarks: get by primary key, get by secondary key and update by primary key. MySQL had a higher peak QPS for all of them. Alas, the results for get by primary key were skewed because pymongo, the Python driver for MongoDB, uses more CPU than MySQLdb, the Python driver for MySQL. The client host was saturated during the test and this limited peak QPS to 80,000 for MongoDB versus 110,000 for MySQL.
I repeated one test using two 16-core client hosts with 40 processes per host. For that test the peak QPS on MongoDB improved to 155,000 while the peak for MySQL remained at 110,000. That is an impressive result. The results for get by secondary key and update by primary key are still valid as the server host saturated on those tests.
A couple of lessons I’ve learned:
- if you are able to use the latest MySQL version and track different patches you can get a lot out of MySQL
- while benchmarking, make sure you also check very carefully the drivers you are using
- while benchmarking, pay attention to how concurrency is influencing the results
- peak QPS is an interesting metric, but it might not be the one you are most interested about. Check out this extensive Riak benchmark run by Mozilla that covers other metrics that might be important for your app
Mozilla has previously published about their detailed plan and extensive investigation into Cassandra, HBase, and Riak that led to choosing Riak. This time they are publishing some extensive Riak benchmark results (against both Riak 0.10 and Riak 0.11 running Bitcask) — they are using Riak benchmarking code, included in the list of correct NoSQL benchmarks and performance evaluations solutions. Both the results, their analysis , and interpretation are fascinating.
Our goal in running these studies was, simply put, no surprises. That meant we needed to run studies to that profiled:
- Stability, especially for long running tests
- Performance when we introduced variable object sizes
- Performance when we introduced pre-commit hooks to evaluate incoming data
I guess Mozilla Test Pilot is one of the Riak’s most interesting case studies.
After saying that MongoDB’s default fire-and-forget behavior is wrong, CouchDB community welcomed this sample Clojure code showing 5500 inserts/second implemented with a fire-and-forget behavior and bulk inserts:
So I contemplated the problem some and wondered whether Clojure’s STM (Software Transactional Memory) could be leveraged. As requests come in, instead of connecting immediately to the database, why not queue them up until we have an optimal number and then do a bulk insert?
- simple key-values
- hashes, a new data type that will be available with the upcoming Redis 2.0
The experiment is interesting as it shares the code used and so you’ll be able to run it for your particular scenarios. Do keep in mind that the results will vary as they depend heavily on the size of the stored values.
This tells me that on a 32GB box, it’s not unreasonable to host 200,000,000 keys (if their values are sufficiently small). […] The resulting dump file (dump-0.rdb) was 1.8GB in size.
If you do the math, that yields 1.25 billion (1,250,000,000) key/value pairs stored. […] So it took about 2 hours and 40 minutes to complete. The resulting dump file (.rdb file) was 13GB in size (compared to the previous 1.8GB) and the memory usage was roughly 17GB.
Salvatore Sanfilippo (@antirez), Redis creator and main developer, has a good explanation about the storage overhead:
If you turn a txt file with a list of “common surnames -> percentage of population” into a binary tree it will get more or less an order of magnitude bigger in memory compared to the raw txt file.
This is a common pattern: when you add a lot of metadata, for fast access, memory management, “zero-copy” transmission of this information, expires, …, the size is not going to be the one of concatenating all this data like in a unique string.
But for now our reasoning is: it’s not bad to be able to store 1 million of keys with less than 200 MB of memory (100 MB on 32bit systems) if an entry level box is able to serve this data at the rate of 100k requests/second, including the networking overhead. And with hashes we have a much better memory performance compared to top level keys. So… with a few GB our users can store ten or hundred of millions of stuff in a Redis server.
Unfortunately both of them are just new examples of useless benchmarks:
- only 1000 keys
- the benchmark doesn’t vary the size of keys and values
- no concurrency
- no mixed reads/writes
Some say it is the right time to start having these around. Others are saying it’s way to early to start the “battle”. Users do want to see them and in case they’re lacking they create their own, most of the time using incomplete or wrong approaches.
But what am I talking about? As some of you might have guessed already:
But users are more interested in seeing cross product benchmarks, even if most of the time constructing these is extremely complicated and they end up comparing apples with oranges.
All these being said and accepting that most of the time someone will figure out a way to invalidate the results, lets see what cross product benchmarks do we have in the NoSQL space.
Yahoo! Cloud Serving Benchmark
The Yahoo! Cloud Serving Benchmark’s goal is to facilitate performance comparisons of the new generation of cloud data serving systems. The source code is available on ☞ GitHub and Yahoo! has also published ☞ the results of running this benchmark against Cassandra, HBase, Yahoo!’s PNUTS, and a simple sharded MySQL implementation.
VoltDB a new storage solution that calls itself the next-generation SQL RDBMS with ACID for fast-scaling OLTP applications has recently ☞ published the results of their benchmark comparing VoltDB and Cassandra.
It is worth noting that while being one of those apples to oranges comparisons (nb and the authors are well aware of it), there are still a couple of interesting and useful things to be learned from it (i.e. benchmarking procedure, tested scenarios, etc.)
Unfortunately at this time the source code is not yet available, but hopefully we will see it soon:
Going forward, we’re planning to release the code we used to do these benchmarks. We’d also like to try a few other storage layers
Hypertable and HBase Performance Evaluation
The guys behind Hypertable ☞ have published their results of comparing Hypertable with HBase using a benchmark based on the Google BigTable paper from which both HBase and Hypertable are inheriting their architecture.
Unfortunately, the benchmark code is not available at this moment.
So, as far as I could gather we have:
- ☞ Riak internal benchmark
- ☞ MongoDB internal benchmark
- ☞ Yahoo! Cloud Serving Benchmark
- results only of VoltDB Benchmark comparing VoltDB and Cassandra
- BigTable-inspired benchmark comparing Hypertable and HBase
Did I miss any?
Judging by the number of posts I’ve seen around I’d guess you’ve already heard about the MongoDB 1.4 release. Anyways, I definitely had to include it here as myNoSQL covers all major NoSQL projects and follows closely all things related to the NoSQL ecosystem.
- background indexing and indexing improvements
- concurrency improvements
- the lack of autosharding (still alpha, still pushing, still…)
- the lack of improvements or alternatives for the MongoDB durability tradeoff
Speaking of performance, the 10gen people have run some benchmarks comparing MongoDB 1.2 with MongoDB 1.4. Without a couple of exceptions, the performance haven’t improved radically, so I’d speculate that there is still a lot of locking involved. The benchmark source code was made available so you can dig deeper into it.
All in all, good and exciting news for the NoSQL world!