Galileo observed how things broke if they were naively scaled up.

Google found the larger the scale the greater the impact of latency variability. When a request is implemented by work done in parallel, as is common with today’s service oriented systems, the overall response time is dominated by the long tail distribution of the parallel operations. Every response must have a consistent and low latency or the overall operation response time will be tragically slow.

Fantastic post from Todd Hoff on the (hopefully) well known truth: “the reponse time in a distributed parallel systems is the time of the slowest component“.

Original title and link: When More Machines Equals Worse Results (NoSQL database©myNoSQL)

via: http://highscalability.com/blog/2012/3/12/google-taming-the-long-latency-tail-when-more-machines-equal.html

European researchers may have found a way to speed up data storage 100-fold, breaking one barrier holding back how fast data can be transferred. […] The researchers at York University in the U.K. and Nijmegen University in the Netherlands accomplished the feat by heating a magnetic material with laser bursts that alter what is called the magnetic spin of the material at the atomic level, according to an explanation by York University. There are two possible spins, parallel and anti-parallel, and in storage, these binary states would represent the ones and zeros that designate bit types.

I still find the salmon storage more mouthwatering.

Original title and link: Possible 100-fold increase in data storage speed (NoSQL database©myNoSQL)

via: http://www.networkworld.com/news/2012/020812-data-storage-speed-255864.html

An interesting look at what happens during the map phase in Hadoop and the impact of emitting key-value pairs:

• a direct negative impact on the map time and CPU usage, due to more serialization
• an indirect negative impact on CPU due to more spilling and additional deserialization in the combine step
• a direct impact on the map task, due to more intermediate files, which makes the final merge more expensive

The main point of the dynaTrace blog post is that even if Hadoop makes it easy to throw more hardware at a problem, wasting resources with bad code in MapReduce tasks comes with a noticeable and measurable cost.

Original title and link: MapReduce With Hadoop: What Happens During Mapping (NoSQL database©myNoSQL)

via: http://blog.dynatrace.com/2012/01/25/about-the-performance-of-map-reduce-jobs/

John Hugg lists the 3 concepts that make VoltDB fast:

1. Exploit repeatable workloads: VoltDB exclusively uses a stored procedure interface.
2. Partition data to horizontally scale: VoltDB devides data among a set of machines (or nodes) in a cluster to achieve parallelization of work and near linear scale-out.
3. Build a SQL executor that’s specialized for the problem you’re trying to solve.: If stored procedures take microseconds, why interleave their execution with a complex system of row and table locks and thread synchronization? It’s much faster and simpler just to execute work serially.

Let’s take a quick look at these.

Using stored procedures — instead of allowing free form queries — would allow the system:

1. to completely skip query parsing, creating and optimizing execution plans at runtime
2. by analyzing (at deploy time) the set of stored procedures, it might also be possible to generate the appropriate indexes

The benefits of horizontally partitioned data are well understood: parallelization and also easier and cost effective hardware usage.

Single threaded execution can also help by removing the need for locking and reducing data access contention.

While these 3 solutions are making a lot of sense and can definitely make a system faster, there’s one major aspect of VoltDB that’s missing from the above list and which I think is critical to explaining its speed: VoltDB is an in-memory storage solution.

Here are a couple of examples of other NoSQL databases that benefit from being in memory (or as close as possible to it). MongoDB, while being a lot more liberal with the queries it accepts, can deliver very fast results by keeping as much data in memory as possible — remember what happened when it had to hit the disk more often? — and using appropriate indexes where needed. Redis and Memcached can deliver amazingly fast results because they keep all data in-memory. And Redis is single threaded while Memcached is not.

Original title and link: VoltDB: 3 Concepts that Makes it Fast (NoSQL databases © myNoSQL)

via: https://voltdb.com/blog/why-voltdb-so-fast

On 8 core Opteron Box we were able to achieve 275K inserts/sec at which time we started to see load to get IO bound because of log writes and flushing dirty buffers. I’m confident you can get to 400K+ inserts/sec on faster hardware and disks (say better RAID or Flash) which is a very cool number. Of course, mind you this is in memory insertion in the simple table and table with long rows and bunch of indexes will see lower numbers.

There are more caveats in the article. Not sure though how to compare this number with the 750k qps on the NoSQLish MySQL.

Original title and link: High Rate insertion with MySQL and Innodb (NoSQL databases © myNoSQL)

via: http://www.mysqlperformanceblog.com/2011/01/07/high-rate-insertion-with-mysql-and-innodb/

Alex Baranau:

The idea behind deferred updates processing is to postpone updating of the existing record and store incoming deltas as a new record. Thus, record update operations become a simple write operations with corresponding performance. Deferred updates technique elaborated here fits well when system handles a lot of updates of stored data and write performance is the main concern, while reading speed requirements are not that strict. The following cases (each of them separately or any combination of them) may indicate  that one can benefit from using the technique (the list is not complete):

• updates are well spread over the whole and large dataset
• lower rate (among write operations) of “true updates” (i.e. low percentage of writes are for completely new data, not really updates of existing data)
• good portion of data stored/updated may never be accessed
• system should be able to handle high write peaks without major performance degradation

Sounds like CQRS event stores.

Project available on ☞ GitHub.

Original title and link: Deferring Processing Updates to Increase HBase Write Performance (NoSQL databases © myNoSQL)

via: http://blog.sematext.com/2010/12/16/deferring-processing-updates-to-increase-hbase-write-performance/

Project Voldemort gets a performance tool from Roshan Sumbaly :

• Run using bin/voldemort-performance-tool.sh
• Has a warmup phase to insert records (—record-count)
• Various record selection distributions
• Can fix client throughput to measure latency under certain load

via: http://github.com/voldemort/voldemort/commit/8462031d2d8b676b27f533f8f7de5631c8eb70dd