Michael Stonebraker has published on Vertica blog an article presenting 6 criteria for characterizing the completeness of a column store implementation:

I/O Characteristics

• IO-1 (basic column store): Every storage block contains data from only ONE column.
• IO-2: Aggressive compression
• IO-3: No record-ids

CPU Characteristics

• CPU-4: A column executor
• CPU-5: Executor runs on compressed data
• CPU-6: Executor can process columns that are key sequence or entry sequence

Michael’s post is going after big fishes in the ocean (SybaseIQ, EMC Greenplum, Aster Data, Oracle) and in case this is the area that interests you, you should also check Curt Monash’s follow up.

But getting back to these 6 criteria for column stores, I confess that this time these seem to make a lot of sense. So, I’m wondering how NoSQL column-stores — Cassandra, HBase, and Hypertable — are doing from this perspective. I’d really appreciate some expert comments so we have a follow up with the status of NoSQL column-stores according to these criteria.

Update: Alex Feinberg pointed me to Daniel Abadi’s article that clarifies the distinction between solutions Michael’s post is mentioning and the new NoSQL column stores.

While not remembering exactly this article, I’ve continued to maintain this separation and my post’s intention is to make sure the separation is kept, but also to get experts feedback on the following questions:

• do any of these criteria apply to NoSQL column stores?
• if a criterion applies than how NoSQL column stores score at it?
• if a criterion doesn’t apply, why doesn’t it apply?

Original title and link: 6 Criteria for Real Column Stores (NoSQL databases © myNoSQL)

Many improvements to garbage collection, a Hypertable monitoring web interface, upgraded Thrift and many more. The complete list of changes for Hypertable 0.9.4.0 can be found ☞ here.

I’ve embedded also a presentation by Doug Judd on Hypertable (nb: if you prefer videos you should check this great presentation: Hypertable: The Ultimate Scaling Machine

Original title and link: Hypertable 0.9.4.0 Released, Over 40 Improvements and Bug Fixes (NoSQL databases © myNoSQL)

Interesting to note that some VoltDB don’ts from the paper ☞ Do’s and Don’ts (pdf) are validating some major assumptions in the NoSQL space:

Don’t create tables with very large rows (that is, lots of columns or large VARCHAR columns). Several smaller tables with a common partitioning key are better.

Basically both wide-column stores (i.e. Cassandra, HBase, Hypertable) with their column-families and document databases (i.e. CouchDB, MongoDB, RavenDB, Terrastore) with their schema-less approach are addressing this issue.

1. Don’t use ad hoc SQL queries as part of a production application.

Firstly this points to the mindset change required by the NoSQL space when doing data modeling: think about data access patterns.

Secondly, it pretty much validates CouchDB and RavenDB approaches of having queries defined upfront making their reads extremely fast.

About one month ago, the Adobe SaaS Infrastructure Team (ASIT) has published two excellent articles^[1] on their experience and work with HBase. I had a chance to get into some more details with the team driving this effort — thanks a lot guys! — and here is the final result of our conversation:

myNoSQL: It looks like during your evaluation phase^[2], you’ve only considered column-based solutions (Cassandra, HBase, Hypertable). Would you mind describing some of your requirements that lead you to not consider others?

ASIT: There weren’t many solutions that could respond to our needs in 2008 (approximately 1 year before the NoSQL movement started).

We considered other solutions as well (e.g. memcachedb, Berkeley DB), but we only evaluated the three you mentioned.

We had two divergent requirements: dynamic schema and huge number of records.

We had to provide a data storage service that could handle 40 Million entities, each with many different 1-N relations. These required real-time read/write access. Due to performance reasons, we used to denormalize data to avoid joins. At the same time we had to offer schema flexibility (clients could add their custom metadata), and this was relatively weird to do in MySQL. In the columnar model you could dynamically and easily add new columns.

Every few hours data had to be aggregated in order to compute some sort of billboards and other statistics (client facing, not internal usage information). We were familiar with Hadoop / Map-Reduce, and we were looking for something that could distribute the processing jobs.

myNoSQL: You also seemed to be concerned by data corruption scenarios — this being mentioned in both articles^[3] — and that sounds like a sign of HBase not being stable enough. Is that the case? Or was it something else to lead you to dig deeper in this direction?

ASIT: Although HBase stability was initially unknown to us (we didn’t have experience running the system in production), we would have went through the same draconic testing with any other system. We promised our clients that we won’t lose or corrupt their data so this criteria is paramount.

In a relational database you have referential integrity, databases rarely get corrupted and there are multiple ways to backup your data and restore it. Moreover, you understand where the data is, and know what the filesystem guarantees are. Many times there are teams that deal with backups and database operations for you (not our case however).

When you go with a different model, and you have huge volumes of data, it becomes very hard to back it up frequently and restore it fast. With HBase and Hadoop under active development we need to make sure all the building blocks are “safe”. Our clients have different requirements, but there is a very important common one: data safety.

myNoSQL: The whole setup for being able to move from HBase to MySQL and back sounds really interesting. Could you expand a bit on what tools have you used?

ASIT: We built most of the system in-house. Everything was running on a bunch of cron jobs. The system would:

1. Export the actual HBase tables to HDFS, using map-reduce. This also performed data integrity checks.
2. We had two backup “end-points”. Once the data was exported to HDFS, we compressed it and pushed it into another distributed file system.
3. Another step was to create a MySQL backup out of it and import it on a stand-by MySQL cluster that could act as failover storage. The MySQL failover didn’t have any processing capabilities, so it was running in “degraded mode”, but it was good enough for a temporary solution.

We had the same thing running “in reverse”. The MySQL backup was kept in more places, and it could be imported back in HBase.

On top of this we had monitoring so whenever something went wrong we got alerted. We also had variable backup frequency (keep 1 for each of last 6 months, one for each last 4 weeks, and the last 7 days, also 2 hourly level) to help with disk utilization.

This was supposed to be a temporarily solution, because beyond a certain threshold the data would have gotten to big to fit in MySQL.

Today we look at multi-site data replication as a way of backing up large amounts of data in a scalable manner and soon HBase will support this. We cannot do failover to a MySQL system with our current data volume :).

myNoSQL: Based on the two articles I have had a hard time deciding if the system is currently in production or not. Is it? In case it is, how many ops people are responsible for managing your HBase cluster?

ASIT: There is actually more than one cluster, one is currently in production and the larger one we’re mentioning in the article will enter production soon. This is going to sound controversial, but we - the team - are solely responsible with managing our clusters. Of course, someone else is dealing with the basics: rack&stack, replacing defective hardware, prepping new servers and making sure we are allocated the right IP addresses.

We believe in the dev-ops movement. We automate everything and do monitoring so the operations work for the production system is really low. Last time we had to intervene was due to a bug in the backup frequency algorithm (we exceeded our quota, and we had alerts pouring in).

For full disclosure: These are not business critical systems, today. At some point we will need a dedicated operations team but we’ll still continue to work on the entire vertical, from the feature development part down to actual service deployment and monitoring.

myNoSQL: You mention that you are developing against the HBase trunk. That could mean that you are updating your cluster quite frequently. How do you achieve this without compromising uptime?

ASIT: This is the area we’re starting to focus more in the same manner we did with data integrity.

The current production cluster runs on HBase and Hadoop 0.18.X. Since the first time it failed in December 2008 we didn’t have any downtime, but we didn’t upgrade it either as we envisioned that the new system would replace it completely at some point.

Currently it’s possible to upgrade a running cluster one machine at a time, without downtime if RPC compatibility is maintained between the builds. Major upgrades that impact the RPC interface or data-model could be done without downtime using multi-site replication, but we haven’t done it yet. However in order to have something solid we would need rolling upgrades capabilities from both HDFS and HBase. It theory, this will be possible once they migrate the current RPC to Avro and enable RPC versioning. In the meantime we make sure our clients understand that we might have downtime when upgrading. We’ll probably contribute to the rolling upgrades features as well.

myNoSQL: Based on the following quote from your articles, I’d be tempted to say that some of the requirements were more about ACID guarantees. How would you comment?

So, for us, juggling with Consistency, Availability and Partition tolerance is not as important as making sure that data is 100% safe.

ASIT: It boils down to the requirements of the applications that run on top of your system. If you provide APIs that need to be fully consistent you have to be able to ensure that in every layer of the system. Some of our clients do need full consistency.

Another reason why we need full consistency is to be able to correctly build higher level features such as distributed transactions and indexing. These go hand in hand and are a lot easier to implement when the underlaying system can provide certain guarantees. It’s much like the way Zookeeper relies on TCP for data transmission guarantees, rather than implementing it from scratch.

We’re not consistency “zealots” and understand the benefits of eventual consistency like increased performance and availability in some scenarios. A good example is how we’ll deal with multi-site replication. It’s just that most of our use-cases fit better over a fully consistent system.

myNoSQL: There are a couple more related to the Full consistency section, but I don’t want to flood you with too many questions.

ASIT: This is indeed an interesting topic and we’ll probably post more about it on ☞ http://hstack.org.

myNoSQL: Thanks a lot guys!

I have spent some time to understand the complex solution for HBase secondary indexes suggested ☞ over here. As I pointed out in that post comment thread, I do see a few major drawbacks to this approach. Anyway, now that the code seem to have been made ☞ available, I expect more experienced HBase users will take a look at it and agree or disagree with its approach.

Meanwhile, I get the feeling that this ☞ other solution might be better as it is built on HBase API and not trying to trick HBase behavior.

Expert opinions?

Update: Bruno Dumon is pointing out in the comments below that the two solutions are in fact pretty similar and that “my indexing package basically goes more into detail on that aspect: generating appropriate index row keys, while ignoring how updates should be pushed to the index (I’m thinking of some scalable queue solution for this)”.

There have been confirmed rumors^[1] about Twitter planning to use Cassandra for a long time. But except the mentioned post, I couldn’t find any other references.

Twitter is fun by itself and we all know that NoSQL projects love Twitter. So, imagine how excited I was when after posting about Cassandra 0.5.0 release, I received a short email from Ryan King, the lead of Cassandra efforts at Twitter simply saying that he would be glad to talk about these efforts.

So without further ado, here is the conversation I had with Ryan King (@rk) about Cassandra usage at Twitter:

MyNoSQL: Can you please start by stating the problem that lead you to look into NoSQL?

Ryan King: We have a lot of data, the growth factor in that data is huge and the rate of growth is accelerating.

We have a system in place based on shared mysql + memcache but its quickly becoming prohibitively costly (in terms of manpower) to operate. We need a system that can grow in a more automated fashion and be highly available.

MyNoSQL: I imagine you’ve investigated many possible approaches, so what are the major solutions that you have considered?

Ryan King:

• A more automated sharded mysql setup
• Various databases: HBase, Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, HyperTable and probably some others I’m forgetting.

MyNoSQL: What kind of tests have you run to evaluate these systems?

Ryan King: We first evaluated them on their architectures by asking many questions along the lines of:

• How will we add new machines?
• Are their any single points of failure?
• Do the writes scale as well?
• How much administration will the system require?
• If its open source, is there a healthy community?
• How much time and effort would we have to expend to deploy and integrate it?
• Does it use technology which we know we can work with? *… and so on.

Asking these questions narrowed down our choices dramatically. Everything but Cassandra was ruled out by those questions. Given that it seemed to be our best choice, we went about testing its functionality (“can we reasonably model our data in this system?”) and load testing.

The load testing mostly focused on the write-path. In the medium/long term we’d like to be able to run without a cache in front of Cassandra, but for now we have plenty of memcache capacity and experience with scaling traffic that way.

MyNoSQL: If you draw a line, what were the top reasons for going with Cassandra?

Ryan King:

• No single points of failure
• Highly scalable writes (we have highly variable write traffic)
• A healthy and productive open source community

MyNoSQL: Will Cassandra completely replace the current solution?

Ryan King: Over time, yes. We’re currently moving our largest (and most painful to maintain) table — the statuses table, which contains all tweets and retweets. After this we’ll start putting some new projects on Cassandra and migrating other tables.

MyNoSQL: How do you plan to migrate existing data?

Ryan King: We have a nice system for dynamically controlling features on our site. We commonly use this to roll out new features incrementally across our user base. We use the same system for rolling out new infrastructure.

So to roll out the new data store we do this:

1. Write code that can write to Cassandra in parallel to Mysql, but keep it disabled by the tool I mentioned above
2. Slowly turn up the writes to Cassandra (we can do this by user groups “turn this feature on for employees only” or by percentages “turn this feature on for 1.2% of users”)
3. Find a bug :)
4. Turn the feature off
5. Fix the bug and deploy
6. GOTO #2

Eventually we get to a point where we’re doing 100% doubling of our writes and comfortable that we’re going to stay there. Then we:

1. Take a backup from the mysql databases

2. Run an importer that imports the data to cassandra

   Some side notes here about importing. We were originally trying to use the BinaryMemtable^[2] interface, but we actually found it to be too fast — it would saturate the backplane of our network. We’ve switched back to using the Thrift interface for bulk loading (and we still have to throttle it). The whole process takes about a week now. With infinite network bandwidth we could do it in about 7 hours on our current cluster.

3. Once the data is imported we start turning on real read traffic to Cassandra (in parallel to the mysql traffic), again by user groups and percentages.

4. Once we’re satisfied with the new system (we’re using the real production traffic with instrumentation in our application to QA the new datastore) we can start turning down traffic to the mysql databases.

A philosophical note here — our process for rolling out new major infrastructure can be summed up as “integrate first, then iterate”. We try to get new systems integrated into the application code base as early in their development as possible (but likely only activated for a small number of people). This allows us to iterate on many fronts in parallel: design, engineering, operations, etc.

MyNoSQL: Please include anything I’ve missed.

Ryan King: I can’t really think of anything else.

MyNoSQL: Thank you very much!

I think this is a great week for the NoSQL column stores, marking new releases from both of the most important column-based solutions we are tracking here on MyNoSQL: Cassandra and HBase.

While their releases went pretty quiet, both Cassandra 0.5.0 and HBase 0.20.3 contain a ton of goodies that MyNoSQL tried to track:

• Release: Cassandra 0.5.0 is Here
• Release: HBase 0.20.3 brings 84 bug fixes and improvements

I really hope that both Cassandra and HBase communities will start being more “public” about their work and will give us a chance to read about them more often here on MyNoSQL.