After a very long silence (my last post about Hypertable dates back in Oct. 2010: NoSQL database architectures and Hypertable), there seems to be a bit of revival in the Hypertable space:

1. there are new packages of (commercial) services (PR announcement):
   1. Uptime support subscription
   2. Training and certification
   3. Commercial license
2. it seems like Hypertable has a customer in Rediff.com (India)
3. it is taking yet another stab at HBase performance

While I’m somehow glad that Hypertable didn’t hit the deadpool, it’s quite disappointing that they are still trying to use this old and completely useless strategy of attacking another product in the market.

There are probably many marketers out there encouraging companies to use this old trick of getting attention by attacking the market leader¹. And one of the simplest ways of doing that is by saying “mine is bigger than yours“.

But these days this strategy isn’t working anymore for quite a few reasons:

1. benchmarks are most of the time incorrect, thus the attention will be pointed in the wrong direction.

   In the case of the Hypertable vs HBase benchmark, JD Cryans (HBase veteran) is demoting the results.

2. For existing users, performance issues are already known. Performance issues are also known by core developers that are always working to address them. So nothing new, just some angry users of the attacked product.

3. For new users, performance is just one aspect of the decision. Most of the time, it’s one of the last considered. Community, support, adoption, and well know case studies are much more important.

Attacking competitors based on feature checklists might be slightly effective in attracting a bit of attention, but it’s not the strategy to get users and customers and grow a community.

Original title and link: Hypertable Revival. Still the wrong strategy (NoSQL database©myNoSQL)

The National Security Agency has submitted to Apache Incubator a proposal to open source Accumulo, a BigTable inspired key-value store that they were building since 2008. The project proposal page provides more details about Accumulo history, building blocks, and how it compares to the other BigTable open source implementation HBase:

• Access Labels: Accumulo has an additional portion of its key that sorts after the column qualifier and before the timestamp. It is called column visibility and enables expressive cell-level access control. Authorizations are passed with each query to control what data is returned to the user.

• Iterators: Accumulo has a novel server-side programming mechanism that can modify the data written to disk or returned to the user. This mechanism can be configured for any of the scopes where data is read from or written to disk. It can be used to perform joins on data within a single tablet.

• Flexibility: Accumulo places no restrictions on the column families. Also, each column family in HBase is stored separately on disk. Accumulo allows column families to be grouped together on disk, as does BigTable.

• Logging: HBase uses a write-ahead log on the Hadoop Distributed File System. Accumulo has its own logging service that does not depend on communication with the HDFS NameNode.

• Storage: Accumulo has a relative key file format that improves compression.

You can read more about Accumulo here and check the Hacker News and Reddit discussions.

Michael Stack has commented on the HBase mailing list:

The cell based ‘access labels’ seem like a matter of adding an extra field to KV and their Iterators seem like a specialization on Coprocessors. The ability to add column families on the fly seems too minor a difference to call out especially if online schema edits are now (soon) supported. They talk of locality group like functionality too — that could be a significant difference. We would have to see the code but at first blush, differences look small.

Original title and link: Accumulo: A New BigTable Inspired Distributed Key/Value by NSA (NoSQL database©myNoSQL)

This paper from Google talks extensively about the usage of BigTable and Megastore, the data model, query processing, and transaction handling in the implementation of Google Fusion Tables.

Google Fusion Tables is a cloud-based service for data management and integration. Fusion Tables enables users to upload tabular data files (spreadsheets, CSV, KML), currently of up to 100MB. The system provides several ways of visualizing the data (e.g., charts, maps, and timelines) and the ability to filter and aggregate the data. It supports the integration of data from multiple sources by performing joins across tables that may belong to different users. […] This paper describes the inner workings of Fusion Tables, including the storage of data in the system and the tight integration with the Google Maps infrastructure.

Download the paper or read it after the break.

Cloudata is the third open source implementation of Google’s BigTable paper, after HBase and Hypertable^[1]. There’s already an 1.0 version even if the Github project page is listing just a couple of commits.

From the home page, Cloudata’s current features:

• Basic data service
  • Single row operation(get, put)
  • Multi row operation(like, between, scanner)
  • Data uploader(DirectUploader)
  • MapReduce(TabletInputFormat)
  • Simple cloudata query and supports JDBC driver
• Table Management
  • split
  • distribution
  • compaction
• Utility
  • Web based Monitor
  • CLI Shell
• Failover
  • Master failover
  • TabletServer failover
• Change log Server
  • Reliable fast appendable change log server
• Support language
  • Java, RESTful API, Thrift

I couldn’t figure out if this is just an experiment or if it actually plans to be a real project.

Update: Cloudata’s author, Jsjangg, mentions in the comment thread that Cloudata is used at www.searcus.com for 2 years already running on a 20 machine cluster.

Original title and link: Cloudata: New Open Source BigTable Implementation (NoSQL databases © myNoSQL)

The slides below summarizing the Google BigTable paper are the result of a NOSQLSummer meeting in Tokyo. Nice!

Update: I just realized that the company that hosted this meeting, Gemini Mobile Technologies, is the same that announced yesterday the new key-value store Hibari

1. Kirk McKusick, Sean Quinlan: ☞ GFS: Evolution on Fast-Forward ¶

   Sounds pretty hackish here and there. But it might be very similar to the whole philosophy behind NoSQL: solve your own problem, come up with a cost effective approach and deliver value to your company.

2. blog.katipo.co.nz: ☞ Installing MongoDB on Mac OS X using Homebrew. ¶

   As a side note, last week I went through installing CouchDB master on my MacOS and discovered that while Homebrew works pretty well, you’ll still need to hack it from time to time. Plus, having MacPorts installed on the same machine made this process almost impossible.

3. Loraine Lawson: ☞ The Business Pros and Cons of NoSQL ¶

Tony Bain was probably ☞ tricked to think so based on news that Cassandra is used by Digg [1], Twitter [2] etc. To me those are just signs that:

• Cassandra has finally gathered a community behind it [3]
• they have identified good or common use cases

Secondly, the NoSQL world is quite wide. Cassandra is a column-oriented store (in the same category: BigTable, Hypertable, HBase), but we also have key-value stores, document stores, graph stores — see [4], [5] and [6] for more details — so saying that it is winning the race is incorrect. So, at best it should be compared with the other column-oriented solution.

Thinking of HBase, we recently learnt [7] that is doing well too, that there are real-life production applications running on it, and that it has seen good performance improvements over the last couple of releases. And as far as I know there is a larger community behind it.

You should also check the HBase vs. Cassandra: NoSQL Battle! article to better understand how they compare and where they differ and also Cassandra Gets (Better) Documentation for some very good references.