Shawn Rogers has a short but compelling list of Big Data deployments in his article Big Data is Scaling BI and Analytics. This list also shows that even if there are some common components like Hadoop, there are no blueprints yet for dealing with Big Data.

• Facebook: Hadoop analytic data warehouse, using HDFS to store more than 30 petabytes of data. Their Big Data stack is based only on open source solutions.

• Quantcast: 3,000 core, 3,500 terabyte Hadoop deployment that processes more than a petabyte of raw data each day

• University of Nebraska-Lincoln: 1.6 petabytes of physics data Hadoop cluster

• Yahoo!: 100,000 CPUs in 40,000 computers, all running Hadoop. Also running a 12 terabyte MOLAP cube based on Tableau Software

• eBay: has 3 separate analytics environments:

  • 6PB data warehouse for structured data and SQL access
  • 40PB deep analytics (Teradata)
  • 20PB Hadoop system to support advanced analytic workload on unstructured data

Original title and link: Big Data Is Going Mainstream: Facebook, Yahoo!, eBay, Quantcast, and Many Others (NoSQL database©myNoSQL)

Links to the most publicized paper of a Hadoop use case, Facebook has presented at SIGMOD 2011: paper and presentation slides:

Facebook recently deployed Facebook Messages, its first ever user-facing application built on the Apache Hadoop platform. Apache HBase is a database-like layer built on Hadoop designed to support billions of messages per day. This paper describes the reasons why Facebook chose Hadoop and HBase over other systems such as Apache Cassandra and Voldemort and discusses the application’’s requirements for consistency, availability, partition tolerance, data model and scalability. We explore the enhancements made to Hadoop to make it a more effective realtime system, the tradeoffs we made while configuring the system, and how this solution has significant advantages over the sharded MySQL database scheme used in other applications at Facebook and many other web-scale companies. We discuss the motivations behind our design choices, the challenges that we face in day-to-day operations, and future capabilities and improvements still under development. We offer these observations on the deployment as a model for other companies who are contemplating a Hadoop-based solution over traditional sharded RDBMS deployments.

Also embedded below for your quick reference:

Dhruba Borthakur started a series of posts — part 1 and part 2 — describing both the process that lead Facebook to using HBase and Hadoop, but also the projects where these are used and their requirements:

After considerable research and experimentation, we chose Hadoop and HBase as the foundational storage technology for these next generation applications. The decision was based on the state of HBase at the point of evaluation as well as our confidence in addressing the features that were lacking at that point via in- house engineering. HBase already provided a highly consistent, high write-throughput key-value store. The HDFS NameNode stood out as a central point of failure, but we were confident that our HDFS team could build a highly-available NameNode (AvatarNode) in a reasonable time-frame, and this would be useful for our warehouse operations as well. Good disk read-efficiency seemed to be within striking reach (pending adding Bloom filters to HBase’’s version of LSM Trees, making local DataNode reads efficient and caching NameNode metadata). Based on our experience operating the Hive/Hadoop warehouse, we knew HDFS was stellar in tolerating and isolating faults in the disk subsystem. The failure of entire large HBase/HDFS clusters was a scenario that ran against the goal of fault-isolation, but could be considerably mitigated by storing data in smaller HBase clusters. Wide area replication projects, both in-house and within the HBase community, seemed to provide a promising path to achieving disaster recovery.

The second part is describing 3 problems Facebook is solving using HBase and Hadoop and provides further details on the requirements of each of these.

The two posts represent a great resource for understanding not only where HBase and Hadoop can be used, but also on how to formulate the requirements (and non-requirements) for new systems.

A Facebook team will present the paper “Apache Hadoop Goes Realtime at Facebook” at ACM SIGMOD. I’m looking forward for the moment the paper will be available.

Original title and link: HDFS: Realtime Hadoop and HBase Usage at Facebook (NoSQL databases © myNoSQL)

More applications of HBase at Facebook, after the new messaging system:

If you are interesting to read more about Facebook messages here’s a list of posts:

• Facebook replacing Cassandara with HBase in new messaging system
• The underlying technology of messages using HBase
• HBase at Facebook and why not MySQL or Cassandra
• HBase at Facebook: A technical presentation of the underlying technology
• Facebook messages: a presentation from FOSDEM

Original title and link: Facebook Builds HBase-based Real-Time Analytics (NoSQL databases © myNoSQL)

From this year’s FOSDEM, Facebook talking about the technology behind the messaging platform:

Original title and link: Facebook Messages: FOSDEM NoSQL Event (NoSQL databases © myNoSQL)

There have been lots of discussions and speculations after the announcement that Facebook is using HBase for the new messaging system. In case you missed it, here are the most important bits:

• Kannan Muthukkaruppan: The underlying Technology of Messages (facebook.com)

  We spent a few weeks setting up a test framework to evaluate clusters of MySQL, Apache Cassandra, Apache HBase, and a couple of other systems. We ultimately chose HBase. MySQL proved to not handle the long tail of data well; as indexes and data sets grew large, performance suffered. We found Cassandra’s eventual consistency model to be a difficult pattern to reconcile for our new Messages infrastructure.

• Quora.com: How does HBase write performance differ from write performance in Cassandra with consistency level ALL

  While setting the a write consistency level of ALL with a read level of ONE in Cassandra provides a strong consistency model similar to what HBase provides (and in fact using quorum writes and reads would as well), the two operations are actually semantically different and lead to different durability and availability guarantees.

• Cassandra mailing list: Facebook messaging and choice of HBase over Cassandra

• Todd Hoff: Facebook’s New Real-Time Messaging System: HBase to Store 135+ Billion Messages a Month (highscalability.com)

  HBase is a scaleout table store supporting very high rates of row-level updates over massive amounts of data. Exactly what is needed for a Messaging system. HBase is also a column based key-value store built on the BigTable model. It’s good at fetching rows by key or scanning ranges of rows and filtering. Also what is needed for a Messaging system. Complex queries are not supported however. Queries are generally given over to an analytics tool like Hive, which Facebook created to make sense of their multi-petabyte data warehouse, and Hive is based on Hadoop’s file system, HDFS, which is also used by HBase.

• Jeremiah Peschka: Facebook messaging - HBase Comes of Age (facility9.com)

  Existing expertise: The technology behind HBase – Hadoop and HDFS – is very well understood and has been used previously at Facebook. […] Since Hive makes use of Hadoop and HDFS, these shared technologies are well understood by Facebook’s operations teams. As a result, the same technology that allows Facebook to scale their data will be the technology that allows Facebook to scale their Social Messaging feature. The operations team already understands many of the problems they will encounter.

• Quora.com: What version of HBase is Facebook using for its new messaging platform?

  Facebook has an internal branch of HBase which periodically updates from the Apache SVN. As far as I know, the current version in production is very similar to the 0.89.20100924 development release with a couple more patches pulled in from trunk.

  Facebook engineers continue to actively contribute to the open source trunk, though - it’s not an internal “fork”

  Todd Lipcon (HBase committer)

The engineering team behind Facebook’s new messaging system has posted now a video talking more about their choice of HBase. You can watch the a bit over 1 hour long video here.

The engineering team behind Facebook Messages spent the past year building out a robust, scalable infrastructure. We shared some details about the technology on our Engineering Blog (http://fb.me/95OQ8YaD2rkb3r). This tech talk digs deeper into some of the twenty different infrastructure services we created for the project as well as how we’re using Apache HBase.

I’m still watching the video, so my notes will follow.

Why HBase?

• Strong consistency model
• Automatic failover

• Multiple shards per server for load balancing

• Prevents cascading failures

• Compression: save disk space and network bandwidth

• Read-modify-write operation support, like counter increment
• Map Reduce supported out of the box

I’m still not sure why one needs a strong consistency model for messages (and that’s the part missing from all these articles).

As a side note, I feel like the decission was based not on some major facts, but rather a sum of small but important features that HBase was offering compared to other solutions (i.e. consistent increments, perfect integration with Hadoop, etc.)

Original title and link: HBase at Facebook: The Underlying Technology of Messages (NoSQL databases © myNoSQL)

When Facebook talks MySQL, it usually means BigData MySQL, high availability and scalable MySQL, and last, but not least NoSQLized MySQL. Mark Callaghan:

And another one from O’Reilly MySQL Conference: