Quantcast released a new Hadoop file system QFS:

1. fully compatible with HDFS
2. licensed under Apache 2.0 license
3. written in C++
4. while HDFS replicates data 3 times, QFS requires only 1.5x raw capacity
5. QFS supports two types of fault tolerance: chunk replication and Reed-Solomon encoding

6. QFS components (more details here):

   

7. QFS performance comparison to HDFS:

   

Now I’m looking forward to hear comments from HDFS experts about QFS.

Original title and link: Quantcast File System for Hadoop (NoSQL database©myNoSQL)

Shawn Rogers has a short but compelling list of Big Data deployments in his article Big Data is Scaling BI and Analytics. This list also shows that even if there are some common components like Hadoop, there are no blueprints yet for dealing with Big Data.

• Facebook: Hadoop analytic data warehouse, using HDFS to store more than 30 petabytes of data. Their Big Data stack is based only on open source solutions.

• Quantcast: 3,000 core, 3,500 terabyte Hadoop deployment that processes more than a petabyte of raw data each day

• University of Nebraska-Lincoln: 1.6 petabytes of physics data Hadoop cluster

• Yahoo!: 100,000 CPUs in 40,000 computers, all running Hadoop. Also running a 12 terabyte MOLAP cube based on Tableau Software

• eBay: has 3 separate analytics environments:

  • 6PB data warehouse for structured data and SQL access
  • 40PB deep analytics (Teradata)
  • 20PB Hadoop system to support advanced analytic workload on unstructured data

Original title and link: Big Data Is Going Mainstream: Facebook, Yahoo!, eBay, Quantcast, and Many Others (NoSQL database©myNoSQL)