Nice article and interviews with important people in the history of Hadoop by Derrick Harris for GigaOm.

Depending on how one defines its birth, Hadoop is now 10 years old. In that decade, Hadoop has gone from being the hopeful answer to Yahoo’s search- engine woes to a general-purpose computing platform that’s poised to be the foundation for the next generation of data-based applications.

This is the first article of a four-part series that promises to explain everything about Hadoop.

Original title and link: The History of Hadoop (NoSQL database©myNoSQL)

via: http://gigaom.com/2013/03/04/the-history-of-hadoop-from-4-nodes-to-the-future-of-data/

What is the big deal about Sort? Sort is fundamental to the MapReduce framework, the data is sorted between the Map and Reduce phases (see below). Syncsort’s contribution allows native Hadoop sort to be replaced by an alternative sort implementation, for both Map and Reduce sides, i.e. it makes Sort phase pluggable.

Tendu Yogurtcu describes a couple of new use cases that the pluggable sort implementation contributed by Syncsort to Apache Hadoop is opening:

1. Optimized sort implementations and full joins
2. Hash-based aggregations with no sort requirements
3. Reducers that can start before all Mappers complete

Original title and link: Use Cases for Hadoop’s New Pluggable Sort (NoSQL database©myNoSQL)

via: http://blog.syncsort.com/2013/02/hadoop-mapreduce-to-sort-or-not-to-sort/

Cat Miller from Mortar Data offers a quick intro to Pig Latin from a SQLish perspective:

Pig is similar enough to SQL to be familiar, but divergent enough to be disorienting to newcomers. The goal of this guide is to ease the friction in adding Pig to an existing SQL skillset.

Pig and SQL similarities are in the operations they both support. But the whole model is different. Pig is an imperative data manipulation tool, while SQL is a declarative query language.

Original title and link: A Brief Guide to Pig Latin for the SQL Guy (NoSQL database©myNoSQL)

via: http://hortonworks.com/blog/pig-eye-for-the-sql-guy/

Almost lost in the tons of Hadoopy releases, I have found the announcement of Apache Pig 0.11, which, as a serious open source project, packages nice new features for a point release:

1. DateTime data type
2. RANK, CUBE, ROLLUP operators
3. Groovy UDFs

Plus tons of improvements.

Original title and link: Apache Pig Goes 0.11 (NoSQL database©myNoSQL)

via: https://blogs.apache.org/pig/entry/apache_pig_it_goes_to

Costin Leau announcing the GA of Spring for Apache Hadoop:

What we have observed is that using the standard out of the box tools that come with Hadoop, you an easily end up with Hadoop applications that are poorly structured collection of command line utilities, scripts and pieces of code stiched together.

Leaving aside the jokes and that I don’t fully understand the purpose of this project (and here and here) , congrats for the release!

Original title and link: Spring for Apache Hadoop 1.0 Goes GA: Wrapping Hadoop in XML (NoSQL database©myNoSQL)

via: http://blog.springsource.org/2013/02/26/shdp-1-0-goes-ga/

Charles Zedlewki takes position for Cloudera to the recent attacks to Hadoop and Impala:

I’m reminded of our open source strategy this week not only because of the further validation of Hadoop’s popularity but also because of the entry of a new round of proprietary imitators. At one point there were six distinct vendors all promoting proprietary filesystems as alternatives to HDFS, many of which included breathless claims of how they could make Apache Hadoop faster and “more powerful.” This year we get to see history repeat itself, this time with SQL engines. The marketing is nearly identical to that of the proprietary filesystem era: damning open source with faint praise, pointing out its limitations and extolling the virtues of some feature(s) proprietary to that particular vendor.

Proprietary SQL vendors will pull a page from the proprietary storage playbook: damn open source Impala with faint praise and point out its limitations, both real and contrived. They will be equally ineffective. We will continue to bet on an open, integrated, and highly flexible big data platform. Saying you are “all in on Hadoop” while simultaneously promoting a proprietary platform means you are missing the point.

Neither Cloudera, nor other companies that invested a lot and everything in the Hadoop ecosystem are at the size not to care about large corporations attacking their bets. Every corporation is trying to emulate the Microsoft strategy: wait for a new technology to be confirmed, then jump at the opportunity with all your forces. But I really hope open source will prevail.

Original title and link: Cloudera Pissed Off (NoSQL database©myNoSQL)

via: http://blog.cloudera.com/blog/2013/02/open-source-flattery-and-the-platform-for-big-data/

I’m not implying that this question appeared on Quora after my link and comments about Redshift’s performance and costs at AirBnb, but Reynold Xin’s answer covers in a more formal way the reasons of Redshift being faster than Hive I’ve suggested in that post:

Some of the advantages you gain from massive scale and flexibility make it challenging to build a more performant query engine. The following outlines how various features (or lack of features) influences performance:

1. data format
2. task launch overhead (nb: this can be optimized in Hive/Hadoop)
3. intermediate data materialization vs pipelining
4. columnar data format
5. columnar query engine
6. faster S3 connection

Original title and link: What Makes Amazon Redshift Faster Than Hive? (NoSQL database©myNoSQL)

via: http://www.quora.com/Hive-computing/What-makes-Amazon-Redshift-faster-than-Hive