Almost lost in the tons of Hadoopy releases, I have found the announcement of Apache Pig 0.11, which, as a serious open source project, packages nice new features for a point release:

1. DateTime data type
2. RANK, CUBE, ROLLUP operators
3. Groovy UDFs

Plus tons of improvements.

Original title and link: Apache Pig Goes 0.11 (NoSQL database©myNoSQL)

via: https://blogs.apache.org/pig/entry/apache_pig_it_goes_to

Costin Leau announcing the GA of Spring for Apache Hadoop:

What we have observed is that using the standard out of the box tools that come with Hadoop, you an easily end up with Hadoop applications that are poorly structured collection of command line utilities, scripts and pieces of code stiched together.

Leaving aside the jokes and that I don’t fully understand the purpose of this project (and here and here) , congrats for the release!

Original title and link: Spring for Apache Hadoop 1.0 Goes GA: Wrapping Hadoop in XML (NoSQL database©myNoSQL)

via: http://blog.springsource.org/2013/02/26/shdp-1-0-goes-ga/

Charles Zedlewki takes position for Cloudera to the recent attacks to Hadoop and Impala:

I’m reminded of our open source strategy this week not only because of the further validation of Hadoop’s popularity but also because of the entry of a new round of proprietary imitators. At one point there were six distinct vendors all promoting proprietary filesystems as alternatives to HDFS, many of which included breathless claims of how they could make Apache Hadoop faster and “more powerful.” This year we get to see history repeat itself, this time with SQL engines. The marketing is nearly identical to that of the proprietary filesystem era: damning open source with faint praise, pointing out its limitations and extolling the virtues of some feature(s) proprietary to that particular vendor.

Proprietary SQL vendors will pull a page from the proprietary storage playbook: damn open source Impala with faint praise and point out its limitations, both real and contrived. They will be equally ineffective. We will continue to bet on an open, integrated, and highly flexible big data platform. Saying you are “all in on Hadoop” while simultaneously promoting a proprietary platform means you are missing the point.

Neither Cloudera, nor other companies that invested a lot and everything in the Hadoop ecosystem are at the size not to care about large corporations attacking their bets. Every corporation is trying to emulate the Microsoft strategy: wait for a new technology to be confirmed, then jump at the opportunity with all your forces. But I really hope open source will prevail.

Original title and link: Cloudera Pissed Off (NoSQL database©myNoSQL)

via: http://blog.cloudera.com/blog/2013/02/open-source-flattery-and-the-platform-for-big-data/

I’m not implying that this question appeared on Quora after my link and comments about Redshift’s performance and costs at AirBnb, but Reynold Xin’s answer covers in a more formal way the reasons of Redshift being faster than Hive I’ve suggested in that post:

Some of the advantages you gain from massive scale and flexibility make it challenging to build a more performant query engine. The following outlines how various features (or lack of features) influences performance:

1. data format
2. task launch overhead (nb: this can be optimized in Hive/Hadoop)
3. intermediate data materialization vs pipelining
4. columnar data format
5. columnar query engine
6. faster S3 connection

Original title and link: What Makes Amazon Redshift Faster Than Hive? (NoSQL database©myNoSQL)

via: http://www.quora.com/Hive-computing/What-makes-Amazon-Redshift-faster-than-Hive

Tylor Arndt about Torbit’s “build-your-own-MapReduce”:

The final system begins with a web-service against which client systems interface. To ensure resiliency, an instance of the web- service runs on each cluster host. When a client request arrives the web-service creates a MapReduce job to fulfill client requests. The reducer function component of the MapReduce job runs within the web- service handling the request.

The requirements listed in the post are too high level to understand why building their own solutions was better. But if it works for them, that’s OK. Just keep in mind that NIH and distributed systems don’t always mix well.

Original title and link: Big Data at Torbit: Custom MapReduce-like System (NoSQL database©myNoSQL)

via: http://torbit.com/blog/2013/02/19/big-data-at-torbit/

Avik Dey (Intel) sent the announcement of the new open source project from Intel to the Hadoop mailing list:

As the Apache Hadoop ecosystem extends into new markets and sees new use cases with security and compliance challenges, the benefits of processing sensitive and legally protected data with Hadoop must be coupled with protection for private information that limits performance impact. Project Rhino is our open source effort to enhance the existing data protection capabilities of the Hadoop ecosystem to address these challenges, and contribute the code back to Apache.

Project Rhino targets security at all levels: from encryption and key management, cell level ACLs to audit logging.

Original title and link: Project Rhino: Enhanced Data Protection for the Apache Hadoop Ecosystem (NoSQL database©myNoSQL)

via: http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201302.mbox/%3cCD5137E5.15610%25avik.dey@intel.com%3e

The Mortar blog:

Mongo was built for data storage and retrieval, and Hadoop was written for data processing. So naturally, data processing is often better offloaded to Hadoop. Here’s why:

1. Easier, more expressive language
2. Libraries to build on
3. Big performance improvements
4. Separate workloads mean less load

For the how part, the post recommends their own Hadoop-as-a-Service platform and a set of libraries the Mortar platform provides.

✚ While browsing the Mortar blog and website I couldn’t find any information related to the costs of transferring data. The AWS services usually have a data transfer dimension, which most often has an important impact on the total costs of a solution.

Original title and link: Integrating MongoDB and Hadoop: Why & How (NoSQL database©myNoSQL)

via: http://blog.mortardata.com/post/43080668046/mongodb-hadoop-why-how

Jimmy Wong:

Which companies use Hadoop for analyzing big data? How big are their clusters? I thought it would be fun to compare companies by the size of their Hadoop installations. The size would indicate the company’s investment in Hadoop, and subsequently their appetite to buy big data products and services from vendors, as well as their hiring needs to support their analytics infrastructure.

Unfortunately the data available is sooo little and soooo old.

Original title and link: Which Big Data Company Has the World’s Biggest Hadoop Cluster? (NoSQL database©myNoSQL)

via: http://www.hadoopwizard.com/which-big-data-company-has-the-worlds-biggest-hadoop-cluster/

Doug Henshen for InformationWeek discussing the results of Infochimps’s survey “CIOs & Big Data: What Your IT Team Wants You to Know”:

What business problem are you trying to solve? If you could tell your IT employees what it is, they’d have a much better crack at big data success.

[…]

“Inaccurate scope” is cited by 58% as the top reason that big data IT projects fail. “Too many big data projects are structured like boil-the-ocean experiments”, Infochimps’ CEO, Jim Kaskade, told InformationWeek.

Some call these vague and unrealistic expectations the “trough of disillusionment”.

Original title and link: Vague Goals Seed Big Data Failures (NoSQL database©myNoSQL)

via: http://www.informationweek.com/software/information-management/vague-goals-seed-big-data-failures/240147135