Gwen Shapira has a very interesting theory about what led Intel of creating its own distribution of Hadoop:

Intel is doing for Hadoop the same thing it did for C compilers – make sure they use the best hardware enhancements available in the CPUs and other hardware components available from Intel. The nice thing is that the enhancements are available as open source – Intel doesn’t care that the software is free, since they are selling the hardware!

But I don’t believe it.

1. Intel is already using Hadoop internally. They can test the hell out of Hadoop and create improvements.
2. As far as I know, Intel already has Hadoop committers that could push these improvements out. Even if Intel wouldn’t have Hadoop committers, I don’t think anyone from the Hadoop community would veto patches providing better utilization of cluster resources.

On the other hand I don’t have an alternative theory¹. But I have a hypothesis that is related to the list of partners Intel signed when announcing their distribution².

Original title and link: Thoughts on Intel’s Hadoop distribution (NoSQL database©myNoSQL)

via: http://www.pythian.com/blog/intel-hadoop-distribution/

Elliott Clarke explains a bit the work that his doing in migrating the HBase metrics to Hadoop’s Metrics2 system:

As HBase’s metrics system grew organically, Hadoop developers were making a new version of the Metrics system called Metrics2. In HADOOP-6728 and subsequent JIRAs, a new version of the metrics system was created. This new subsystem has a new name space, different sinks, different sources, more features, and is more complete than the old metrics. When the Metrics2 system was completed, the old system (aka Metrics1) was deprecated. With all of these things in mind, it was time to update HBase’s metrics system so HBASE-4050 was started. I also wanted to clean up the implementation cruft that had accumulated.

The post is more about the specific implementation details than the wide range of metrics HBase already supports and how this new system would unify and allow extending it.

Original title and link: HBase migration to the new Hadoop Metrics2 system (NoSQL database©myNoSQL)

via: https://blogs.apache.org/hbase/entry/migration_to_the_new_metrics

Christian Prokopp explaining the advantages of the RCFile storage:

The state-of-the-art solution for Hive is the RCFile. The format has been co-developed by Facebook, which is running the largest Hadoop and Hive installation in the world. RCFile has been adopted by the Hive and Pig projects as the core format for table like data storage. The goal of the format development was “(1) fast data loading, (2) fast query processing, (3) highly efficient storage space utilization, and (4) strong adaptivity to highly dynamic workload patterns,” as can be seen in this PDF from the development teams.

Questions:

1. is there any connection between the RCFile and Parquet the new columnar storage format? At first glance, the goals of the two are pretty similar.
2. It looks like there’s already a new format that will supersede RCFile: ORC Files. Are all these 3 approaches independent of each other? If yes, then would are the pros and cons of each of them?

Original title and link: RCFile - OCFile - Parquet: Storing Big Data With Hive (NoSQL database©myNoSQL)

via: http://www.bigdatarepublic.com/author.asp?section_id=2840&doc_id=262756

Stories from Spotify in Hadoop trenches:

Who and why could be a killer? The answer probably could be only one. The kernel out-of-memory killer that under desperately low memory conditions, starts murdering processes according to their “badness” score. It looks that the OOM killer takes out a Hadoop process (in this case TaskTracker). You can read how “badness” score is calculated here, but in case of “tradional” Hadoop slave servers, TaskTracker usually becomes the prime candidate to be killed, because together with its child processes (JVM running map and reduce tasks, and potentially an external scripts invoking map and reduce functions, if Hadoop Streaming is used), it consumes a lot of memory.

Original title and link: Hadoop, the Swap, and the OOM Killer (NoSQL database©myNoSQL)

via: http://hakunamapdata.com/two-memory-related-issues-on-the-apache-hadoop-cluster/

The project founder, Roman Shaposhnik defining what is Apache Bigtop:

The elevator pitch for Bigtop has always been: Bigtop is to Hadoop what Debian is to Linux. The most surprising development to me was how well that message resonates with the commercial vendors in the Big Data space. I’m still amazed at how quickly the “Powered by Bigtop” list is growing.

Original title and link: What is Apache Bigtop? (NoSQL database©myNoSQL)

via: http://blog.cloudera.com/blog/2013/05/meet-the-project-founder-roman-shaposhnik/

Nokia’s big data ecosystem consists of a centralized, petabyte-scale Hadoop cluster that is interconnected with a 100-TB Teradata enterprise data warehouse (EDW), numerous Oracle and MySQL data marts, and visualization technologies that allow Nokia’s 60,000+ users around the world tap into the massive data store. Multi-structured data is constantly being streamed into Hadoop from the relational systems, and hundreds of thousands of Scribe processes run every day to move data from, for example, servers in Singapore to a Hadoop cluster in the UK. Nokia is also a big user of Apache Sqoop and Apache HBase.

In the coming years you’ll hear more often stories—sales pitches—about single unified platforms solving all these problems at once. But platforms that will survive and thrive are those that will accomplish two things:

1. keep the data gates open: in and out.
2. work with different other platform to make this efficiently for users

Original title and link: Nokia’s Big Data Ecosystem: Hadoop, Teradata, Oracle, MySQL (NoSQL database©myNoSQL)

via: http://blog.cloudera.com/blog/2013/04/customer-spotlight-nokias-big-data-ecosystem-connects-cloudera-teradata-oracle-and-others/

To deploy, configure, manage and scale Hadoop clusters in a way that optimizes performance and resource utilization there is a lot to consider.

The 6 aspects presented in the post: OS, MapReduce slots available across nodes, memory, storage, capacity, network. It would be a lot more useful to put these in some order based on the scenarios the Hadoop cluster will have to solve.

Original title and link: 6 Key Hardware Considerations for Deploying Hadoop in Your Environment (NoSQL database©myNoSQL)

via: http://hortonworks.com/blog/6-key-hardware-considerations-for-deploying-hadoop-in-your-environment/

But the eWeek article demonstrates that the same concerns [nb: about security] exist where Hadoop implementations are concerned. The article says: “It [Hadoop] was not written to support hardened security, compliance, encryption, policy enablement and risk management.”

The story goes like this: in the early days of NoSQL, when no NoSQL database had any sort of security features, people behind the projects answered: “it’s too early. we’re focusing on more important features. and you can still get around security by placing your database behind firewalls”. Today, when more and more NoSQL databases are adding security features, the story these same people are telling is quite different: “ohhh, security is critical. we don’t really see how you could run a database without these features”.

Security is always critical. And exactly the same can be said about maintaining a solid, coherent story of what you are telling your users.

Original title and link: Hadoop, Security, and DataStax Enterprise (NoSQL database©myNoSQL)

via: http://www.datastax.com/2013/04/hadoop-security-and-the-enterprise

Robin Wauters for TNW:

Seattle startup GraphLab claims it is building the “fastest machine-learning analytics engine for graph datasets”, based on the popular open-source distributed graph computation framework with the same name, and it has just raised capital to come through on its promise.

Good luck to GraphLab’s team.

✚ Here’s a short list of MapReduce implementations for graphs.

Original title and link: Hadoop for graphs - GraphLab picks up $6.75m from Madrona and NEA (NoSQL database©myNoSQL)

via: http://thenextweb.com/insider/2013/05/14/graphlab-funding/

Robert Novak’s proposes Moh’s law and Rob’s corollaries to Hadoop and Big Data:

1. Hadoop is hard.
2. Make sure your’re measuring what you think you’re measuring.
3. Make sure you’re measuring what you need to be measuring.

For the first, I’m somehow confident that Cloudera and Hortonworks and others will finally solve it over time. But for the latter you are the only responsible. Not even a SaaS can save you.

Original title and link: Hadoop, Moh’s Law and Corollaries (NoSQL database©myNoSQL)

via: https://rsts11.wordpress.com/2013/05/14/mohs-law-and-big-data-rsts11/