Machine learning is an entire field devoted to Information Retrieval, Statistics, Linear Algebra, Analysis of Algorithms, and many other subjects. This field allows us to examine things such as recommendation engines involving new friends, love interests, and new products. We can do incredibly advanced analysis around genetic sequencing and examination, distributed search and frequency pattern matching, as well mathematical analysis with vectors, matrices, and singular value decomposition (SVD).

All these fields have deep connections in the big data space.

Original title and link: Why Is Cloudera Packing Mahout With Hadoop? (NoSQL database©myNoSQL)

via: http://www.cloudera.com/blog/2011/11/cdh3u2-apache-mahout-integration/

At a time when everyone is reading, writing, or talking about Steve Job’s biography, Wired has published a long article looking at the history of Hadoop (Yahoo-era), the Hortonworks spin-off, and Cloudera. While the article doesn’t cover the late rush into Hadoop world by giants like Oracle, IBM, EMC, and others which all want a piece, it gives an interesting overview of the Hadoop ecosystem dynamics:

The initial result is an amusingly heated rivalry between Cloudera and Hortonworks — the kind of rivalry you only see in the open source world. […] But ultimately, this Hadoop civil war shows just how vibrant the platform is.

“Additional investment in the platform and more people concentrating on the open source distro is good for community and good for Cloudera,” Olson says. It’s the sort of thing you always hear from a competitor when a new company enters a market. But in this case, there’s a truth to it. Bearden and Baldeschwieler’s efforts to expand the open source project can only help Cloudera — and the rest of the market.

Original title and link: Hadoop, Hortonworks, Cloudera: A Page of History (NoSQL database©myNoSQL)

via: http://www.wired.com/wiredenterprise/2011/10/how-yahoo-spawned-hadoop/all/1

Brian Smith (Datameer Regional Director of Sales):

DAS is an open book at every stage of the data pipeline, with plug and play support at each phase – integration, analysis and visualization. Under the covers, DAS generates Java/MapReduce code that runs natively on the Hadoop cluster. All current Hadoop distros are supported – we’re Switzerland when it comes to platform support for Apache, Cloudera, MapR, IBM and the rest, we run all of it in a browser on Windows, Mac and Linux.

As always I won’t comment on statements referring to “first” or “best”. But I find Brian Smith’s assessment of the Hadoop economics very accurate:

The economics are compelling — Hadoop is moving out costly analytic databases and warehouses, driving IT to re-look at ADBMS sales cycles, shifting IT dollars and vendor roadmaps, and generally wreaking havoc in the traditional vendor community. We’ve gone from one or two distributions to nine in the last year! And, literally every vendor in the BI/DBMS space has a Hadoop connector, the latest being the recent Oracle announcement. Everybody is on board this train — All this based upon the premise of unlimited scale and data variety at a fraction of traditional costs.  Technical challenges exist, but its clear that there’s a sea change.

Original title and link: Datameer Is the First BI/Analytics Platform Built Natively on Hadoop (NoSQL database©myNoSQL)

via: http://datameer.com/blog/uncategorized/why-i-am-at-datameer.html

Cloudera has created a set of tools named Hoop allowing access through HTTP/S to HDFS. My first question was why would you use HTTP to access HDFS? Here is the answer:

• Transfer data between clusters running different versions of Hadoop (thereby overcoming RPC versioning issues).
• Access data in a HDFS cluster behind a firewall. The Hoop server acts as a gateway and is the only system that is allowed to go through the firewall.

Not sure though how many will use HTTP for transfering large amounts of data. But if you want to see how it is implemented, you can find the source code on GitHub.

Original title and link: Hoop - Hadoop HDFS Over HTTP (NoSQL database©myNoSQL)

via: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/

I assume people on both sides of data warehouses (users and providers) are asking the same question. IBM Netezza and Cloudera seem to agree on the answer:

IBM Netezza had worked with Cloudera to put together a compelling demo to highlight the value of our combined solution of CDH/Hadoop and Netezza.  Through an interesting use case, the demo showed how businesses could have their “hot” data (most recent data) residing in Netezza, “warm” data (longer time range data) residing in HDFS, while leveraging the Cloudera Connector for Netezza and Oozie (workflow engine part of CDH) to provide deeper insights to business executives.

I would have liked to know more details about the use case though. Just categorizing data in “hot” and “warm” is not enough to understand the advantages of each piece.

Original title and link: Hadoop and IBM Netezza: Compete or Co-Exist? (NoSQL database©myNoSQL)

via: http://www.cloudera.com/blog/2011/06/reflections-from-enzee-universe-2011/

Curt Monash quoting Omer Trajman (Cloudera) in a post counting petabyte-scale Hadoop deployments:

The number of Petabyte+ Hadoop clusters expanded dramatically over the past year, with our recent count reaching 22 in production (in addition to the well-known clusters at Yahoo! and Facebook). Just as our poll back at Hadoop World 2010 showed the average cluster size at just over 60 nodes, today it tops 200. While mean is not the same as median (most clusters are under 30 nodes), there are some beefy ones pulling up that average. Outside of the well-known large clusters at Yahoo and Facebook, we count today 16 organizations running PB+ clusters running CDH across a diverse number of industries including online advertising, retail, government, financial services, online publishing, web analytics and academic research. We expect to see many more in the coming years, as Hadoop gets easier to use and more accessible to a wide variety of enterprise organizations.

First questions that bumped in my head after reading it:

1. How many deployments DataStax’ Brisk has? How many close or over petabyte?
2. How many clients run EMC Greenplum HD and how many are close to this scale?
3. Same question about NetApp Hadoopler clients.
4. Same question for MapR.

Answering these questions would give us a good overview of the Hadoop ecosystem.

Original title and link: Petabyte-Scale Hadoop Clusters (NoSQL database©myNoSQL)

via: http://www.dbms2.com/2011/07/06/petabyte-hadoop-clusters/

Many of our systems use Amazon’s S3 as a backup repository for log data.  Our data became too large to process by traditional techniques, so we started using Amazon’s Elastic MapReduce (EMR) to do more expensive queries on our data stored in S3.  The major advantage of EMR for us was the lack of operational overhead.  With a simple API call, we could have a 20 or 40 node cluster running to crunch our data, which we shutdown at the conclusion of the run. We had two systems interacting with EMR.  The first consisted of shell scripts to start an EMR cluster, run a pig script, and load the output data from S3 into our data warehousing system.  The second was a Java application that launched pig jobs on an EMR cluster via the Java API and consumed the data in S3 produced by EMR.

What might make you consider moving from the cloud version of MapReduce, the Amazon Elastic MapReduce, to an on-premise Hadoop cluster:

1. performance and tuning
2. monitoring
3. API access
4. lack of latest features

Original title and link: Moving Away From Amazon’s EMR Service to an In-House Hadoop Cluster (NoSQL database©myNoSQL)

via: http://www.cloudera.com/blog/2011/06/migrating-from-elastic-mapreduce-to-a-cloudera?s-distribution-including-apache-hadoop-cluster/

If you don’t count the download, you’ll get this up and running in 5 minutes tops. At the end you’ll have Hadoop, Sqoop, Pig, Hive, HBase, ZooKeeper, Oozie, Hume, Flume, and Whirr all configured and ready to experiment with.

Making it easy for users to experiment with these tools increases the chances for adoption. Adoption means business.

Original title and link: Experimenting with Hadoop using Cloudera VirtualBox Demo (NoSQL databases © myNoSQL)

via: http://www.cloudera.com/blog/2011/06/cloudera-distribution-including-apache-hadoop-3-demo-vm-installation-on-mac-os-x-using-virtualbox-cdh/

Moving Federal Gov analytics from MySQL to Hadoop and Hive:

HDFS offered us a distributed, resilient, and scalable filesystem while Hadoop promised to bring the work to where the data resided so we could make efficient use of local disk on multiple nodes. Hive, however, really pushed our decision in favor of a Hadoop-based system. Our data is just unstructured enough to make traditional RDBMS schemas a bit brittle and restrictive, but has enough structure to make a schema-less NoSQL system unnecessarily vague. Hive let us compromise between the two — it’s sort of a “SomeSQL” system.

Original title and link: Adopting Apache Hadoop and Hive (NoSQL databases © myNoSQL)

via: http://www.cloudera.com/blog/2011/04/adopting-apache-hadoop-in-the-federal-government/