To deploy, configure, manage and scale Hadoop clusters in a way that optimizes performance and resource utilization there is a lot to consider.

The 6 aspects presented in the post: OS, MapReduce slots available across nodes, memory, storage, capacity, network. It would be a lot more useful to put these in some order based on the scenarios the Hadoop cluster will have to solve.

Original title and link: 6 Key Hardware Considerations for Deploying Hadoop in Your Environment (NoSQL database©myNoSQL)

via: http://hortonworks.com/blog/6-key-hardware-considerations-for-deploying-hadoop-in-your-environment/

But the eWeek article demonstrates that the same concerns [nb: about security] exist where Hadoop implementations are concerned. The article says: “It [Hadoop] was not written to support hardened security, compliance, encryption, policy enablement and risk management.”

The story goes like this: in the early days of NoSQL, when no NoSQL database had any sort of security features, people behind the projects answered: “it’s too early. we’re focusing on more important features. and you can still get around security by placing your database behind firewalls”. Today, when more and more NoSQL databases are adding security features, the story these same people are telling is quite different: “ohhh, security is critical. we don’t really see how you could run a database without these features”.

Security is always critical. And exactly the same can be said about maintaining a solid, coherent story of what you are telling your users.

Original title and link: Hadoop, Security, and DataStax Enterprise (NoSQL database©myNoSQL)

via: http://www.datastax.com/2013/04/hadoop-security-and-the-enterprise

Robin Wauters for TNW:

Seattle startup GraphLab claims it is building the “fastest machine-learning analytics engine for graph datasets”, based on the popular open-source distributed graph computation framework with the same name, and it has just raised capital to come through on its promise.

Good luck to GraphLab’s team.

✚ Here’s a short list of MapReduce implementations for graphs.

Original title and link: Hadoop for graphs - GraphLab picks up $6.75m from Madrona and NEA (NoSQL database©myNoSQL)

via: http://thenextweb.com/insider/2013/05/14/graphlab-funding/

Robert Novak’s proposes Moh’s law and Rob’s corollaries to Hadoop and Big Data:

1. Hadoop is hard.
2. Make sure your’re measuring what you think you’re measuring.
3. Make sure you’re measuring what you need to be measuring.

For the first, I’m somehow confident that Cloudera and Hortonworks and others will finally solve it over time. But for the latter you are the only responsible. Not even a SaaS can save you.

Original title and link: Hadoop, Moh’s Law and Corollaries (NoSQL database©myNoSQL)

via: https://rsts11.wordpress.com/2013/05/14/mohs-law-and-big-data-rsts11/

This will open up Hadoop to a large number of organizations that have no in- house Linux skills. Shaun Connolly, vice president of Corporate Strategy at Hortonworks, explains the thinking behind moving HDP to Windows in this way: “Essentially it’s a market-driven decision,” he says. “Hadoop is built for the scaleout commodity hardware market, and the commodity hardware market is 70% Windows by install base and expertise.”

Employees in Windows-only companies will be able to make use of Hadoop easily because Excel can be used as a business intelligence tool to view the results of Hadoop Big Data analysis (whether Hadoop is running on Windows or Linux). “Ideally we want Microsoft users to be oblivious to the fact that everything is coming from Hadoop,” says Connolly. “If end users can consume data without any learning curve, thanks to tools like Excel, then they get more value.”

Either the data or the logic above is not sound:

1. those Windows machines that make up the 70% of the market are probably running Excel
2. those 70% of the market Windows machines are not going to run Hadoop

Based on this sort of market-share decisions, tomorrow we should see Hadoop for iOS and Android and Nokia. Sometime soon Microsoft will release Excel for iOS and maybe Android.

Original title and link: What Open Source Hadoop Coming to Windows Means to IT (NoSQL database©myNoSQL)

via: http://www.cio.com/article/733260/What_Open_Source_Hadoop_Coming_to_Windows_Means_to_IT

Darryl K. Taft reporting the experience of using Hadoop at UC Irvine Medical Center:

Because they were bleeding money, the team wanted a cost-effective solution. “Our target was $500 per terabyte. We were at $100,000 per terabyte with the old system,” Peterson said. “With our Hadoop cluster, we’re now at $900 per terabyte.”

How are these costs calculated?

1. Fixed costs: hardware, any one time licenses
2. Recurring costs: hardware replacement, energy, HR

Is this all?

Original title and link: Hadoop Drives Down Costs (NoSQL database©myNoSQL)

via: http://www.eweek.com/print/cloud/hadoop-drives-down-costs-drives-up-usability-with-sql-convergence/

Roberto V. Zicari interviewing Joe Russell¹ about Hadoop virtualization with Serengeti:

A common misconception when virtualizing Hadoop clusters is that we decouple the data nodes from the physical infrastructure. This is not necessarily true. When users virtualize a Hadoop cluster using Project Serengeti, they separate data from compute while preserving data locality. By preserving data locality, we ensure that performance isn’t negatively impacted, or essentially making the infrastructure appear as static. Additionally, it creates true multi-tenancy within more layers of the Hadoop stack, not just the name node.

I’m not 100% sure I get this, but the way I explained it to myself to actually make sense this would mean that HDFS lives directly on the physical hardware and only the compute part is virtualized. Is that what he means?

Original title and link: Hadoop Virtualization (NoSQL database©myNoSQL)

via: http://www.odbms.org/blog/2013/04/on-virtualize-hadoop-interview-with-joe-russell/

Timothy Prickett Morgan for The Register about Project Savanna, a collaboration between Mirantis, Hortonworks, and Red Hat:

Batman and Robin. Peanut butter and chocolate. OpenStack and Hadoop. These are things that go together, with the latter pairing being something that commercial OpenStack distie Mirantis, commercial Hadoop distie Hortonworks, and commercial KVM and Linux distie (and soon to be OpenStack commercializer) Red Hat are putting together under a new OpenStack effort dubbed Project Savanna.

Hadoop is at the age where everyone tries to package it and claim they’ll be the Red Hat of the Hadoop ecosystem. I cannot really dot the i-s and cross the t-s, but my gut feeling is that right now all these are actually more similar to the attempts of bringing Linux to the desktop.

We know how successful these have been so far.

Original title and link: Project Savanna: Hadoop and OpenStack (NoSQL database©myNoSQL)

via: http://www.theregister.co.uk/2013/04/18/project_savanna_hadoop_on_openstack/

Alex Williams for TechCrunch:

Boundary‘s application performance monitoring technology is now integrated into Splunk‘s enterprise platform, providing a window into apps that increasingly are distributed across cloud and on-premise virtualized environments.

At first I thought this means Boundary will use Splunk as the backend for the data. But Boundary is a service so that’s not the case. Plus Splunk can already be used for network management and monitoring.

According to the post, “Splunk real-time alerts are tagged as annotations in Boundary’s time-series graphs. Customers can then correlate alerts against application flow and performance data.” So basically this is monitoring your monitoring system, right?

Original title and link: Boundary for Splunk app for correlating alerts (NoSQL database©myNoSQL)

via: http://techcrunch.com/2013/04/25/new-boundary-app-for-splunk-predicts-root-cause-of-app-brownouts/

Venkatesh Seetharam announcing a new Apache incubating project in the Hadoop ecosystem open sourced by InMobi and Hortonworks:

Today we are excited to see another example of the power of community at work as we highlight the newly approved Apache Software Foundation incubator project named Falcon. This incubation project was initiated by the team at InMobi together with engineers from Hortonworks. Falcon is useful to anyone building apps on Hadoop as it simplifies data management through the introduction of a data lifecycle management framework.

I think this diagram describes Project Falcon best:

✚ Was there any other project addressing this space?

Original title and link: Project Falcon: Tackling Hadoop Data Lifecycle Management (NoSQL database©myNoSQL)

via: http://hortonworks.com/blog/project-falcon-tackling-hadoop-data-lifecycle-management-via-community-driven-open-source/