Steve Loughran covering the pro and con arguments of running Hadoop in a cloud environment:

1. If your data is stored in a cloud provider’s storage infrastructure, doing the analysis locally is the only rational action. It’s that “work near the data” philosophy.
2. If you are only doing some computation -say nightly- then you can rent some cluster time. Even if compute performance is worse, you can just rent some more machines to compensate.
3. You may be able to achieve better security through isolation of clusters (depends on your IaaS vendor’s abilities).
4. No upfront capex; fund from ongoing revenue.
5. Easier to expand your cluster; no need to buy more racks, find more rack space.
6. You don’t need to care about the problems of networking.
7. Less of a problem of heterogenous clusters if you expand later.

Interestingly the list of counter-arguments is much shorter and the important bit, further detailed in the post, is: “Hadoop contains lots of assumptions about running in a static infrastructure; it’s scheduling and recovery algorithms assume this.”

Original title and link: Hadoop in the Cloud: Pros and Cons (NoSQL database©myNoSQL)

via: http://steveloughran.blogspot.com/2012/03/hadoop-in-cloud-infrastructures.html

Sometimes the only valid comment is wow.

We have now launched a cluster 3 times the size of Tanuki, or 30,000 cores, which cost $1279/hour to operate for a Top 5 Pharma. It performed genuine scientific work — in this case molecular modeling — and a ton of it. The complexity of this environment did not necessarily scale linearly with the cores.

In fact, we had to implement a triad of features within CycleCloud to make it a reality:

1. MultiRegion support: To achieve the mind boggling core count of this cluster, we launched in three distinct AWS regions simultaneously, including Europe.
2. Massive Spot instance support: This was a requirement given the potential savings at this scale by going through the spot market. Besides, our scheduling environment and the workload had no issues with the possibility of early termination and rescheduling.
3. Massive CycleServer monitoring & Grill GUI app for Chef monitoring: There is no way that any mere human could keep track of all of the moving parts on a cluster of this scale.

Facebook runs a 30PB Hadoop analytic data warehouse and Yahoo! has a 100,000 cores/40,000 machines Hadoop cluster. I’m wondering what are the largest Amazon Elastic MapReduce jobs ever run. Any ideas?

Original title and link: Tanuki: A 30000 Cores AWS Cluster (NoSQL database©myNoSQL)

via: http://blog.cyclecomputing.com/2011/09/new-cyclecloud-cluster-is-a-triple-threat-30000-cores-massive-spot-instances-grill-chef-monitoring-g.html

I agree this title is misleading but problem is clear: today Oracle does not provide any product can compete with new cloud computing needs and with the NoSQL movement. It is not possibile to think that actually the RAC technology of oracle can be used in a cloud environment and also a cloud service cannot be deployed over an Exadata.

The real question though is if Oracle is really interested by the market currently served by NoSQL databases and/or hybrid solutions. And judging by the latest versions of MySQL and MySQL Cluster^[1] it looks like they are testing the waters.

Original title and link: Will Oracle Win the NoSQL Competition (NoSQL database©myNoSQL)

via: http://www.stefanocislaghi.eu/2011/09/will-oracle-win-the-nosql-competition/

Using MongoDB replicate sets:

At Famigo, we house all of our valuable data in MongoDB and we also serve all requests from Amazon EC2 instances. We’ve devoted many mental CPU cycles to finding the right architecture for our data in the cloud, focusing on 3 main factors: cost, reliability, and performance.

Original title and link: Reliable, Scalable, and Kinda Sorta Cheap: A Cloud Hosting Architecture for MongoDB (NoSQL database©myNoSQL)

via: http://www.codypowell.com/taods/2011/08/a-cloud-hosting-architecture-for-mongodb.html

Chris Marsh:

Cloud storage can be an attractive means of outsourcing the day-to-day management of data, but ultimately the responsibility and liability for that data falls on the company that owns the data, not the hosting provider. With this in mind, it is important to understand some of the causes of data corruption, how much responsibility a cloud service provider holds, some basic best practices for utilizing cloud storage safely, and some methods and standards for monitoring the integrity of data regardless of whether that data resides locally or in the cloud.

This reminded me of how the Adobe SaaS Infrastructure Team has tested HBase.

Original title and link: Data Integrity in the Cloud (NoSQL database©myNoSQL)

via: http://www.wwpi.com/index.php?option=com_content&view=article&id=12800:data-integrity-in-the-cloud&catid=99:cover-story&Itemid=2701018

Big processors or little processors, scale-up or scale-out, on-premise or in the cloud […] The plethora of choices for application architecture and delivery model are great if you like variety, but I don’t envy anyone tasked with choosing which system on which to spend their limited budget dollars.

Too little options is bad^[1]. Too many options are paralizing^[2]. Then what’s the solution? I think the only answer is to build experience. By trying, failing, learning, and sharing with everyone else.

Original title and link: The Server Architecture Debate Rages On (NoSQL database©myNoSQL)

via: http://gigaom.com/cloud/the-server-architecture-debate-rages-on/