ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

Beyond Hadoop - Next-Generation Big Data Architectures

Bill McColl[1]:

Simple batch processing tools like MapReduce and Hadoop are just not powerful enough in any one of the dimensions of the big data space that really matters. Sure, Hadoop is great for simple batch processing tasks that are “embarrassingly parallel”, but most of the difficult big data tasks confronting companies today are much more complex than that. They can involve complex joins, ACID requirements, real-time requirements, supercomputing algorithms, graph computing, interactive analysis, or the need for continuous incremental updates. In each case, Hadoop is unable to provide anything close to the levels of performance required.

That’s to say there are problems where Hadoop or more generically mapreduce might not be the solution. MapReduce is fits problems that deal with data parallelism and not necessarily task parallelism. Most of the issues described above are related to processing deeply interconnected data.

McColl also writes in the article:

The only problem with this story is that the people who really do have cutting edge performance and scalability requirements today have already moved on from the Hadoop model.

Which are “all these companies”? The only reference I can think of is Google Caffeine, but that’s only one.


  1. Bill McColl: founder and CEO of Cloudscale Inc. His company is mentioned as one of the solutions.  ()

Original title and link: Beyond Hadoop - Next-Generation Big Data Architectures (NoSQL databases © myNoSQL)

via: http://www.nytimes.com/external/gigaom/2010/10/23/23gigaom-beyond-hadoop-next-generation-big-data-architectu-81730.html