ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

A Guide to Elastic MapReduce and Hadoop Streaming for Astrophysicists

Arfon Smith1:

A couple of months ago I wrote about how the astrophysics community should place more value on those individuals building tools for their community - the informaticians. One example of a tool that I don’t think is particularly well known in many areas of research is the Apache Hadoop software framework.

Hadoop is a great tool but it can be fiddly to configure. With Elastic MapReduce you can focus on the design of your map/reduce workflow rather than figuring out how to get your cluster setup. Next I’m planning on making some small changes to software used by radio astronomers to find astrophysical sources in data cubes of the sky to make it work with Hadoop Streaming - bring it on SKA!

Clearly Hadoop has issues. Meanwhile it helps local communities to plan for snow removal, geophysicists find oil in the oceans, and who knows exactly how many other similar problematic implementations are out there.

Peter Skomoroch


  1. Arfon Smith is Director of Citizen Science at The Adler Planetarium where I build citizen science projects for The Zooniverse 

Original title and link: A Guide to Elastic MapReduce and Hadoop Streaming for Astrophysicists (NoSQL database©myNoSQL)

via: http://arfon.org/getting-started-with-elastic-mapreduce-and-hadoop-streaming