A Guide to Elastic MapReduce and Hadoop Streaming for Astrophysicists
Arfon Smith1:
A couple of months ago I wrote about how the astrophysics community should place more value on those individuals building tools for their community - the informaticians. One example of a tool that I don’t think is particularly well known in many areas of research is the Apache Hadoop software framework.
Hadoop is a great tool but it can be fiddly to configure. With Elastic MapReduce you can focus on the design of your map/reduce workflow rather than figuring out how to get your cluster setup. Next I’m planning on making some small changes to software used by radio astronomers to find astrophysical sources in data cubes of the sky to make it work with Hadoop Streaming - bring it on SKA!
Clearly Hadoop has issues. Meanwhile it helps local communities to plan for snow removal, geophysicists find oil in the oceans, and who knows exactly how many other similar problematic implementations are out there.
-
Arfon Smith is Director of Citizen Science at The Adler Planetarium where I build citizen science projects for The Zooniverse ↩
Original title and link: A Guide to Elastic MapReduce and Hadoop Streaming for Astrophysicists (©myNoSQL)
via: http://arfon.org/getting-started-with-elastic-mapreduce-and-hadoop-streaming
