mapreduce: All content about mapreduce in NoSQL databases and polyglot persistence
In my post about in-memory databases vs Aster Data and Greenplum vs Hadoop market share, I’ve proposed a scenario in which Aster Data and Greenplum could expand into the space of in-memory databases by employing hybrid storage.
What I haven’t covered in that post is the possibility of Hadoop, actually HDFS, expanding into hybrid storage.
But that’s happening already and Hortonworks is already working on introducing support for heterogeneous storages in HDFS:
We plan to introduce the idea of Storage Preferences for files. A Storage Preference is a hint to HDFS specifying how the application would like block replicas for the given file to be placed. Initially the Storage Preference will include:
- The desired number of file replicas (also called the replication factor) and;
- The target storage type for the replicas.
Even if the costs of memory will continue to decrease at the same rate as before 2012, when they flat-lined, a cost effective architecture will almost always rely on hybrid storage.
Original title and link: Heterogeneous storages in HDFS ( ©myNoSQL)
Donald Miner, author of MapReduce Design Patterns and CTO at ClearEdge IT Solutions discusses how he chooses between Pig and MapReduce, considering developer and processing time, maintainability and deployment, and repurposing engineers that are new to Java and Pig.
Video and slides after the break.
If you are running out of interesting projects to experiment with during this seasonal break, Parkour is a Clojure library for writing MapReduce jobs.
Parkour is our new Clojure library that carries this philosophy to the Apache Hadoop’s MapReduce platform. Instead of hiding the underlying MapReduce model behind new framework abstractions, Parkour exposes that model with a clear, direct interface. Everything possible in raw Java MapReduce is possible with Parkour, but usually with a fraction of the code.
Original title and link: Parkour - Idiomatic Clojure for Map Reduce ( ©myNoSQL)