ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

Hadoop, the Swap, and the OOM Killer

Stories from Spotify in Hadoop trenches:

Who and why could be a killer? The answer probably could be only one. The kernel out-of-memory killer that under desperately low memory conditions, starts murdering processes according to their “badness” score. It looks that the OOM killer takes out a Hadoop process (in this case TaskTracker). You can read how “badness” score is calculated here, but in case of “tradional” Hadoop slave servers, TaskTracker usually becomes the prime candidate to be killed, because together with its child processes (JVM running map and reduce tasks, and potentially an external scripts invoking map and reduce functions, if Hadoop Streaming is used), it consumes a lot of memory.

Original title and link: Hadoop, the Swap, and the OOM Killer (NoSQL database©myNoSQL)

via: http://hakunamapdata.com/two-memory-related-issues-on-the-apache-hadoop-cluster/