A post by Pandora FMS team about monitoring options for MongoDB:

If MongoDB goes wrong, all your apps will fail. So monitoring the main variables and configuration parameters of your database is the best option to make sure that your values are right and your users are happy.

The post talks about the tools that come with MongoDB (mongotop, mongostat, the stats available through the MongoDB shell, logfiles, etc.), but also introduces their PandoraFMS library. There’s no word about 10gen’s hosted MongoDB Monitoring Service, nor other MongoDB utilities for monitoring or the latest MongoMem collection memory usage library.

In terms of what are the most interesting stats, Simon Maynard’s 5 things to Monitor in MongoDB and these other 3 metrics should be a good start.

Original title and link: How to Monitor MongoDB (NoSQL database©myNoSQL)

via: http://blog.pandorafms.org/?p=821&buffer_share=0406c

Staying on the topic of monitoring, Jelle Smet writes about getting Hadoop metrics into Graphite using MetricFactory:

Using this setup we can accept Ganglia metrics over UDP from Hadoop, convert using multiple parallel processes the metrics to Graphite format in and submit the converted metrics in batches to Graphite. I’m planning to add more functionality to MetricFactory. Currently it can tackle mod_gearman and Ganglia data. Using the examples in this article you should be able to setup your own MetricFactory based setups relatively easy.

Original title and link: Using MetricFactory to Get Hadoop Metrics Into Graphite (NoSQL database©myNoSQL)

via: http://smetj.net/2013/01/10/using-metricfactory-to-get-hadoop-metrics-into-graphite/

In previous versions of Riak the same process gathered metrics and calculated statistics. In certain situations, under load, reading statistics would slow down, or even timeout altogether. The call to read stats would block that same process that updates stats, leading to large message queue backlogs.

This is the sort of observation and improvement that only a product that got into production (heavy production) could make.

Original title and link: Riak Metrics With Folsom (NoSQL database©myNoSQL)

via: http://basho.com/blog/technical/2012/07/02/folsom-backed-stats-riak-1-2/

This one is for ops, devops, noops and everyone else that loves some state-of-art real-systems behavior visualizations and investigation:

The subsecond offset heat map puts time on two axes. The x-axis shows the passage of time, with each column representing one second. The y-axis shows the time within a second, spanning from 0.0s to 1.0s (time offsets). The z-axis (color) show the count of samples or events, quantized into x- and y-axis ranges (“buckets”), with the color darkness reflecting the event count (darker == more).

Speechless

Original title and link: Visualizing Systems Behavior: Subsecond Offset Heat Maps (NoSQL database©myNoSQL)

via: http://dtrace.org/blogs/brendan/2012/03/26/subsecond-offset-heat-maps/

Using Netezza’s in-database analytics package FPGROWTH, database administrators can identify the most commonly used combination of tables and the performance of the queries that reference those sets of tables.

Nice feature. Sort of the rich men’s all-included slow query log in MySQL. Do you know if other databases support a similar feature?

Original title and link: Netezza Query History Table (NoSQL database©myNoSQL)

via: http://netezzaadmin.wordpress.com/2012/03/16/ibm-netezza-analytics-to-analyze-query-history-table-usage/

While some may learn a few new things or get a confirmation in the very details of the outage, what caught my attention in the Heroku’s postmortem analysis is the conclusions:

• higher sensitivity and more aggressive monitoring on a variety of metrics
• improved early warning systems
• better containment
• improved flow controls, both manual and automatic
• expanding simulations of unusual load conditions in our staging environment

None of these are particular to a specific storage or NoSQL database. But they all reflect the reality of operating at large scale where even the most operationally friendly solutions—think of Dynamo-inspired NoSQL databases—cannot and should not be left unmonitored or unsupervised or with no clear recovery strategies and processes in place.

In the NoSQL world, one of the most covered outages was the MongoDB outage at Foursquare. And in case you don’t remember the details, most of the circumstances that led to that event could have been prevented by having:

1. better monitoring
2. early warnings
3. better operational procedures

Aren’t these two lists looking very alike?

Original title and link: What Can Be Learned From Heroku Outage Postmortem (NoSQL database©myNoSQL)

via: https://status.heroku.com/incident/308

Starting today customers can view graphs of 23 job flow metrics within the EMR Console by selecting the Monitoring tab in the Job Flow Details page. These metrics are pushed CloudWatch every five minutes at no cost to you and include information on:

• Job flow progress including metrics on the number of map and reduce tasks running and remaining in your job flow and the number of bytes read and written to S3 and HDFS.
• Job flow contention including metrics on HDFS utilization, map and reduce slots open, jobs running, and the ratio between map tasks remaining and map slots.
• Job flow health including metrics on whether your job flow is idle, if there are missing data blocks, and if there are any dead nodes.

That’s like free pr0n for operations teams.

On a different note, I’ve noticed that the Hadoop stack (Hadoop, Hive, Pig) on Amazon Elastic MapReduce is based on second to last versions, which says that extensive testing is performed on Amazon side before rolling new versions out:

• Hadoop: 0.20.205 precursor of Hadoop 1.0.0 supports append and security, but doesn’t have RAID, symlinks or MR2
• Hive: 0.7.1 (precursor of latest 0.8.0)
• Pig: 0.9.1 (precursor of latest 0.9.2)

Original title and link: Amazon Elastic MapReduce New Features: Metrics, Updates, VPC, and Cluster Compute Support (NoSQL database©myNoSQL)

via: http://aws.typepad.com/aws/2012/01/new-elastic-mapreduce-features-metrics-updates-vpc-and-cluster-compute-support-guest-post.html

Denish Patel:

It turns out you can plug all the critical Riak Stats metrics into Circonus^[1] with no effort and very little time. […] I could add all the required checks for Riak Database server under 5 minutes into Circonus!!

It is Riak to be praised here for publishing useful stats that can make an admin feel happy and in control.

Original title and link: Monitoring Riak Using Circonus (NoSQL database©myNoSQL)

via: http://slowquery.blogspot.com/2011/07/monitoring-riak-using-circonus.html

A hosted monitoring tool for Redis:

Radish works side-by-side with your existing Redis instances in your hosting environment. All you need to do is start a daemon which handles connecting to Redis and sending data out to Radish over HTTPS.

Using Radish, you’ll be able to:

• analyze the amount of reads/writes
• track information like “average commands per second, memory usage, changes since last save, and keyspace size.” (here.)
• identify high-frequency commands and keys

Even if it looks like a very well polished product, the Hacker News community was prompt to point out that monitoring Redis using Munin or Cacti is cheaper.

Original title and link: Radish: Redis Analysis and Monitoring Service (NoSQL databases © myNoSQL)

via: http://robots.thoughtbot.com/post/6037147900/radish-dig-deep-into-redis