NoSQL @ QCon London

  1. ☞ QCon London is hosting a full day track on NoSQL. Presentations in the track:
    • Geir Magnusson: Project Voldemort at Gilt Groupe: When Failure isn’t An Option
    • Enda Farrell: Auntie on the Couch
    • Emil Eifrém: Not Only SQL: Alternative Data Persistence and Neo4J
    • Lars George & Fabrizio Schmidt: Social networks and the Richness of Data: Getting distributed webservices done with NoSQL
    • Mark Ramm: MongoDB: huMONGOus Data at SourceForge

    The track was recorded and InfoQ will publish the presentations in the upcoming month. As an example of what I’m talking about you can watch ☞ Facebook’s Petabyte Scale Data Warehouse using Hive and Hadoop.

Priority Queue with Barbershop and Redis

We have already talked about queues being a good Redis usecase and covered QR, Resque and thanks to readers heard of RestMQ.

Now we also have Babershop:

Barbershop is a simple priority queue daemon written in C using libevent and some well-crafted indexes and reverse indexes. With Barbershop, your created jobs’ ids are injected into Barbershop and the clients then query Barbershop for the next task/job to perform. You get the power that is Redis to scale tasks/jobs horizontally and the ability to increment and peak into a priority queue to adjust your application as needed.

Interesting ☞ comment from Salvatore Sanfilippo (@antirez)

Btw IMHO the way to go for Redis priority queues is mapping the priority to a discrete number of Redis lists instead going continuous

MySQL and MongoDB Sitting In a Boat

An interesting post from lunar logic guys about using MySQL and MongoDB for their Kanban product, how that get there and the tools they are using.

As a personal note, I thought how this system would be characterized in terms of CAP. It should be quite clear that we cannot speak about consistency over the two systems as MongoDB doesn’t really support transactions (you can check these notes on MongoDB for more details). So, in case their system would be using master-master MySQL replication and replica-pairs for MongoDB, and the internal tools would know how to work with this setup, we could probably say that we have an AP system. But if any of these preconditions are not fulfilled, I’d say both A and P are lost.

Using Map/Reduce for Network Forensics and Troubleshooting

Interesting technology stack behind ☞ xtractr — a network packet analysis tool - combining Ferret, SQLLite, V8, jQuery, Flot, Sammy and CouchDB. According to the linked article, xtractr uses a CouchDB inspired MapReduce for performing packet analysis. I’d say that if you could use it, PIG would have been nice too.

One of the huge challenges in packet forensics is that packets have incredibly rich information content and they come at many different layers each of which might be interesting on its own. Now, we didn’t want to build crazy SQL joins (I’m personally JOIN-challenged) across 90,000+ Wireshark fields. So we ended up using Map/Reduce very much inspired by CouchDB.

NoSQL Ecosystem News 2010-03-09

  1. A Redis monitor for ZenPack (Open Source Network Monitoring and Systems Management)
  2. Just a quick reminder that today there will be an ad-hoc NoSQL & beers meetup at Bag O’Nails at 7pm. ☞ See the place on the map.
  3. Found this amazing poster on ☞ DarkGreyIndustries blog and I was wondering if we could try to plot each NoSQL project on it. What do you think?