NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter



FleetDB: An Interview with Mark McGranaghan

FleetDB is an MIT licensed schema-free database implemented primarily on Clojure that provides a combination of schema-free records, declarative queries, optimizing query planner and a few more interesting features[1]. While not exactly targeting those scenarios that involve tons of data and require massive scalability, FleetDB seems to be a nice tool to have around when prototyping your next app. Mark McGranaghan, the project creator, has been kind enough to answer a couple of questions for us.

MyNoSQL: What made you create FleetDB? Why FleetDB? What is its ‘selling point’?

Mark McGranaghan: FleetDB is a solution to the problems that I encountered when trying to use existing relational and NoSQL database to rapidly develop applications. In particular, FleetDB offers a unique combination of schema-free data modeling, expressive and composable queries, automatically maintained indexes, excellent consistency and concurrency characteristics, in-memory operation, simple append-only durability, and universal client API that cannot be found in existing databases.

FleetDB is also a great example of the power of functional programming in general and of Clojure’s persistent data structures in particular. Many of the features of FleetDB - its ACID guarantees, concurrent performance, and powerful query language in particular - are due largely to it having been implemented in Clojure.

MyNoSQL: Where would you position FleetDB? (according to CAP, data model, etc.)

Mark McGranaghan: FleetDB is a document-oriented database. It also offers dynamic queries that are aided by indexes and an optimizing query planner. FleetDB currently operates as a single process and is therefore not subject to CAP. Indeed the database provide uncommonly strong consistency guarantees; multi-document, multi-collection, and even multi-query. In terms of memory versus disk, FleetDB answers all queries out of memory but keeps an append-only log on disk for full durability.

In comparison to other databases, FleetDB combines the optimizing query planner of relational databases, the document orientation of MongoDB, the main memory operation of Redis, the functional data model of CouchDB, the embedability and single-file durability of SQLite, and adds an original composable query interface that allows for increased consistency guarantees and general expressiveness.

One thing that I am not trying to do with FleetDB, at least right now, is build a massively scalable database. A lot of apps have a relatively modest amount of core data, especially as they are being prototyped and iteratively developed. In these cases the ease-of-use, flexibility, consistency guarantees, and performance of a well-designed single node database may be more desirable than a fully distributed database and its associated complexity and decreased durability guarantees. With FleetDB I’ve tried get the single node use case right, were a lot of other NoSQL stores are great at massive scale but not much use for the single node case.

MyNoSQL: Are you aware of any usage of FleetDB in production?

Mark McGranaghan: I’ve used FleetDB for several personal projects and prototypes, though the only one of these that is public now is ☞ GitCred. I’ve also heard from a few startups that they are considering FleetDB for use in their applications. That said, I would be surprised to see public-facing production use while the product is still in alpha; at this point I’m still working with users to ensure that the interface is good, that performance is high, and that we catch any bugs. I hope to see more production use after an 0.1.0 release in the Spring.

MyNoSQL: Anything you’d like to add.

Mark McGranaghan: FleetDB is implemented in about 1300 lines of Clojure and 100 lines of Java, much less code than any of the other database systems that I have considered. I was able to keep the code base so small because of the expressiveness of the Clojure language, the power of its data persistent structures, the availability of a variety of Java libraries on which to build, and a judicious choice of features to implement in FleetDB. Having a small code base helps me with rapidly developing features and containing bugs, but its also nice for contributers and end-users who are curious about the internals of the database.

I’m also working on a database performance evaluation suite called ☞ db-compare. That project started as a means to test FleetDB as I was developing it, but it has since evolved into a tool to evaluate the performance of a dozen open source databases under a variety of workloads and client concurrency levels. Furthermore, the evaluations produced by the suite will be rigorous, repeatable, and properly statistically analyzed. I’ll be sure to ping you when I release the first set of db-compare benchmark results. In the meantime, if you have any thoughts about what you or the community would like to see from database benchmarks feel free to let me know.