What’s the point in writing a new dbm style DBMS when so many exist already? Surely there is nothing new that anyone can add to this long line of projects?
Having done exactly that, I’m going to dedicate the rest of this post to convincing you that I haven’t lost my sanity.
I became interested in databases about six months ago when I started studying databases that spanned more than one machine. I found that it seems to be the accepted norm that if one ACID compliant database server can’t handle the number of writes that you need it to then you’ll have to loosen your reliability guarantees.
This can manifest in a number of ways:
Amongst the most frequent is manually sharding your database. Using two or more databases is a sure fire way to loose your atomicity and isolation guarantees for cross database operations.
Forsaking durability is common. It is the founding principle of database systems like Redis.
Settling for eventual consistency is the latest craze. CouchDB is heading the wave.
Of course, this isn’t always a bad thing. CouchDB, for instance, was designed for environments where partitioning is common and in these circumstances eventual consistency is a logical choice. ACID guarantees aren’t always a deal-breaker and when you can afford to give them up in order to gain performance, you definitely should.
However, I couldn’t find a database implementation that would allow me to scale without loosing multi-record atomic updates, durability or immediate consistency. I set about designing my perfect DBMS.
Following several false starts, I settled upon a design. AODBM (Append Only Database Manager) is the storage backend for this future DBMS. However, AODBM has merit on its own! It is a simple, hackable, open-source implementation of an append-only B+Tree written in C. It uses MVCC. It has an easy to use API (for C and Python) and it is designed to be fast for heavy, highly-concurrent loads.
In practice, this means that if you find a need to update a key-value store from many threads and you need ACID compliance, then I have a solution for you.
AODBM is currently in (early) beta and I am pushing for the first release. Early performance figures look promising; on my five year old desktop I can do 8500 insertions per second on a single thread from Python. I have implemented get, set and delete; iteration is on it’s way. I’m hoping to release the first version by the end of this month.
I hope I’ve succeeded in persuading you that my sanity is still intact. Have fun.