Ben Curtis ☞ thinks that using Redis for managing friends list as described in the ☞ EngineYard post is overly complicated:

Yesterday I read a post over at the EngineYard blog about a use case for Redis (in the name of being a polyglot, trying new things, etc.), and I just had to scratch my head. I love Redis — it rocks my world — but that example was too much for me. If you just want to store a set of ids somewhere to avoid normalization headaches, introducing Redis is overkill… just do it in MySQL!

He goes on and proposes a MySQL solution in which friends IDs are serialized as a comma separated list. Frankly speaking, I do see quite a few advantages Redis has compared to this one:

1. Redis knows how to handle sets
   1. you don’t have to deal with de-duplication
   2. (most probably) the storage is optimized
2. with manual serialization you’ll have to deal with all concurrency issues occurring when updating these lists

So what is the advantage of Ben’s suggested solution?

Original title and link for this post: Too Much Redis? (published on the NoSQL blog: myNoSQL)

You probably already know by now that Redis is that super fast in-memory (disk persistency being available through snapshots or append only mode) key-value store providing smart data types. In the past we’ve also talked about Redis virtual memory as a solution to improve memory efficiency.

Rich Schumacher and Will Larson started an experiment to better understand the memory costs of Redis datastructures. The testing methodology and all results have been published ☞ here, but you can also see the data summarized below:

In his last ☞ Redis update, Salvatore Sanfilippo explains why hashes are more memory efficient:

If you store an N (with N small) fields object as independent keys (one field per key), or the same object as an hash with five fields, the latter will require in the average five times less memory. No magic involved, simply hashes (and lists and sets in 2.2) are designed to be encoded in a much more efficient fashion when they are composed of a few elements.

In practice instead of using a real hash table, with all the overhead involved (the hash table itself, more pointers, redis objects for every field name and value) we store it as a single “blob” with prefixed-length strings. The speed will not suffer as we do this trick only for small fields and small number of elements and there is more cache locality, but the memory reduction is impressive.

Salvatore goes on and exposes some plans for improving Redis memory consumption for the complete Redis key space:

What is the memory gain here? It is simply huge!

• Space needed to store 10 million keys as normal keys: 1.7 GB (RSS)
• Space needed to store 10 million keys with this trick: 300 MB (RSS)

Note that the dump file size is 200MB, so we are very near to the theoretical overhead limits.

In the future we’ll make this transparent changing the key space implementation to do this tricks automatically, but I guess client library hackers may want to implement a similar interface already. Note that since the hashes also provide the HINCRBY command it’s possible to have tons of atomic counters using very little memory.

Interesting days ahead for Redis.

Original title and link for this post: Redis Memory Usage (published on the NoSQL blog: myNoSQL)

Gleicon Moraes’ slide deck about RestMQ, an HTTP/Redis based message queue. More about RestMQ can be found ☞ here and the source code is available on ☞ GitHub.

Keep in mind that Redis-backed queues is one very often cited use case for Redis.

Original title and link for this post: Presentation: RestMQ - HTTP/Redis based Message Queue (published on the NoSQL blog: myNoSQL)

Salvatore Sanfilippo (@antirez), creator and main developer of Redis, has shared a new project named BIGDIS

Bigdis is a weekend experiment about writing a Redis “clone” implementing a very strict subset of Redis commands (plain key-value basically) and using the file system as back end, that is: every key is represented as a file.

What is the goal of such a monster you may ask? Short answer: storing very large values.

Many kind of DBs are not well suited for storing large “files” as values. I mean things like images, or videos, and so forth. Still in the web-scale era it is very convenient to be able to access this kind of objects in a distributed fashion, with a networking layer, possibly with a protocol that contains already a large number of tested implementations.

While the goal is clearly stated in the above description, I’m not very sure in what scenarios is this new tool considering. For example, what are the advantages of using such a tool instead of say Amazon S3?

Another thing worth pointing is that BIGDIS seems to go the opposite direction of filesystem interfaces to NoSQL databases. BIGDIS proposes a simplified Redis API on top of the FS, while the later aim to provide the FS interface on top of NoSQL solutions.

Jeremy Zawodny (of craigslist.com) has published two articles, ☞ here and ☞ here, sharing his experiments on measuring the recently released Redis 2.0.0 RC3 storage overhead for two scenarios:

• simple key-values
• hashes, a new data type that will be available with the upcoming Redis 2.0

The experiment is interesting as it shares the code used and so you’ll be able to run it for your particular scenarios. Do keep in mind that the results will vary as they depend heavily on the size of the stored values.

This tells me that on a 32GB box, it’s not unreasonable to host 200,000,000 keys (if their values are sufficiently small). […] The resulting dump file (dump-0.rdb) was 1.8GB in size.

[…]

If you do the math, that yields 1.25 billion (1,250,000,000) key/value pairs stored. […] So it took about 2 hours and 40 minutes to complete. The resulting dump file (.rdb file) was 13GB in size (compared to the previous 1.8GB) and the memory usage was roughly 17GB.

Salvatore Sanfilippo (@antirez), Redis creator and main developer, has a good explanation about the storage overhead:

If you turn a txt file with a list of “common surnames -> percentage of population” into a binary tree it will get more or less an order of magnitude bigger in memory compared to the raw txt file.

This is a common pattern: when you add a lot of metadata, for fast access, memory management, “zero-copy” transmission of this information, expires, …, the size is not going to be the one of concatenating all this data like in a unique string.

[…]

But for now our reasoning is: it’s not bad to be able to store 1 million of keys with less than 200 MB of memory (100 MB on 32bit systems) if an entry level box is able to serve this data at the rate of 100k requests/second, including the networking overhead. And with hashes we have a much better memory performance compared to top level keys. So… with a few GB our users can store ten or hundred of millions of stuff in a Redis server.

— ☞ Hacker News