ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

Instagram: All content tagged as Instagram in NoSQL databases and polyglot persistence

Instagram Impressed by Redis Hashes Space Efficiency

The size difference was pretty striking; with our 1,000,000 key prototype (encoded into 1,000 hashes of 1,000 sub-keys each), Redis only needs 16MB to store the information. Expanding to 300 million keys, the total is just under 5GB—which in fact, even fits in the much cheaper m1.large instance type on Amazon, about 1/3 of the cost of the larger instance we would have needed otherwise. Best of all, lookups in hashes are still O(1), making them very quick.

Last year, there were a couple of articles—Jeremy Zawodny (of craigslist.com): Measuring Redis Storage Overhead and Rich Schumacher and Will Larson: Redis Memory Usage—that took a look at Redis data structure space requirements. Knowing Salvatore’s and Peter’s obsession with space efficiency—which makes a lot of sense considering Redis is mostly an in-memory NoSQL database—things have probably changed since then, but the posts I’ve mentioned already feature some code that could be used to test space requirements with Redis’ latest versions.

Original title and link: Instagram Impressed by Redis Hashes Space Efficiency (NoSQL database©myNoSQL)

via: http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value-pairs