ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

redis: All content about redis in NoSQL databases and polyglot persistence

Storage technologies at HipChat - CouchDB, ElasticSearch, Redis, RDS

As per the list below, HipChat’s storage solution is based on a couple of different solutions:

  • Hosting: AWS EC2 East with 75 Instance currently all Ubuntu 12.04 LTS
  • Database: CouchDB currently for Chat History, transitioning to ElasticSearch. MySQL-RDS for everything else
  • Caching: Redis
  • Search: ElasticSearch
  1. This post made me wonder what led HipChat team to use CouchDB in the first place. I’m tempted to say that it was the master-master replication and the early integration with Lucene.
  2. This is only the 2nd time in quite a while I’m reading an article mentioning CouchDB — after the February “no-releases-but-we’re-still-merging-BigCouch” report for ASF. And according to the story, CouchDB is on the way out.

Original title and link: Storage technologies at HipChat - CouchDB, ElasticSearch, Redis, RDS (NoSQL database©myNoSQL)

via: http://highscalability.com/blog/2014/1/6/how-hipchat-stores-and-indexes-billions-of-messages-using-el.html


Some fun with Redis Cluster testing

Salvatore Sanfilippo:

One of the steps to reach the goal of providing a “testable” Redis Cluster experience to users within a few weeks, is some serious testing that goes over the usual “I’m running 3 nodes in my macbook, it works”. […] In order to perform some testing I assembled an environment like that:

  • Hardware: 6 real computers: 2 macbook pro, 2 macbook air, 1 Linux desktop, 1 Linux tiny laptop called EEEpc running with a single core at 800Mhz.

I’m still amazed by the lack of help and support Redis is getting from Pivotal, its official sponsor. And Pivotal wants to offer commercial support for Redis.

Original title and link: Some fun with Redis Cluster testing (NoSQL database©myNoSQL)

via: http://antirez.com/news/69


MySQL is a great Open Source project. How about open source NoSQL databases?

In a post titled Some myths on Open Source, the way I see it, Anders Karlsson writes about MySQL:

As far as code, adoption and reaching out to create an SQL-based RDBMS that anyone can afford, MySQL / MariaDB has been immensely successful. But as an Open Source project, something being developed together with the community where everyone work on their end with their skills to create a great combined piece of work, MySQL has failed. This is sad, but on the other hand I’m not so sure that it would have as much influence and as wide adoption if the project would have been a “clean” Open Source project.

The article offers a very black-and-white perspective on open source versus commercial code. But that’s not why I’m linking to it.

The above paragraph made me think about how many of the most popular open source NoSQL databases would die without the companies (or people) that created them.

Here’s my list: MongoDB, Riak, Neo4j, Redis, Couchbase, etc. And I could continue for quite a while considering how many there are out there: RavenDB, RethinkDB, Voldemort, Tokyo, Titan.

Actually if you reverse the question, the list would get extremely short: Cassandra, CouchDB (still struggling though), HBase. All these were at some point driven by community. Probably the only special case could be LevelDB.

✚ As a follow up to Anders Karlsson post, Robert Hodges posted The Scale-Out Blog: Why I Love Open Source.

Original title and link: MySQL is a great Open Source project. How about open source NoSQL databases? (NoSQL database©myNoSQL)

via: http://karlssonondatabases.blogspot.com/2014/01/some-myths-on-open-source-way-i-see-it.html


With the release of Redis 2.8, Pivotal starts offering support for Redis

Redis 2.8 was finally released on Nov. 22nd. And with this release, Pivotal starts offering support:

As of today, Pivotal will be offering 2 types of services for companies that have grown to depend on Redis and would like commercial support.

  • Developer Support. Priced per developer, this support is designed to support developers as they build out and support their own systems. Support is available for 12 hours a day, 5 days a week and is valid for one year.
  • Production Support. Priced per CPU, this support is the type of round-the-clock support that is needed for customer facing apps companies use to run their business.

Original title and link: With the release of Redis 2.8, Pivotal starts offering support for Redis (NoSQL database©myNoSQL)

via: http://blog.gopivotal.com/news-2/pivotal-offering-redis-support-redis-2-8-0-now-available


Redis Benchmark & Rackspace Performance VMs

This is quite interesting: using the benchmark of a NoSQL database (Redis), and not the one you already own and offer hosting for, to brag about the performance of your hosting servers:

I immediatley thought of using redis-benchmark to illustrate the difference between our (soon to be deprecated) standard flavors and our new performance flavors. Based on some reading on the redis-benchmark page, I decided to use the same critiera as the Redis Benchmarks page for evaluating bare-metal and virtual machines.

Original title and link: Redis Benchmark & Rackspace Performance VMs (NoSQL database©myNoSQL)

via: http://developer.rackspace.com/blog/redis-benchmark-and-rackspace-performance-vms.html


Building a Message Queue using Redis in Go

The team from adeven about their Redis-based message queue:

In order to be used in production at adeven we need the following features:

  • persistent (none to little data loss in case of a crash)
  • reliable, atomic (each message can only be fetched once)
  • monitorable (message rates, queue length and worker statuses need to be monitored through an API)
  • multi-message ack and reject (acknowledge multiple messages at once while rejecting them individually)

We do not need any complex broker system. Simple queues with multi publishers and multi consumers are the only thing we currently use. So if you want to have a fancy broker or exchange system this is probably not what you’re looking for.

By now this is already a well-known use case in the Redis community. Things can become more complicated if you need more than 1 Redis instance.

Original title and link: Building a Message Queue using Redis in Go (NoSQL database©myNoSQL)

via: http://big-elephants.com/2013-09/building-a-message-queue-using-redis-in-go/


Amazon Web Services support for Redis

Amazon decided to expand their ElastiCache service with support for Redis:

Today, we added two important choices for customers running high performance apps in the cloud: support for Redis in Amazon ElastiCache and a new high memory database instance (db.cr1.8xlarge) for Amazon RDS.

Some more details can be found in this blog post by Jeff Bar. I couldn’t find any references if this new AWS service offers features like those provided by GarantiaData: auto-scaling, auto-failover.

Original title and link: Amazon Web Services support for Redis (NoSQL database©myNoSQL)

via: http://www.allthingsdistributed.com/2013/09/amazon-elasticache-redis.html


Redis at Twilio and What can be learned from an incident

Twilio is using for its billing system a combination of Redis, for temporary state tracking, and a relational database, as the primary source of truth. They’ve recently had a critical incident with this system and posted a detailed timeline and analysis.

Redis has been involved in this system malfunction and Salvatore Sanfilippo wrote about a couple of things that could be improved from a Redis perspective.

Problem:

We experienced a loss of network connectivity between all of our billing redis-slaves and our redis-master. This caused all redis-slaves to reconnect and request full synchronization with the master at the same time.

Future solution:

Redis 2.6 always needs a full resynchronization between a master and a slave after a connection issue between the two. Redis 2.8 addressed this problem, but is currently a release candidate, so Twilio had no way to use the new feature called “partial resynchronization”.

Problem:

Observing extreme load on the host, the redis process on redis-master was misdiagnosed as requiring a restart to recover. This caused redis-master to read an incorrect configuration file, which in turn caused Redis to attempt to recover from a non-existent AOF file, instead of the binary snapshot. As a result of that failed recovery, redis-master dropped all balance data. In addition to forcing recovery from a non-existent AOF, an incorrect configuration also caused redis-master to boot as a slave of itself, putting it in read-only mode and preventing the billing system from updating account balances.

Solution:

Fortunately Redis 2.8 provides a better workflow for on- the-fly configuration changes […] Basically the config rewriting feature will make sure to change the currently used configuration file, in order to contain the configuration changes operated by CONFIG SET, which is definitely safer.

Problem:

Services relying on the redis-master began to fail due to the load generated by the slave synchronization.

This seems to be an unconfirmed problem and Salavatore ran a test to double check it:

Actually for the way Redis works a single slave or multiple slaves trying to resynchronize should not make a huge difference, since just a single RDB is created. […] However what is true is that Redis may use additional memory with many slaves attaching at the same time, since there are multiple output buffers to “record” to transfer when the RDB file is ready. This is true especially in the case of replication over WAN.

Some thoughts:

  1. I’m wondering if chained replication, where some of the slaves would be masters for other slaves, could have help alleviate the problem and not lead to the restart of the main master.
  2. Another idea could be to have a primary slave (that could also serve as a hot standby) and have all the slaves connect to it. The pressure on the master would be relieved and there would always be a hot standby. The downside might be the lag by which the end-of-chain slaves would be behind.

Original title and link: Redis at Twilio and What can be learned from an incident (NoSQL database©myNoSQL)


Redis configuration rewriting

Salvatore Sanfilippo about a new feature coming with Redis 2.8 simplifying Redis configuration management:

I believe that now that CONFIG REWRITE somewhat completes the triad of the configuration API, users will greatly benefit from that, both in the case of small users that will do configuration changes from redis-cli in a very reliable way, without a restart, without the possibility of configuration errors in redis.conf, and for big users of course where scripting a large farm of Redis instances can be very useful.

✚ Salvatore says that this new feature completes the configuration API. One future extra feature could be replicating some master configuration options to its slave automatically.

Original title and link: Redis configuration rewriting (NoSQL database©myNoSQL)

via: http://antirez.com/news/54


Adding a new command to Redis

Kenny Hoxworth shows how to add a new command to Redis:

A strength of Redis from a programmer’s perspective is the simplicity of the codebase. One of Sal’s goals with the project is to minimize external dependencies, and thus Redis performs minimal linking with shared libraries. The simplicity and organization of the Redis codebase makes minor (and sometimes major) modifications fairly intuitive.

The emphasis here is not on the fact that you can easily add custom commands to Redis, but that the Redis source code is so clean that you could add new commands. Also clean code means less bugs (or at least easier fixes for bugs).

Original title and link: Adding a new command to Redis (NoSQL database©myNoSQL)

via: http://www.starkiller.net/2013/05/02/hacking-redis-adding-a-command/


Redis on Windows Stress Tests

Claudio Caldato1 reports on the advance the Microsoft team is making towards releasing a stable, (very well) tested version of Redis for Windows:

In phase I of our stress testing, we put Redis on Windows through various tests with execution times ranging from 1 to 16 days, and configurations ranging from a simple single-master setup to more complex configurations such as the one shown below, with one master and four replicas.

The team also published the details of their stress tests here


  1. Claudio Caldato is Principal Program Manager Lead in Microsoft Open Technologies team. 

Original title and link: Redis on Windows Stress Tests (NoSQL database©myNoSQL)

via: http://blogs.msdn.com/b/interoperability/archive/2013/04/22/redis-on-windows-stable-and-reliable.aspx


MongoDB's TTL Collections in OpenStack's Marconi queuing service

Flavio Percoco describing some workaround OpenStack’s queing system is when using MongoDB’s TTL collections:

Even though it is a great feature, it wasn’t enough to cover Marconi’s needs since the later supports per message TTL. In order to cover this, one of the ideas was to implement something similar to Mongodb’s thread and have it running server-side but we didn’t want that for a couple of reasons: it needed a separated thread / process and it had a bigger impact in terms of performance.

This got me thinking it might be one of the (few) features missing from Redis.

✚ Redis supports timeouts for keys. Redis 2.6 brought the accuracy of expiring keys from 1 second to 1 millisecond.

✚ Redis has support for different data structures like lists, sets, and sorted sets. But it’s missing the combination of the two.

Original title and link: MongoDB’s TTL Collections in OpenStack’s Marconi queuing service (NoSQL database©myNoSQL)

via: http://blog.flaper87.org/post/517c3ea50f06d3497faffe5a/?buffer_share=d639c