ALL COVERED TOPICS

NoSQL Benchmarks NoSQL use cases NoSQL Videos NoSQL Hybrid Solutions NoSQL Presentations Big Data Hadoop MapReduce Pig Hive Flume Oozie Sqoop HDFS ZooKeeper Cascading Cascalog BigTable Cassandra HBase Hypertable Couchbase CouchDB MongoDB OrientDB RavenDB Jackrabbit Terrastore Amazon DynamoDB Redis Riak Project Voldemort Tokyo Cabinet Kyoto Cabinet memcached Amazon SimpleDB Datomic MemcacheDB M/DB GT.M Amazon Dynamo Dynomite Mnesia Yahoo! PNUTS/Sherpa Neo4j InfoGrid Sones GraphDB InfiniteGraph AllegroGraph MarkLogic Clustrix CouchDB Case Studies MongoDB Case Studies NoSQL at Adobe NoSQL at Facebook NoSQL at Twitter

NAVIGATE MAIN CATEGORIES

Close

Presentation: Redis - REmote DIctionary Server by Ezra Zygmuntowicz

Fantastic presentation by Ezra Zygmuntowicz (@ezmobius) on Redis:

My notes from the slides:

Redis usecases:

  • Memcached on steroids
  • Tag clouds, leaderboards
  • Stat collections, circular log buffers
  • Share state between processes
  • A/B testing
  • REDIStribute your load
  • Distributed Lock Manager for process coordination
  • Full Text Inverted Index Lookups
  • Caching with extra smarts using lists, sets and atomic ops on said structures

Some of these usecases are detailed later in the slides. You should definitely check this great list of Redis usecases.

I’ve heard that there might be a video for this presentation, so I hope that will be published pretty soon as I’d really like to include it here with the slides.

You can find below the text on slides (no formatting though)

  • Slide: 1

    Redis

    REmote DIctionary Server

    Ezra Zygmuntowicz

    twitter: @ezmobius

    #redis

  • Slide: 2

    · Fast,in memory key/value store

    · STRING,LIST,SET & ZSET data types

    · Persistence via async snapshots orAOF

    · Perfect Data Structure/State/Cache Server

  • Slide: 3

    Data Structure Server

  • Slide: 4

    Data Structure Server

    Key:String => Value:String

  • Slide: 5

    Data Structure Server

    Key:String => Value:String

    Key:String => Value:List

  • Slide: 6

    Data Structure Server

    Key:String => Value:String

    Key:String => Value:List

    Key:String => Value:Set

  • Slide: 7

    Data Structure Server

    Key:String => Value:String

    Key:String => Value:List

    Key:String => Value:Set

    Key:String => Value:Zset

  • Slide: 8

    Operations on any Type

    · Exists,Del,Type

    · Keys,Randomkey

    · Rename,RenameNX

    · Dbsize,Select,Move,Flushdb,Flushall

    · TTL,Expire

  • Slide: 9

    Operations on STRING’s

    · Get,Set,GetSet,SetNX

    · Mget,Mset,MgetNX,MsetNX

    · Incr,Incrby

    · Decr,Decrby

  • Slide: 10

    Operations on LISTS’s

    · Atomic Operations:

    · Push/Pop

    · Index (array indexing)

    · Lrange,Ltrim,Llen

    · Blpop,Brpop

    · RpopLpush,BRpopLpush

  • Slide: 11

    Operations on SET’s

    · Sadd,Srem,Spop,Smove

    · Scard,Sismember,Smembers,Srandmember

    · Sinter,Sinterstore,Sunion,Sunionstore

    · Sdiff,Sdiffstore

  • Slide: 12

    Operations on ZSET’s

    · Zadd,Zrem,Zincrby

    · Zrange,Zrevrange

    · Zrangebyscore,Zcard

    · Zscore,Zremrangebyscore

  • Slide: 13

    Seems cool, but what are the use cases?

    · Memcached on Steroids

    · Tag Clouds,Leaderboards

    · Stat collections,circular log buffers

    · Share state between processes

    · A/B testing

    · REDIStribute your load

  • Slide: 14

    Example:Tagging

  • Slide: 15

    Example:Tagging

    SADD article:42 magick

    SADD article:42 unicorns

    SADD article:42 rainbows

  • Slide: 16

    Example:Tagging

    SADD article:42 magick

    SADD article:42 unicorns

    SADD article:42 rainbows

    SADD article:12 magick

    SADD article:12 bar

    SADD article:12 qux

    SADD article:12 foo

  • Slide: 17

    Example:Tagging

    SADD article:42 magick

    SADD article:42 unicorns

    SADD article:42 rainbows

    SADD article:12 magick

    SADD article:12 bar

    SADD article:12 qux

    SADD article:12 foo

    SINTER article:42 article:12 

    #=> magick

  • Slide: 18

    Example: Fair Work

    Scheduler

  • Slide: 19

    Example: Fair Work

    Scheduler

  • Slide: 20

    Example: Fair Work

    Scheduler

    Each worker node periodically issues:

    ZADD worker:nodes 1.5 hostname

    1.5 is the load ave of the host

  • Slide: 21

    Example: Fair Work

    Scheduler

    Each worker node listens for work with:

    BLPOP hostname 10 

  • Slide: 22

    Example: Fair Work

    Scheduler

    When a producer wants to issue a work request:

    ZRANGE worker:nodes 0 2 

    returns top 3 least loaded nodes

  • Slide: 23

    Example: Fair Work

    Scheduler

    Pick a random node out of the 3 least loaded nodes in order to add jitter so we don’t overload hosts

  • Slide: 24

    Example: Fair Work

    Scheduler

    Then issue the work request:

    LPUSH hostname {“work”: {“foo”:”bar”}}

  • Slide: 25

    Example: Fair Work

    Scheduler

    This setup uses ZSETS and the load average as a score in order to evenly distribute load across a farm of worker instances listening for jobs with BLPOP

  • Slide: 26

    Example:AMQP

    message de-dupe

  • Slide: 27

    Example:AMQP

    message de-dupe

    Route messages through 2 separate rabbitmq brokers for redundancy

    Use redis for message de-dupe by tagging messages with a guid

    Use guid as key into redis

    Use SETNX when setting the guid key so first one wins and second messages gets discarded as already received

  • Slide: 28

    Other Ideas

    · Distributed Lock Manager for process coordination

    · FullText Inverted Index Lookups

    · Caching with extra smarts using lists,sets and atomic ops on said structures

    · Share data structures between multiple processes like a blackboard/tuplespace

  • Slide: 29

    Speed

    · 110k GETS/SETS/second on average linux box

    · Mostly faster then memcached for same ops

    · Event Driven,can handle thousands of clients with epoll/kqueue support

    · Ops happen in memory,persistence is async so everything is very fast

  • Slide: 30

    Persistence

    · Async Snapshots with configurable triggers

    · Append Only File: AOF

    · Redis 2.0 Virtual Memory

  • Slide: 31

    Replication

    · Built in Master -> SlaveAsync replication

    · Create replication chains as needed

  • Slide: 32

    Sharding

    · Client side sharding (like memcached)

    · Consistent ring hashing

    · Special case keys “foo{tags}” for operations on keys that must live on the same server in ruby client

  • Slide: 33

    Redactor

    · SimpleActor Library based around Redis

  • Slide: 34

    Questions?