Presentation: Redis - REmote DIctionary Server by Ezra Zygmuntowicz

Fantastic presentation by Ezra Zygmuntowicz (@ezmobius) on Redis:

My notes from the slides:

Redis usecases:

• Memcached on steroids
• Tag clouds, leaderboards
• Stat collections, circular log buffers
• Share state between processes
• A/B testing
• REDIStribute your load
• Distributed Lock Manager for process coordination
• Full Text Inverted Index Lookups
• Caching with extra smarts using lists, sets and atomic ops on said structures

Some of these usecases are detailed later in the slides. You should definitely check this great list of Redis usecases.

I’ve heard that there might be a video for this presentation, so I hope that will be published pretty soon as I’d really like to include it here with the slides.

You can find below the text on slides (no formatting though)

• Slide: 1

  Redis

  REmote DIctionary Server

  Ezra Zygmuntowicz

  twitter: @ezmobius

  #redis

• Slide: 2

  · Fast,in memory key/value store

  · STRING,LIST,SET & ZSET data types

  · Persistence via async snapshots orAOF

  · Perfect Data Structure/State/Cache Server

• Slide: 3

  Data Structure Server

• Slide: 4

  Data Structure Server

  Key:String => Value:String

• Slide: 5

  Data Structure Server

  Key:String => Value:String

  Key:String => Value:List

• Slide: 6

  Data Structure Server

  Key:String => Value:String

  Key:String => Value:List

  Key:String => Value:Set

• Slide: 7

  Data Structure Server

  Key:String => Value:String

  Key:String => Value:List

  Key:String => Value:Set

  Key:String => Value:Zset

• Slide: 8

  Operations on any Type

  · Exists,Del,Type

  · Keys,Randomkey

  · Rename,RenameNX

  · Dbsize,Select,Move,Flushdb,Flushall

  · TTL,Expire

• Slide: 9

  Operations on STRING’s

  · Get,Set,GetSet,SetNX

  · Mget,Mset,MgetNX,MsetNX

  · Incr,Incrby

  · Decr,Decrby

• Slide: 10

  Operations on LISTS’s

  · Atomic Operations:

  · Push/Pop

  · Index (array indexing)

  · Lrange,Ltrim,Llen

  · Blpop,Brpop

  · RpopLpush,BRpopLpush

• Slide: 11

  Operations on SET’s

  · Sadd,Srem,Spop,Smove

  · Scard,Sismember,Smembers,Srandmember

  · Sinter,Sinterstore,Sunion,Sunionstore

  · Sdiff,Sdiffstore

• Slide: 12

  Operations on ZSET’s

  · Zadd,Zrem,Zincrby

  · Zrange,Zrevrange

  · Zrangebyscore,Zcard

  · Zscore,Zremrangebyscore

• Slide: 13

  Seems cool, but what are the use cases?

  · Memcached on Steroids

  · Tag Clouds,Leaderboards

  · Stat collections,circular log buffers

  · Share state between processes

  · A/B testing

  · REDIStribute your load

• Slide: 14

  Example:Tagging

• Slide: 15

  Example:Tagging

  SADD article:42 magick

  SADD article:42 unicorns

  SADD article:42 rainbows

• Slide: 16

  Example:Tagging

  SADD article:42 magick

  SADD article:42 unicorns

  SADD article:42 rainbows

  SADD article:12 magick

  SADD article:12 bar

  SADD article:12 qux

  SADD article:12 foo

• Slide: 17

  Example:Tagging

  SADD article:42 magick

  SADD article:42 unicorns

  SADD article:42 rainbows

  SADD article:12 magick

  SADD article:12 bar

  SADD article:12 qux

  SADD article:12 foo

  SINTER article:42 article:12 

  #=> magick

• Slide: 18

  Example: Fair Work

  Scheduler

• Slide: 19

  Example: Fair Work

  Scheduler

• Slide: 20

  Example: Fair Work

  Scheduler

  Each worker node periodically issues:

  ZADD worker:nodes 1.5 hostname

  1.5 is the load ave of the host

• Slide: 21

  Example: Fair Work

  Scheduler

  Each worker node listens for work with:

  BLPOP hostname 10 

• Slide: 22

  Example: Fair Work

  Scheduler

  When a producer wants to issue a work request:

  ZRANGE worker:nodes 0 2 

  returns top 3 least loaded nodes

• Slide: 23

  Example: Fair Work

  Scheduler

  Pick a random node out of the 3 least loaded nodes in order to add jitter so we don’t overload hosts

• Slide: 24

  Example: Fair Work

  Scheduler

  Then issue the work request:

  LPUSH hostname {“work”: {“foo”:”bar”}}

• Slide: 25

  Example: Fair Work

  Scheduler

  This setup uses ZSETS and the load average as a score in order to evenly distribute load across a farm of worker instances listening for jobs with BLPOP

• Slide: 26

  Example:AMQP

  message de-dupe

• Slide: 27

  Example:AMQP

  message de-dupe

  Route messages through 2 separate rabbitmq brokers for redundancy

  Use redis for message de-dupe by tagging messages with a guid

  Use guid as key into redis

  Use SETNX when setting the guid key so first one wins and second messages gets discarded as already received

• Slide: 28

  Other Ideas

  · Distributed Lock Manager for process coordination

  · FullText Inverted Index Lookups

  · Caching with extra smarts using lists,sets and atomic ops on said structures

  · Share data structures between multiple processes like a blackboard/tuplespace

• Slide: 29

  Speed

  · 110k GETS/SETS/second on average linux box

  · Mostly faster then memcached for same ops

  · Event Driven,can handle thousands of clients with epoll/kqueue support

  · Ops happen in memory,persistence is async so everything is very fast

• Slide: 30

  Persistence

  · Async Snapshots with configurable triggers

  · Append Only File: AOF

  · Redis 2.0 Virtual Memory

• Slide: 31

  Replication

  · Built in Master -> SlaveAsync replication

  · Create replication chains as needed

• Slide: 32

  Sharding

  · Client side sharding (like memcached)

  · Consistent ring hashing

  · Special case keys “foo{tags}” for operations on keys that must live on the same server in ruby client

• Slide: 33

  Redactor

  · SimpleActor Library based around Redis

• Slide: 34

  Questions?