Redis Conference 2015 at the Innovation Hangar at the beautiful Palace of Fine Arts in San Francisco.
Thanks to sponsors Rackspace, Hulu, Redis Labs, Heroku, and others.
Talks list
Redis’ 6th birthday today!
Redis Keynote by Salvatore Sanfilippo (Pivotal) a.k.a. @antirez
Genesis
Big equivalence principle : Today’s startups can do everything with everything, but a subset
of tools can be better suited for a given use case
Focus & Relevance
Cache, queue, shared state, database?
Redis is a small, networked, DSL interpreter, with optionally persistent state. Hence
multipurpose.
Economically sustainable?
It is rare to get a lot of work from non paid developers
Pieter, Matt, Salvatore : VMware, Pivotal sponsorship
Most contributions to core and other areas often sponsored or with ROI in contributing.
Development model
95% users happy with feature set, 5% create pull requests and want more; number of users
grow, 5% grows bigger but core team size remains same, so number of github issues is not a
good metric.
New branching model
Stable (2.8) <- Testing
Testing (3.0) <- Unstable
Unstable (3.2) : New features go here
Future optimizations (prefix “maybe” to every point below)
If you do not use pipelining, most time will be spent in network I/O
Memcached threaded I/O?
Threads for socket I/O, request parsing, etc.
Command execution via Global Lock
May require big core changes
Worth a try
Quicklists
90% less memory - RDBs more compact
Faster to serialize
Same concept is applicable to Sets and Hashes at least
Cluster optimizations
Non-blocking migrate
Auto rebalancing
The Big Refactoring Project
Redesign in terms of granularity + additional layers of API
Operations optimizations
Diskless ops WIP
nosync branch for slaves to get only new data from master, not history
Sentinel changes
API changes
Bloom filters
API is not obvious, since Redis has no object creation rule
Can’t use fixed params like HyperLogLog
Design effort needed
Geo coding
Matt’s branch 90% ready to be merged
Useful for lot of users
More list ops
Lists don’t have obvious APIs
LJOIN & LSPLICE
Unification of types?
Hashes and Sorted sets unified?
Ordered maps
Frequency estimation
You can already use sorted sets for top N problems
Good companion to HyperLogLog
Binary counters
Specialized stripped down spin-offs
Disque / queues
Multimaster cache
Evolution of caching
Make it simple to mutate only existing state
More APIs to check / analyze data
Why Redis
Better system software -> Better technology -> Better world
Redis in the Real World by Bill Anderson (Rackspace)
How are users using Redis
Rackspace surveyed tens of thousands of instances of Redis hosted for customers
600 billion commands
Top Commands
23% : Get
18% : Setnx
13% : Llen
7% : Lpop
6% : Zrangebyscore
Operations by Data type
Lists > Strings > Sorted sets > Hash > Set
Is it coincidental that most used commands are the fastest commands?
Reads, Writes, Checks
54% : Reads
35% : Writes
11% : Checks
List operations
36% : LPOP (write)
36% : LLEN (read)
So we use list data structure as a queue?
Sorted set
84% : ZRANGEBYSCORE (reads)
Hash
42% : HGETALL (all)
28% : HMSET (subset)
But shouldn’t we just use a string key + JSON value to store hash instead?
Strange that HGET is not commonly used!
What does all this mean?
Top commands are indeed fastest
Since we know time complexity, we can predict performance of Redis commands, this is great
Favor Setnx > Set in general benchmarking
Favor list ops in general benchmarking, then sorted set
Benchmarking
Benchmarking makes you go bald ;-)
First rule of benchmarking : Don’t do it
Benchmarking is like choosing a car based on it’s top speed
Don’t benchmark Redis, benchmark your application!
Instrument your code, gather the data, analyze the data. That’s what matters.
If you still want to : redis-benchmark, memtier
redis-benchmark is by Salvatore
Good
Gold standard
Bad
Doesn’t support all commands
Can’t do complex scenarios
Terrible output format for analysing
memtier
Good
Can tune writes vs reads
Can adjust concurrency
Bad
GET/SET isn’t a Redis benchmark, it tests networking (99% of time in networking)
Terrible output format for analysing
Time of day for benchmarking is important because your network could be saturated
Let us benchmark by use case
Leaderboard
Object cache
Queue
Page cache
DB query cache
Introducing Commissar, to do exactly this
Both benchmarking as well as documentation on design patterns a.k.a use cases
Separate server time vs. network/client time
Example Leaderboard output - 99.91% time spent in network/client, 0.09% in Redis server
Will allow us to compare uses cases
Will allow us to compare and improve implementations for a use case
Will allow us to compare and improve client libraries
But there are many different ways to do a thing…
How can you help?
Write more implementations
Write a spec for a new scenario
Port to your favorite programming language
Monaco, Redis as a Service at Hulu by Keith Ainsworth
What is Monaco
Clustered service that provides Redis as a service
Supports sharding for large data sets
Addresses production Redis challenges (setting up clusters, failovers, etc.) that was faced
by multiple teams
Production Redis issues
Clustering
Redis Cluster is great BUT one management layer for one cluster of redis servers
Persistence
FS based persistence has quirks
RDB can use huge amounts of RAM, fork speed
Clustering
Monaco provides one management layer for many redis server clusters
Per cluster
Monitor state of all servers
Maintain replication roles & failover
Expose servers transparently to clients (use existing client libraries)
Persistence
Rack-aware clustering + External backups
Web interface
Create a cluster, etc.
Failovers
Monaco nodes do health checks on redis servers
If a failure, promotes DB replicas for any lost servers in order of their replication
sequence. After ensuring all user servers have a responsive master, Monaco creates replica
servers as necessary
If redis-server failure (never seen in production), Monaco slaves notify Monaco master that
will pick ideal redis slave to promote to redis master
Major components
Load balancer
Provides interface to redis-clusters in Monaco
Monaco masters
Monaco slaves
Hosted on internal Heroku-like service
(incomplete notes, could not capture many points because speaker had to run through the slides)
Monaco cluster node
Daemon : Maintains threads
LeaderPaxos : Maintains leader <- Quorum of Monaco nodes
ZeroMQ
Similar in behavior to redis-sentinel
Master : Responsible for MGMT DB <- Web API
Slave : Maintains user DB
Stat reporter : Continually polls local MGMT DB for redis-servers allocated to that node
You built Redis as a service on top of Redis!?
Demo
Results
Monaco eliminated a lot of inefficient hardware use
There were many single-purpose VMs and clusters
Redis can be guiltlessly used for a tiny cache or huge persistent DB, without a barrier of
operations
Hardware : 12 x 24core, 128 GB RAM servers, 5 more this month
110 production clusters of redis-server
190 unique redis-servers
468.5 GB total hosted space
1.6 GB/s in, 1.1 GB/s out
~1 million ops per sec
Beyond Hulu
Desire to open source
Hard part is load balancer configuration API which is currently internal Hulu API
Problems & Future Work
Beta support for Twemproxy as a proxy for sharding
and multiple databases (did I understand that right?)
Questions
Latency issues because of load balancer? Not really, an extra hop in the network path
Redis-server version upgrades? You can mark a node in maintenance mode, upgrade and then
continue
Paxos implementation used? Custom code following “Paxos made simple”
How to benchmark Redis against other databases by Itamar Haber (Redis Labs)
Redis Watch newsletter
Interesting developments such as:
https://github.com/seppo0010/rlite
https://github.com/yahoo/redislite
Similar in topic to Bill’s talk earlier
https://github.com/RedisLabs/redisbenchmarkframework
Python & Fabric
Let’s differentiate profiling and benchmarking
Lahav Savir comes on stage to talk about use case of a TV show voting platform and comparison of
NoSQL for the high-spike traffic use case (2 million votes per second / 7 million rps)
Redis pipeline feature is critical for performance
(Lunch discussions)
Redis seems to be usually an individual developer’s decision to use and then it grows and grows in
scope of use, as opposed to centralized decisions of the main database, etc.
Redis as a data store for personalization engines by Ted Price (Knowd)
Web analytics
Redis changed everything, it made storing web analytics trivial
Use hashes, sorted sets
Use case : Tracking Links + primary category + secondary category
Started tracking users
Recommend content based on click-through rate (CTR)
Use HttpRedis2Module : where possible, hit Redis server directly from nginx
Use case : Classifiers
Gender classification of website, article (women, neutral, men)
Fact: Celebrity gossip visited almost equally by women and men
For returning users, filter widgets by classification, and this resulted in traffic
increase
Geo classifier
Publishers wanted more North American traffic
(This also supports Salvatore earlier saying that geocoding would be a great API addition
to Redis server)
Tier classifier
Tier based on website’s traffic levels
Use SVM (Support Vector Machine)
Use lists, hashes, sorted sets
Ruby gems aplenty
Other Redis use cases
Cache busting
Job queueing
Summary
Performance makes it an ideal choice for real-time personalization, clustering, segmentation
of content
Q&A
Audience who went through similar exercise suggested using a
queue for activity stream processing
(immutable write-ahead log) instead of a mutable merged database such as Redis
Real-time Events-based Document Index Store by Siddarth Kothari (Appbase)
https://appbase.io : now open source
http://appbaseio.github.io/appbase-redis/
Built on node.js
why?
relational data vs linked data
Pull a Mongo on top of Redis!
Collection has documents (JSON object)
Add references : { "/ref1" : "user/item#3" }
user is a collection
Paths
Network topology : a -> b -> z -> x becomes “/a/b/z/x”
Event subscriptions
All built on top of simple Redis data structures
Redis at Heroku (missed attending this since multiple tracks)
https://speakerdeck.com/hgmnz/redis-at-heroku
Building a massively scalable job queue on Redis by Jon Hyman (Appboy)
Appboy is marketing automation for mobile apps
Job queues
Hundreds of millions of jobs per day
100K rps
“Redis is the only database that actually delivers what it promises”
Scheduled jobs
zadd payload to sorted set key with score of timestamp for job
zrangebyscore (O(1)) + zrem
Use Lua to overcome race conditions
(More details, hopefully Jon will put up the slides online)
Use jitter (random microseconds) added to the scheduled time so that it doesn’t load Redis
zrangebyscore
Maintain counters on top of the queue using MULTI
Reliability
rpop means queue message can be lost in crash
Use rpoplpush to pop from one queue and push to another reliable queue
When job is finished, lrem from reliable queue
Check queue on process start for data recovery
Scaling
One instance for everything
Increased machine size
Dedicated Redis process for job queue
EC2 latency / noisy neighbours hurt performance, moved to physical machines via
Rackspace OnMetal
Spikes in job queue peg individual CPU
At some point, you will hit your max rps and you may not know it
Application-level sharding
Round-robin lpush to multiple instances
Queue starvation
First thing everyone does is add simple priority queues or weights
Queue weights are not a bad idea, but static queues are
Queue per customer is hard to scale because a lot to manage
rpoplpush supports only one key at a time
Create a queue as-needed for work, remove it when work is done
Add new queue name to a “dynamic queue” set
When looking for work, smembers to find all dynamic queues
When rpoplpush-ing dynamic queue, if no job found, atomically srem using Lua to
re-check queue to avoid race condition and losing jobs
Be conscious about rpoplpush, use brpoplpush where possible
Pause processing for maintenance of data
How to pause generic, non-dynamic queues?
Tag all jobs with customer identifier
Push customers under maintenance using PUBSUB
When jobs are picked up by worker server, check customer identifier, re-enqueue
Easily done with middleware around job processing
Here be dragons
Restoring from backup = hard / dangerous; definitely use slaves
Q&A
What if a job worker died, what happens to process in a reliable queue?
Cron job that checks entries in reliable queues and sends alerts
Running Twemproxy in production by Manju Rajashekhar
Proxy for memcache and redis, built at Twitter, open sourced in 2012, used now at pinterest,
wikipedia, twitch, snapchat
Motivation: Lots of remote connections to Redis, so familiar computer science solution of
indirection : twemproxy is deployed as a local proxy on each app server
Not recommended to deploy as a remote proxy
Features
Fast & lightweight
Persistent server connections
Implicit protocol pipelining (will work across multiple connections connected to one
twemproxy)
Shards data automatically
Ketama a.k.a consistent hashing
Avoid double-copy problem (?) using mbuf (fixed-size memory buffers)
Fault Tolerance
Client retries on transient errors
Don’t do it on invalid argument errors, etc.
What happens if it is a slow redis server?
Configure timeout on twemproxy, it is infinite by default
Re-routing on failures
Remove host from ring after 3 consecutive server failures / timeouts
Use auto_eject_hosts: true when redis is being used as a cache
Use auto_eject_hosts: false and node names (instead of host names when doing sharding),
when redis is being used as a database
Have number of client retries > number of twemproxy server retries if you want a query to
have highest chance of success
Simultaneous failures
Have larger server_retry_timeout
Trade-off between transient failures and permanent failures
Use node names for hashing, instead of host names or IP addresses
This makes deployment easy
Use server1, server2, … instead of 127.0.0.1:6379:1, 127.0.0.1:6380:1, …
Check configuration syntax errors early
Use --test-conf flag
Logging
By default, disabled
Use at log level LOG_NOTICE (-v 5) in production
Graphing ejected servers
hash_tag: “{}”
What if you wanted to do SDIFF on set 1 on server 1 and set 2 on server 2?
Map two different keys to same server by suffixing a common tag to the keys, but this should
be in your design early
Q&A
@antirez requests supporting redis cluster protocol
Manju is busy with his own startup and would love help and is willing to mentor
volunteers
Scaling Video Views by Andres Rangel (Hulu)
Problem setting
High traffic (10K+ RPS) of data to be tracked per user, per video (even video progress!)
Low latency (99% < 20ms)
High Availability
Scalability
Architecture
Long ago, when there was no redis cluster, built masters and slaves via zookeeper
From MySQL to Redis + Cassandra
Redis as cache
Add job to queue to load data from Cassandra to Redis
Redis as job queue
Redis as database : video progress, video metadata, recommendations list, etc.
Redis as sync point : locking for cron jobs
Configuration
64 shards
12 physical boxes
Sharded by user id, simple % operation
384 GB RAM boxes
Dual Hex core, 24 hyper-threads
1 GB/s network
Redis 2.8.19 - recently upgraded
HA-Redis, similar to official Redis-Sentinel design
sentinel : monitoring nodes
client : drop-in replacement for standard python/java redis client
reads from zookeeper to determine who is master, slave, and then will make the actual
connection
Why not Redis Sentinel? HA-Redis predates Sentinel
(Lots of implementation details of how they use Redis & Lua scripting)
Keep up to 1/2 of RAM free in case there’s a slave sync
Single-process architecture can be scaled by multiple instances per machine
Lua scripting is very powerful, helps reduce data in redis, helps reduce network trips
If you want low latency, do not save to disk
To save memory usage (50% reduction) in redis (tradeoff is CPU usage), set
*-max-{ziplist,intset}-entries to 4096
http://blog.pivotal.io/pivotal/case-studies-2/case-study-how-hulu-scaled-serving-4-billion-videos-using-redis
The Secrets behind Redis Development by Matt Stancliff (Pivotal)
Freewheeling talk
Worked 16 hours per day for 6 weeks to implemented quicklist (mentioned by Salvatore that it
reduced memory usage to one-tenth of what it was before)
Went dark on GitHub, did not look into there at all
Present
People build valuable, multi-billion dollar businesses on top of Redis
Redis: You use Redis? Great
Users: We use Redis, we expect features and bug fixes, but we won’t pay
So how to balance what everybody wants? Arrow’s Impossibility Theorem
Redis internals are no longer simple, so many different concepts
Secret: We have enough work for 50 people. One-person-does-everything does not scale!
Pull requests for spelling mistakes are ignored
Secret: Limited time
A Plan
Extract : make reusable C libraries, server, distributed systems
Refactor
Make it easy for new people to contribute
Improve
Fix many many holes
Grow
Add JSON support
Add geocoding support
Allow reading config from network config storage (etcd, consul, etc.)
Again, Redis development does not scale! More community participation and contributions needed.
Closing Comments by Matt Barker
All talks have been recorded and will be put online in the coming weeks
(PostScript)
Thanks to @vedang for letting me know about this conference!
Final note - Totally amazed that there is so much being built on top of Redis.
Comments
@harikt says:
Nice post
@erictyliaw says:
Thanks for sharing your notes!
@RedisLabsInc says:
good summary until the videos are online, thank you!
@appbaseio says:
A good summary!