edit: all this is based on retwis.antirez.com memory usage.
Ok, just did some math. In order to run Twitter using just Redis 1.001 without to use any new feature that may allow for memory savings, and guessing that Twitter currently contains 4,000,000,000 of tweets (assuming they save the full history for all the users, and that the recent 32 bit overflow means they rougly have 4 billion messages), 30 128GB Linux Boxes are needed.
Is this crazy? I don't know honestly as I don't know how many servers they may be using currently for the DB backend.
Btw the whole point is, IMHO, that many times to take the full dataset on Redis is not needed. For instance in twitter only recent messages are accessed frequently, together with user data, so it's probably a good idea to take only the latest N messages of every user in Redis (with background jobs moving old messages on disk incrementally), and take all the rest on MySQL or other on disk solution suitable to access stuff by-id.
So when you want to get a message from Redis, and from time to time get a NULL accessing to message:<id> you can run the same query against MySQL to get the data. That's something like this:
def getMessageById(id)
m = redis.get("message:"+id)
if !m
m = getMessageFromMySQL(id)
end
return m
end
In this context it is very simple to move old messages from a Redis server to a MySQL server, since messages are on a redis list, so it's possible to RPOP to get old elements, if LLEN (List length) reports that this user has more tweets that we want to take in the "fast path".
Also note that Redis supports expires in keys. So old messages got from MySQL can be set as expiring keys in order to avoid that a message that got linked in some front page will stress the MySQL too much.
This is just to give a feeling about scaling a pretty big service using Redis as main DB without caching layers.
There's a big difference between sharding across 30 redis nodes, where your application has to be shard-aware, and your ops team has to manually handle failover, etc, and using a database that looks to the app like a single system. In other words redis's story here isn't really any better than sharding a relational db, and everyone knows how much that sucks.
So saying on the home page that "Redis can do [sharding] like any other key-value DB, basically it's up to the client library" is inaccurate. Distributed key-oriented databases like cassandra, voldemort, dynomite, riak handle all of that so it's totally invisible to your app, including (at least in Cassandra's case, and I think dynomite) adding nodes to the cluster.
it's really a matter of design. I like the idea that the Redis servers are dummy, and it's up to the client logic to handle sharding. For instance the Ruby client supports this feature in a way mostly transparent to the client.
In traditional databases sharding is hard not because they are not good at it form the point of view of "feature set" (like in Redis VS Cassandra), but because the data model itself is not right for working with data split across different servers. If you use an SQL DB just with tables accessed by IDs and without queries more complex than lookups by primary key, then sharing starts to become simpler.
Even if Redis will ever get server-side sharding, I'll code another process that handles this issue instead to put the logic inside Redis itself.
Btw how is it possible to build something really horizontally scalable without to use client-level sharding?
What you want is to have N web server and M databases, without any single-dispatch-node. At least this is how I'm used to thing at it.
Without any kind of client help I guess there is some kind of master node handling the dispatching of requests. Maybe I missed the point, please give me some hint.
Update 6: Some interesting changes from Twitter's Evan
Weaver: everything in RAM now, database is a backup; peaks
at 300 tweets/second; every tweet followed by average 126
people; vector cache of tweet IDs; row cache; fragment
cache; page cache; keep separate caches; GC makes Ruby
optimization resistant so went with Scala; Thrift and HTTP
are used internally; 100s internal requests for every
external request; rewrote MQ but kept interface the same;
3 queues are used to load balance requests; extensive A/B
testing for backwards capability; switched to C memcached
client for speed; optimize critical path; faster to get the
cached results from the network memory than recompute them locally.
I'm investigating about this issue more, it could be very interesting to know how many tweets twitter itself is indexing currently to do a precise estimation.
Ok, just did some math. In order to run Twitter using just Redis 1.001 without to use any new feature that may allow for memory savings, and guessing that Twitter currently contains 4,000,000,000 of tweets (assuming they save the full history for all the users, and that the recent 32 bit overflow means they rougly have 4 billion messages), 30 128GB Linux Boxes are needed.
Is this crazy? I don't know honestly as I don't know how many servers they may be using currently for the DB backend.
Btw the whole point is, IMHO, that many times to take the full dataset on Redis is not needed. For instance in twitter only recent messages are accessed frequently, together with user data, so it's probably a good idea to take only the latest N messages of every user in Redis (with background jobs moving old messages on disk incrementally), and take all the rest on MySQL or other on disk solution suitable to access stuff by-id.
So when you want to get a message from Redis, and from time to time get a NULL accessing to message:<id> you can run the same query against MySQL to get the data. That's something like this:
In this context it is very simple to move old messages from a Redis server to a MySQL server, since messages are on a redis list, so it's possible to RPOP to get old elements, if LLEN (List length) reports that this user has more tweets that we want to take in the "fast path".Also note that Redis supports expires in keys. So old messages got from MySQL can be set as expiring keys in order to avoid that a message that got linked in some front page will stress the MySQL too much.
This is just to give a feeling about scaling a pretty big service using Redis as main DB without caching layers.