I'd like to pose a Redis question to the community at large. On HN, I keep hearing about how awesome it is, how readable the codebase is, how efficient and well thought out it generally is.
That's all fine, but there is incredible resistance in many enterprises to adopting new technology. I recently faced a similar problem trying to get people to move to Redis, and the feeling is that there isn't enough "enterprise social proof".
So my questions to anyone who has deployed a large enough Redis cluster (and by cluster, I mean replication, since sharding isn't apparently supported yet):
-- What is the largest instance you have run?
-- How reliable has that instance been?
-- How easy is it to maintain and administer?
-- What are some issues you would warn other enterprise users about?
I can't understand this comment now, after the youporn success story... Isn't this enough for enterprise people?
Ok seriously now... Groupon did an event a few days ago about how they use Redis with success, Twitter uses Redis for caching timelines, Github does a good use of Redis, and similarly do many other non trivial companies, like Craigslist, Stack Overflow, Digg, ...
Those are not "enterprise" companies, but the problems they solve are not different, so IMHO the real problem is a cultural barrier. I think that this is just a matter of time.
Seriously, neither can I, and you're right that it's a culture problem. I am genuinely trying to do my part to overcome that. Towards that, it would be nice to see more in-depth expositions of setups, and above all, an honest description of any issues and problems. I'm trying, antirez, I'm trying... :)
One tried-and-true solution is to use it on "non-enterprise" projects (small, invisible, failure-tolerant) within your enterprise, then mention that you've used it, then use it on something slightly less obscure and mention it again ... worked for Linux in the '90's.
What worked for me: I tried new things (ie: ruby, continuous integration, redis...) first on low-risk projects, then used that feedback as a self-proof (look! it works - maybe it can go a bit further?).
The other simple thing to point out is all of the issues with "enterprise" software you guys use. That label doesn't mean software doesn't have bugs. That being said...i agree with the others to use it on smaller projects (to get used to oddities with it, etc.) and to show the "higher ups" ... look..it's fine..we've been using it in X, Y, and Z for months. Also, I think it would become _more_ prevalent once a 10gen / datastax / percona like place pops up for it (so mgmt can put a checkbox next to "support")
Don't work in enterprise. I'm not joking, and this isn't a trite stab or troll. I'm totally serious. I think most of the companies that folks describe as 'enterprise' shops ought to go away, along with the all of the expensive, broken, 'best practice' mentalities.
Also, you don't need proof. You're up against complacency, not suspicion.
Every command blocks every other command in Redis, because Redis is single threaded. However the implementation of scripting in Redis does a good use of Lua, so a command implemented in Lua does not run too much slower than a command implemented in C, for simple tasks (since we use a single Lua interpreter, and cache compiled functions). So this is an example of what you can do with Lua scripting:
redis 127.0.0.1:6379> set foo '{"b":20,"a":10}'
OK
The user wants to increment the value of "b" with a single query to Redis in an atomic way:
The example you cited just looks like an awfully convoluted way to recreate HINCRBY. Obviously there are much more powerful things that you could do with scripting, but this example makes me wonder if all those H-commands might be replaced with JSON & scripting (or handled internally by JSON & scripting) some time in the future.
Well it's not really HINCRBY, it increments a field of a JSON object that is otherwise not supported by Redis natively. It was just an example, but it is not a strange one, because there are many users with JSON objects, and similarly there are many users with specific needs: Do this if there is that in the key, add this element only if the list is < N elements, and so forth. With Lua scripting you can model all this without back-and-forth without the client and server, in other words, with 1/100 of the latency (or less).
Given that we'll implement new commands only when they are "fundamental" operatons. For instance there is some plan to implement list splicing. When this happens it's worth to write it in C for max performances, so I think we'll unlikely write parts of Redis in Lua itself. Scripting systems is something I love (see Emacs for instance) but IMHO Redis is the instance of system where this does not work: space and time are too important for Redis.
Thanks for the clarification. It's great to hear that scripting will reduce the need to create new commands. I'm sure some people have been getting slightly annoyed with the rapid proliferation of Redis commands, which already number well over 100. Can't remember them all!
"rapid proliferation of Redis commands"? If the total is indeed, well over 100, that figure is spread among completely different data structures--lists, hashes, strings, sets, sorted sets--so i'm not sure it's even meaningful to compare the number of commands in redis with the number in some other DB which is based on a single data structure. Also, the increase in the number of commands might indeed just most reflect the incremental addition of new data structures to redis since its beginning (just strings, i believe).
That reminds me, I'd really like to see an official wiki (or just a simple pastebin-like thing like the Flask snippets site) for Redis scripts, use cases, best practices...
I've seen quite a few blog posts that highlight some amazing way to use Redis and it would be very helpful to have them all in one place.
Wavii is using LUA in Redis to implement extremely simple single object atomic consistency. Some but not all of the Redis data structure types support "do XYZ only if the key exists". For example lists support LPUSHX but there is no "X" equivalent for sorted sets. With LUA I can add the ZADDX command in about 4 lines of LUA and 4 lines of Ruby in my Redis driver. We have lots of sorted sets backed by the database. Suppose we add an item to the set. If the set is in Redis we want to do the O(logN) ZADD, if it is not in Redis we need to do an O(N) reload from the DB. The use of ZADDX + transactions in our DB lets multiple writers contend on adding an item to a sorted set and guarantee that at the end all the items are in the sorted set while doing O(logN) operations almost all the time.
In order to write the kind of programs you write to script Redis, Lua or Javascript are substantially equivalent, but V8 is a huge implementation, less portable, and so forth. This is why I avoided it.
You want to be small, but C bit operations as native operators inside the core are a good idea.
Not being a Lua user, it took me a little while to parse that. You're right, implementing these as library functions is silly, and having a dozen implementations to pick from is ludicrous!
I am continually amazed by Redis. I am involved in a project where parallel processing and work distribution is achieved via 0mq and Redis acts as a sort of "shared memory". The performance is outstanding, and it is a fun architecture to work in.
I am also working on parallel processing for my Phd and implementing a Redis-based distributed shared memory queueing system that looks a lot like a tuple space. The Redis primitives fit this messaging model perfectly, allowing very loose coupling between language-neutral computational elements. I do have concerns that the single-threading may eventually impose a bottleneck with the scale of deployment intended (1000s of CPUs and GPUs). I have been looking into 0MQ to overcome these limitations and wonder how feasible it would be to implement a 0MQ socket connection option for Redis.
We use 0MQ and Redis together at work. We have an incremental flow of updates that we distribute internally over 0MQ pubsub, and have one listener process that updates the full state in Redis from each update message. That way, we have both (a) a way to easily get the latest state (and fast), and (b) an easy way to listen for updates to the state. It works beautifully, even with a high rate of updates.
I would love to hear more about that architecture. I just added 0MQ into our product to handle pushing and pulling events, among other things. Some of the events are OK if they get lost; others, not so much so. I've been thinking about adding something like redis or mongodb to the mix to provide a layer of persistence to those messages.
We're not necessarily using Redis to persist the 0mq messages, but more so allowing parallelized processes a medium to share data. The single threaded nature of Redis and 0mq work distribution eliminate a lot of nasty concurrency issues. You may want to look into the Redis pub/sub though for what you are describing (http://redis.io/topics/pubsub)
Is it possible to use a script to roll back redis transactions. Currently transactions aren't like SQL transactions in that they don't roll back if there is a problem executing the commands between the multi/exec.
Say if I add a new hash, and also increment a counter, and update a set to track a relation. I currently do this within a multi/exec to ensure it's atomic. It would be nice to roll back the commands, if say there was an internal error between the exec/multi or if there was an external event like losing a disk / bouncing the server by possibly writing to the AOF log.
Out of interest, how do people deal with this now?
I'll be adding redis into our $work stack in the next couple weeks to add pub/sub (email subscriptions) and a job queue (Resque). The ordered sets will likely be very useful to help us scale sorting of search results, etc.
From what I've read, it's going to blow my mind and open so many new possibilities. I can't wait for the clustering in 3.0 to come out!
You can still use Redis with C#, the only caveat I've found seems to be the current drivers don't support authentication but that's not a huge issue if it's not publicly accessible.
ServiceStack.Redis has always supported Auth in the individual byte[], string and POCO RedisClients, and thx to @dan_b it now also supports Auth in the Basic/PooledRedisClientsManagers as well.
I know nothing about this (neither redis auth nor MS' work), but if I were managing the server products, I would want to make sure the redis authentication doesn't confuse users of my platform. MS has a pretty clear goal of making everything authenticate via ActiveDirectory. Keeping that consistent may be more than a five minute task.
Well, first of all, he's not talking about a Microsoft-written library, he's just speaking generically of C#/.NET libraries written by others.
But Redis's authentication is very simple in any case. There's a password. A password, one. That's it. There are no accounts, no usernames, there's no "infrastructure" or any way to hook it into LDAP or other insanity. There's no accounting.
It's not for the kind of authentication ActiveDirectory does. In practice, it's more of an "Oops, did you really mean to connect to that server?" thing.
maybe this has something to do with the fact that AUTH makes the connection stateful and you have to remember the AUTH state on reconnection? Just guesswork.
Nah, the code for that is simple and generic in any sane language (and C# is reasonably sane).
But I think the guy I was responding to is at least partly wrong, a quick grep through the repo he linked to shows they have everything necessary for authentication, even if it's not as blindingly obvious as in, say, redis-py and redis-rb.
ServiceStack.Redis has always supported Auth on all the RedisClients themselves. It was just a feature that wasn't initially available on the Pooled Redis Managers - it's on there now with new PooledRedisClientManager("myauth@mythz.redistogo.com:9287").
That's all fine, but there is incredible resistance in many enterprises to adopting new technology. I recently faced a similar problem trying to get people to move to Redis, and the feeling is that there isn't enough "enterprise social proof".
So my questions to anyone who has deployed a large enough Redis cluster (and by cluster, I mean replication, since sharding isn't apparently supported yet):
-- What is the largest instance you have run? -- How reliable has that instance been? -- How easy is it to maintain and administer? -- What are some issues you would warn other enterprise users about?