A while back Eric Florenzano has published a post describing how Convore is using Redis Pub/Sub. There were a couple of things in that post that made me reach out to Eric for more details. Eric obliged:
Alex: Why did you go with Celery and Redis? Even Celery page recommends using RabitMQ.
Eric: Basically my philosophy is that all things being equal, less moving parts is preferable to more moving parts, especially with a small team and no dedicated ops person. We were already using Redis for other things, like pub/sub, so using lists along with its blocking pop operation seemed natural as a simple queue. Celery supports that kind of setup out of the box. Additionally, the newest version of Celery works much better with Redis as it’s now a first-class citizen (instead of being supported by a module literally called ghettoq).
Alex: If I read you post correctly, for each group you’ll have a PUB/SUB in Redis. This means not only the number of subscribers could vary largely, but also the number of PUB/SUBs. How far have you tested the PUB/SUB in Redis? How do you plan to go with it?
Eric: Since I wrote that post, we’ve actually changed the pub/sub targets to be one for each person instead of one for each group. This allows us to send messages like @mentions only to the user that’s interested, rather than to the whole group. In any case, you’re right, the number of pub/sub targets will get large, and each one could have many subscribers. I’ve tested it with a few thousand concurrent connections, and Redis has consumed a negligible amount of memory and CPU.
I’d love to be able to say that we’re ready to scale up to millions of concurrent connections, but that’s simply not the case quite yet. As a company we’re focusing more right now on getting the product right and the business right, so that we have a reason to scale up to millions of connections. It’s not a very interesting answer or approach, but I think it’s the right one.
Alex: Is Redis availability of concern for your application? How do you deal with it?
Eric: Our system is architected such that Redis being unavailable won’t block the write path into our RDBMS (our canonical source of data). In practice this means that users may stop seeing real-time updates to their messages, but refreshing the page will show the messages properly as they arrived. Alex: Sounds like a very pragmatic approach. Good luck with Convore and thanks a lot for your answers.