An old post from Santosh Kumar describes the sequence of steps to parallelize processing from single threaded apps to using Redis and Resque and going through forking and process pools. Santosh has written extensively about using Redis for concurrency or as a Message oriented middleware (MOM).
The next time you need to get some background job action going, stop yourself from just grabbing a library. Instead, toy around with redis lists a little. You’ll be surprised by how much you can accomplish with just straight redis primitives.
While I agree with the usefulness of Redis, none of his posts say what comes after it. And that’s specialized solutions.
Original title and link: Parallelizing Work: From Single-Thread to Redis ( ©myNoSQL)
A Resque, the Redis backed background jobs tool, screencast by the Railscasts peeps. The 12 minutes video is available after the break and you can download the source code from the Railscast website.
Continue to read ➤
The 5 R: Ruby on Rails, Redis, Resque, Rufus.
Now the question is what’s the best way to run scheduled tasks in a Rails environment? Generally, Rails developers can use an application specific crontab to run application tasks. […] Downsides of Cron are:
- Cron works well on a single server but what if you need to scale to multiple app servers? You’ll need to introduce some kind of lock to avoid concurrency problems. Also, you need to maintain these shared locks.
- The other problem with Cron is that they are difficult to debug.
- Cron is for scheduling things, not doing them.
This can almost always be better to place jobs in queue and place the worker system to perform or to execute those jobs. Luckily there is a very good gem named ‘resque‘ available in Ruby on Rails.
Original title and link: Ruby On Rails: Cron Job Scheduling using Redis, Resque, and Rufus (NoSQL databases © myNoSQL)
After some research, we decided to use resque from github, but to adapt it to use mongodb instead of redis. The advantage here is that we already have a significant investment in mongodb, so we would not be introducing a new type of server to our infrastructure. Mongodb also has some features that redis does not, and we used those to build some interesting new features into resque.
After reading the post, I couldn’t identify those additional MongoDB features mentioned above. But I do agree you shouldn’t bring in new infrastructure requirements with every new tools you are using. Very heterogeneous platforms are difficult and expensive to maintain.
Original title and link: Simple Ruby Workers with MongoDB and Resque (NoSQL databases © myNoSQL)