The Story of Scaling Union Station
You think you are ready to scale:
Even before the launch, our software architecture was designed to be redundant (no single point of failure) and to scale horizontally across multiple machines if necessary. The most important components are the web app, MySQL, MongoDB (NoSQL database), RabbitMQ (queuing server) and our background workers.
And then the reality check:
Unfortunately MongoDB’s shard rebalancing system proved to be slower than we hoped it would be. During rebalancing there was so much disk I/O we could process neither read nor write requests in reasonable time. We were left with no choice but to take the website and the background workers offline during the rebalancing.
There are a couple of lessons to be learned here:
- there’s no silverbullet solution for scaling
- if you haven’t tested a specific scenario, you might have surprises when you’ll have do to it in the production environment
Original title and link: The Story of Scaling Union Station (NoSQL databases © myNoSQL)
