Tylor Arndt about Torbit’s “build-your-own-MapReduce”:
The final system begins with a web-service against which client
systems interface. To ensure resiliency, an instance of the web-
service runs on each cluster host. When a client request arrives the
web-service creates a MapReduce job to fulfill client requests. The
reducer function component of the MapReduce job runs within the
web- service handling the request.
The requirements listed in the post are too high level to understand why building their own solutions was better. But if it works for them, that’s OK. Just keep in mind that NIH and distributed systems don’t always mix well.
Original title and link: Big Data at Torbit: Custom MapReduce-like System ( ©myNoSQL)