Hadoop vs PIG vs Vertica for Counting Triangles
Very interesting post on Vertica blog, comparing the solution of counting triangles using Hadoop, PIG, and Vertica. As you’d expect, Vertica shows the best results, but this is still a nice example of using different tools for solving a problem. Plus all code is available on GitHub.
PIG beat my Hadoop program, so my colleague who wrote the PIG script earned his free lunch. One major factor is PIG’s superior join performance – its uses hash join. In comparison, the Hadoop solution employs a join method very close to sort merge join.
Vertica’s performance wasn’t even close to that of Hadoop – thankfully. It was much much better. In fact Vertica ate PIG’s and Hadoop’s lunch – its best time is 22x faster than PIG’s and 40x faster than the Hadoop program (even without configuration tweaks Vertica beats optimized Hadoop and PIG programs by more than a factor of 9x in comparable tests).
Here are a few key factors in Vertica’s performance advantage:
- Fully pipelined execution in Vertica, compared to a sequence of MR jobs in the Hadoop and PIG solutions, which incurs significant extra I/O. We quantify the differences in how the disk is used among the solutions below in the “disk usage” study.
- Vectorization of expression execution, and the use of just-in-time code generation in the Vertica engine
- More efficient memory layout, compared to the frequent Java heap memory allocation and deallocation in Hadoop / PIG
The conclusion is interesting too:
Overall, Hadoop and PIG are free in software, but hardware is not included. With a 22x speed-up, Vertica’s performance advantage effectively equates to a 95% discount on hardware. Think about that. You’d need 1000 nodes to run the PIG job to equal the performance of just 48 Vertica nodes, which is a rack and a half of the Vertica appliance.
Original title and link: Hadoop vs PIG vs Vertica for Counting Triangles (©myNoSQL)
