Code: Spark-Bench
Spark-Bench is a ground-up rewrite I did of a benchmarking suite pioneered by folks at IBM Research.
While researchers are great at researching, they also have a strong affinity for bash and Java.
The ground up rewrite moved multiple independent maven projects into one, central Scala project built using SBT.
All configuration was moved from a scattering of bash variables to one, centralized config file.
Spark-bench is now more than just a benchmarking suite: it's a flexible platform for simulation and comparison of Spark use cases of all sorts.