Avishay Traeger from the IBM Haifa Research Lab and Erez Zadok from Stony Brook University are raising awareness of issues relating to proper benchmarking practices of file and storage systems. They hope that with greater awareness, standards will be raised, and more rigorous and scientific evaluations will be performed and published.
In May 2008 they published a paper in the ACM Transactions on Storage entitled “A Nine Year Study of File System and Storage Benchmarking’” in which they surveyed 415 file system and storage benchmarks from 106 papers that were published in four highly-regarded conferences (SOSP, OSDI, USENIX, and FAST) between 1999 and 2007. They found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system’s true performance. They have provided a set of guidelines that they hope will improve future performance evaluations. An updated version of the guidelines is available.
Traeger and Zadok have also set up a mailing list for information on future events, as well as discussions. More information can be found on their File and Storage System Benchmarking Portal