Benchmarking File Systems and Storage

Avishay Traeger from the IBM Haifa Research Lab and Erez Zadok from Stony Brook University are raising awareness of issues relating to proper benchmarking practices of file and storage systems.  They hope that with greater awareness, standards will be raised, and more rigorous and scientific evaluations will be performed and published.

In May 2008 they published a paper in the ACM Transactions on Storage entitled “A Nine Year Study of File System and Storage Benchmarking’” in which they surveyed 415 file system and storage benchmarks from 106 papers that were published in four highly-regarded conferences (SOSP, OSDI, USENIX, and FAST) between 1999 and 2007.  They found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system’s true performance.  They have provided a set of guidelines that they hope will improve future performance evaluations. An updated version of the guidelines is available.

Traeger and Zadok have also set up a mailing list for information on future events, as well as discussions.  More information can be found on their File and Storage System Benchmarking Portal
http://fsbench.filesystems.org/.

IT Benchmark Spreadsheets

ServiceXen, an IT firm located in Atlanta, Georgia, has provided six (6) interactive spreadsheets to assist in IT benchmarking activities.  Each spreadsheet is a shared Zoho Sheet.  See below:

1. Data Center Security Audit
2. New Employee Cost Calculator
3. Server Buy vs. Lease Calculator
4. Total Cost of Ownership (TCO) Calculator
5. Virtualization Fit Tool

Metric for Server Evaluation: Space, Watts and Power

Sun Microsystems

SWaP (Space, Watts and Power) is the new standard for calculating server efficiency before you buy. This metric that allows you to calculate the impact of a server in your data center.

How SWaP Works

Server A Server B Server A Difference Performance 500 operations 500 operations Equal Space 2RU 4RU x2 smaller Power 300 Watts 800 Watts x2.7 less SWaP Rating 0.83 0.16 x5.2 more

In the example above, Server A and Server B produce equal performance, however Server A is half the size and less than half the power of Server B. Using the SWaP formula, it is revealed that Server A is over 5X more efficient than Server B–providing a huge impact to rack dense deployments in your data center.

Source: SWaP (Space, Watts and Performance) Metric, Sun Microsystems, 2008

In a report published by David Greenhill, Distinguished Engineer at Sun Microsystems, (Space Watts and Power), he indicates that older model servers can consume 45 times the energy of  newer models.  In his case study the 1997 server used 13,456 watts / hr. compared to only 300 watts / hr. for the 2005 server.    But even 300 watts/hr. (0.3 Kw/hr.) add up quickly when you realize you’re typically running it 7/24/365.  In fact, each 300 w server uses about 2600 Kw per year.  Then you factor in the hundreds or even thousands of servers most large organizations are using and the air conditioning, UPS, lighting, etc. to house them, and you can start to see why less is definitely more.

Source: “Green IT Innovation – Data Center Efficiencies to Reduce Cabon Footprint – Part 1.”, Blog – Sharing Sustainability Innovation, December 19, 2008, http://globeforum.wordpress.com

Benchmark: Virtualization at Amazon

Amazon Virtualization

Virtualization Benchmark

Amazon sold storage to external customers for 15 cents/GB/month (estimated).

Bechtel’s internal storage costs were $3.75/GB/month.

WHAT BECHTEL LEARNED: Amazon could sell storage cheaply, Ramleth believes, because its servers were more highly utilized.

Source: CIO Magazine, Bechtel’s New Benchmarks, October 24, 2008.

Microsoft – Power Usage Effectiveness for Data Center

Microsoft Data Center

Microsoft has set some benchmarks for their Data Center of the future.   “A key driver is our goal to achieve a PUE (power usage effectiveness) at or below 1.125 by 2012 across our data centers”, said Michael Manos, general manager of global foundation services at Microsoft.

Source: http://loosebolts.wordpress.com/2008/12/02/our-vision-for-generation-4-modular-data-centers-one-way-of-getting-it-just-right/

Benchmark – Data Center Staffing and Energy Usage

Microsoft will staff its 500,000-square-foot Chicago data center with only 35 people. Read more about Microsoft’s new data center.

Source:‘How to Staff Your Next Data Center’, CIO Magazine, June 10, 2008.

Data centers built for today’s equipment range from 150 to 300 watts per square foot.
Source: ‘The 5 Pitfalls of Data Center Consolidation and Relocation”, CIO Magazine, November 19, 2008.