Benchmarking File Systems and Storage

Avishay Traeger from the IBM Haifa Research Lab and Erez Zadok from Stony Brook University are raising awareness of issues relating to proper benchmarking practices of file and storage systems.  They hope that with greater awareness, standards will be raised, and more rigorous and scientific evaluations will be performed and published.

In May 2008 they published a paper in the ACM Transactions on Storage entitled “A Nine Year Study of File System and Storage Benchmarking’” in which they surveyed 415 file system and storage benchmarks from 106 papers that were published in four highly-regarded conferences (SOSP, OSDI, USENIX, and FAST) between 1999 and 2007.  They found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system’s true performance.  They have provided a set of guidelines that they hope will improve future performance evaluations. An updated version of the guidelines is available.

Traeger and Zadok have also set up a mailing list for information on future events, as well as discussions.  More information can be found on their File and Storage System Benchmarking Portal
http://fsbench.filesystems.org/.

Apache Server – Requests per Second

This post on www.ilovebonnie.net documents some impressive system performance improvements by the addition of Squid Cache (a caching proxy) and APC Cache (opcode cache for PHP).
* Apache is able to deliver roughly 700% more requests per second with Squid when serving 1KB and 100KB images.
* Server load is reduced using Squid because the server does not have to create a bunch of Apache processes to handle the requests.
* APC Cache took a system that could barely handle 10-20 requests per second to handling 50-60 requests per second. A 400% increase.
* APC allowed the load times to remain under 5 seconds even with 200 concurrent threads slamming on the server.
* These two caches are easy to setup and install and allow you to get a lot more performance out of them.

The post has an in-depth discussion and a number of supporting charts. The primary point is how simple it can be to improve performance and scalability by adding caching.

Source: http://www.ilovebonnie.net/2009/07/14/benchmark-results-show-400-to-700-percent-increase-in-server-capabilities-with-apc-and-squid-cache/

Drupal Benchmarked on Amazon ec2

According to a benchmark test run by John Quinn &  Cailin Nelson,

Drupal systems perform very well on amazon ec2, even with a simple single machine deployment. The larger hardware types perform significantly better, producing up to 12,500 pages per minute. this could be increased significantly by clustering as outlined here.  The apc op-code cache increases performance by a factor of roughly 4x.  The average response times were good in all the tests. The slowest tests yielded average times of 1.5s. again, response times where significantly better on the better hardware and reduced further by the use of apc.

Amazon uses Xen based virtualization technology to implement ec2. The cloud makes provisioning a machine as easy as executing a simple script command. when you are through with the machine, you simply terminate it and pay only for the hours that you’ve used. ec2 provides three types of virtual hardware that you can instantiate.

Source: John & Cailin Blog, “lamp performance on the elastic compute cloud: benchmarking drupal on amazon ec2″, January 28, 2008.

IT Benchmark Spreadsheets

ServiceXen, an IT firm located in Atlanta, Georgia, has provided six (6) interactive spreadsheets to assist in IT benchmarking activities.  Each spreadsheet is a shared Zoho Sheet.  See below:

1. Data Center Security Audit
2. New Employee Cost Calculator
3. Server Buy vs. Lease Calculator
4. Total Cost of Ownership (TCO) Calculator
5. Virtualization Fit Tool

Metric for Server Evaluation: Space, Watts and Power

Sun Microsystems

SWaP (Space, Watts and Power) is the new standard for calculating server efficiency before you buy. This metric that allows you to calculate the impact of a server in your data center.

How SWaP Works

Server A Server B Server A Difference Performance 500 operations 500 operations Equal Space 2RU 4RU x2 smaller Power 300 Watts 800 Watts x2.7 less SWaP Rating 0.83 0.16 x5.2 more

In the example above, Server A and Server B produce equal performance, however Server A is half the size and less than half the power of Server B. Using the SWaP formula, it is revealed that Server A is over 5X more efficient than Server B–providing a huge impact to rack dense deployments in your data center.

Source: SWaP (Space, Watts and Performance) Metric, Sun Microsystems, 2008

In a report published by David Greenhill, Distinguished Engineer at Sun Microsystems, (Space Watts and Power), he indicates that older model servers can consume 45 times the energy of  newer models.  In his case study the 1997 server used 13,456 watts / hr. compared to only 300 watts / hr. for the 2005 server.    But even 300 watts/hr. (0.3 Kw/hr.) add up quickly when you realize you’re typically running it 7/24/365.  In fact, each 300 w server uses about 2600 Kw per year.  Then you factor in the hundreds or even thousands of servers most large organizations are using and the air conditioning, UPS, lighting, etc. to house them, and you can start to see why less is definitely more.

Source: “Green IT Innovation – Data Center Efficiencies to Reduce Cabon Footprint – Part 1.”, Blog – Sharing Sustainability Innovation, December 19, 2008, http://globeforum.wordpress.com

Benchmark: Virtualization at Amazon

Amazon Virtualization

Virtualization Benchmark

Amazon sold storage to external customers for 15 cents/GB/month (estimated).

Bechtel’s internal storage costs were $3.75/GB/month.

WHAT BECHTEL LEARNED: Amazon could sell storage cheaply, Ramleth believes, because its servers were more highly utilized.

Source: CIO Magazine, Bechtel’s New Benchmarks, October 24, 2008.

Benchmark: number of servers per adminstrator

Does a thumb rule exists for “server administrators to a number of servers for a large and distributed enterprise infrastructure.”

I know this cannot be applied in every situation, but is there some reference than can be used to derive a head count for 24×7 support operations.
Answers:

1) check out the paper: “How Many Administrators are Enough” http://www.verber.com/mark/sysadm/how-many-admins.html

2) If there is a remote mangement tool to access the same. then the number is around 200-300.
if no software is present and physical present is required, then not more than 50-100 servers per server administrator. I am assuming these servers to be windows servers.
in case of Unix boxes, more servers can be managed even without management software, as patches to be applied are less often and they hardly need to be rebooted. so zero admin scenarios can exisit with unix boxes.
3) Administrators vs no. of servers depends on the stability of the organization and the stability of the OS being used. I have driven an initiative where I could drive the ratio from 150 per SA to 900 per SA. This involved a lot of standardization on the OS level as well as a focus effort to reduce known problems and working on eradication of the problems rather than break fix efforts. A lot of focus was also driven on the quality and training of the SA’s.

Source: LinkedIn Answers (March 2008) http://www.linkedin.com/answers/hiring-human-resources/staffing-recruiting/HRH_SFF/184490-5730787?browseCategory=&goback=.ahp

Benchmark – number of system adminstrators per server

Google employs one systems administrator for about 20,000 servers.

Bechtel employed one systems administrator per 100 servers.

Source: ‘Bechtel’s New Benchmarks’, CIO Magazine, October 24, 2008