<A HREF="http://ad.doubleclick.net/jump/idg.us.cw.snw/whats_new;abr=!ie;pos=top;sz=468x60;tile=1;u=2341;ord=303443.282770157?"><IMG SRC="http://ad.doubleclick.net/ad/idg.us.cw.snw/whats_new;abr=!ie;pos=top;sz=468x60;tile=1;ord=303443.282770157?" border=0 height="60" width="468">

Subscribe
|   About Us   |   Resources

Please provide the
following information:

Your name:

Your E-mail:

Recipient's E-mail:

Message:

ARTICLE POSTED January 14th,
2002

Storage
Performance Council offers benchmark performance tests

By Elisabeth Horwitt

After more than four years of hard work,
and a fair amount of dickering, a consortium of leading
storage vendors has introduced the first industry-standard
benchmark for measuring performance on both direct-attach and
networked storage systems.

The Storage Performance Council (SPC)
hopes to ensure a level playing field by providing vendors,
VARs and IT managers with a standardized, vendor-neutral
methodology for measuring and comparing performance on storage
subsystems, says Roger Reich, a senior technical director at
Veritas Software, an SPC member based in Mountain View,
CA.

Analysts and IT sources agree that the
storage industry desperately needs a standardized benchmark,
particularly one that helps buyers make meaningful comparisons
on the enterprise level. Corporate IT managers are struggling
to make sense of the new generation of storage products and
increasingly complex storage networking configurations,
declares Chris Eidler, CTO at Storageway, Inc., a storage
utility service provider in Fremont, CA.

According to Eidler, "My experience with
benchmarks is that each vendor can come up with its own
configuration that's contrived to show its products to best
advantage, so you end up with a bunch of numbers that don't
mean anything."

"Measuring storage performance is a very
difficult feat," adds Dave Hill, a research director at
Aberdeen Group, an IT consultancy in Boston. "The SPC had to
work through a lot of issues, which is why it took so long.
The benchmark will enable buyers to determine not just what is
the top-of-the-line product but also what's sufficient to meet
their particular needs."

SPC-1 emulates TPC
model
Announced December 3 at the Computer Measurement
Group conference, SPC-1 is modeled after the highly successful
Transaction Processing Performance Council (TPC) benchmarks.
It comprises two testing environments:

SPC-1 I/O Operations Per Second
(IOPS). This benchmark uses a multithreaded Workload
Generator to emulate enterprise-class multiuser I/O
applications such as database/online transaction processing
and mail servers. "IOPS tests how much work a storage
subsystem can produce and how well it can it keep up with
the load at peak traffic," Reich states.

Least Response Time (LRT). This
benchmark tests performance on sequential I/O processing
environments that support applications such as
backup/restore and rebuilding a big database. "What's at
stake here is how long it takes to clock completion time for
an application that is made up of a series of I/O requests,"
Reich notes.

While SPC-1 does not directly measure the
performance of a SAN switch, it does enable users to measure
end-to-end performance between SAN-connected hosts and
subsystems. It also provides the ability to test overall
throughput and to pinpoint bottlenecks, using a quantifiable,
sustainable workload. Users or vendors can use the SAN
company's own monitoring tools to break out specific SAN
capacity and performance numbers. A special version of the
benchmark kit, planned for 2002, will allow test sponsors to
dial settings "so you can more precisely exercise individual
SAN components and storage subsystems," Reich says.

This capability is important if the
benchmark is to provide meaningful data about SANs, observes
Hill. "When you're dealing with a storage network, you're
dealing with a lot of different components," he says. "You
want to use a benchmark to discover the potential bottlenecks
but also to do apples-to-apples comparisons between different
storage and SAN products."

An SPC working group has begun the
development of a benchmark that measures performance on NAS
storage subsystems. That benchmark is scheduled for release
sometime next year.

Getting users – and vendors –
involved
The first task, however, is to get the
standard into wide use, not only among vendors, but also among
corporate users, VARs and systems integrators. "What we want
is that when a vendor gets an RFQ from someone like Chase, it
would specify a minimum level of performance in accordance
with the SPC benchmark," states SPC chairperson Leah Schoeb of
Sun Microsystems.

Before big corporate customers start
putting the benchmark in their RFQs, however, the standard has
to generate a critical mass of industry support. Once this
happens, "I think we would ask vendors we're talking to if
they are using the benchmark," Eidler notes. "And if they said
no, they would have to explain why."

The SPC has initially addressed this
chicken-and-egg problem by ensuring that most of the major
storage industry players support and actively participated in
SPC-1. Current SPC members include Sun, Dell, Adaptec, Compaq,
Hitachi, IBM, LSI Logic, NEC, Unisys and Veritas.

Hewlett-Packard strongly supports the
benchmark, but the company is too consumed by its own
reorganization to participate in the introduction, Schoeb
says.

EMC: "Arbitrary benchmarks like this
are irrelevant"
The one major missing player is EMC,
which withdrew from the SPC in early 2000. It did so based on
the "irrelevance of the scope of the test to customers' actual
performance requirements," an EMC spokesman says, adding,
"Arbitrary benchmarks like this are irrelevant unless they can
closely replicate a customer's actual environment. SPC-1 uses
a random I/O generator and focuses on performance of a
specific back-end storage device. The resulting numbers are
easily manipulated to create the illusion of fast storage
performance."

For example, a vendor could beef up the
performance numbers of a low-end subsystem by coupling it with
a high-powered server, EMC asserts.

The SPC claims to have already addressed
these concerns. "We spent four years arguing and compromising
to ensure that the benchmark doesn't play to one or another
vendor's strengths," states Schoeb.

The IOPS and LRT benchmarking
environments simulate real, widely used application
environments. And vendors can't publish SPC benchmark data
without a comprehensive description of the host, SAN and
storage configuration used.

Furthermore, the SPC has set up a
stringent auditing and review process to ensure that SPC-1
benchmark test results are, in Reich's words, "credible,
viable data that users can trust."

Test sponsors must turn in
full-disclosure reports that document in detail the storage,
host and network configuration and activities that produced
the benchmark result. Results must be repeatable and
sustainable over a test run of several hours.

Finally, benchmark results must go
through a 60-day peer review process by SPC members. This
gives other vendors a chance to challenge test results that
they believe are skewed.

Official benchmark results will be
published at http://www.Storageperformance.org.

Anyone can test anyone
Anyone
can run SPC-1 tests on a vendor's product and publish the
results, with or without that vendor's permission, as long as
they go through the necessary steps.

The SPC designed the benchmark test kit
to be as simple as possible, to make it accessible by
companies with limited IT resources and to encourage vendors
to use it early and often. Major vendors will probably produce
one official set of benchmark results to demonstrate their
products' prowess, as well as additional benchmarks that
evaluate performance for a particular customer's
configuration, Reich predicts. The goal is to cover the
spectrum of configurations and applications so that virtually
any company can find published benchmark data that applies to
its particular needs.

Configuration setup will average between
a few hours and a few days, depending on the complexity of the
storage subsystem and configuration. Execution of the
benchmark – minus restarts, tuning and multiple iterations –
should average between six and eight hours.

The SPC-1 Benchmark Kit is available for
download on the Storageperformance.org site. It runs on a
variety of host server platforms and storage network
topologies. Two council members – The Evaluator Group and
Ideas International – are available to help analyze benchmark
results.

mailto:[email-masked]
mailto:[email-masked]
mailto:[email-masked]

<A HREF="http://ad.doubleclick.net/jump/idg.us.cw.snw/whats_new;abr=!ie;pos=skyscraper;sz=120x800;tile=2;u=2341;ord=303443.282770157?"><IMG SRC="http://ad.doubleclick.net/ad/idg.us.cw.snw/whats_new;abr=!ie;pos=skyscraper;sz=120x800;tile=2;ord=303443.282770157?" border=0 height="800" width="120">

Sponsored
Links:

Storage
Knowledge Center

Computerworld's
vast collection of resources on Storage