The good and bad of storage benchmarking

In my post last week, I talked about new record-breaking SPC-1 performance benchmark results for TMS’s new RamSan-630. Judging by the email response, that lead many of you to wonder exactly what the Storage Performance Council is and what makes those results any more interesting than other third-party storage benchmarks.

In a field as complex as enterprise storage, building testing mechanisms that accurately reflect real life and provide any real value to end-users is fantastically difficult. With such an incredibly wide range of enterprise storage workloads and products, it’s impossible to build a benchmark that has any hope of resembling all of them.

That said, the testing mechanisms used by the SPC are among the best — not only due to the thoroughness of the tests themselves, but because the documentation and auditing required to submit a public result is quite elaborate. In fact, the documentation is often more interesting to a prospective storage buyer than the performance results themselves.

Who is the SPC?

The Storage Performance Council is made up of a wide cross-section of storage vendors, academics, and industry analysts. The SPC’s goal is to serve as a catalyst for performance improvement in storage subsystems by fostering a free and open exchange of ideas and information while ensuring fair and rigorous competition between storage vendors. It has managed to attract a wide range of vendors, including NetApp, HP, Dell, Hitachi, Fujitsu, and IBM.

But the SPC is not without controversy. Long absent from the roster is EMC, one of the largest storage vendors. If you want to know how EMC felt about the SPC a few years ago, check out this flaming review of its weaknesses penned back in 2007 by EMC Global Marketing CTO, Chuck Hollis.

But now EMC has decided to sign on to the SPC, a decision driven by several interesting factors. In a response to an article posted at the Register in February, Hollis said that EMC’s entrance into the SPC was in part to prevent other SPC members from testing EMC hardware without the company’s consent (something that NetApp famously did back in 2008) and a realization that some governmental bodies are starting to require publically posted and audited SPC-1 test results to consider storage products for purchase.

EMC’s entrance into the SPC further bolsters the organization’s credibility. Plus, more disclosure from storage vendors is always a good thing.

What are the tests?

The SPC has collaboratively developed two primary testing and documentation mechanisms, and a third is still in development. The first and most popular are the often referenced SPC-1 and SPC-1/E tests. These tests attempt to mimic an OLTP (Online Transaction Processing) workload that is common to storage workloads such as high-performance databases. For these tests, the most critical results include the total system cost, number of SPC-1 IOPS it could achieve, and a basic cost per SPC-1 IOPS.

The “E” in the SPC-1/E test reflects the industry’s increasing gravitation toward showing their energy efficiency. This test variant extends the basic testing to include annual energy costs and number of SPC-1 IOPS per watt of power consumed.

Unlike the SPC-1 and 1/E tests, the less-quoted SPC-2 test attempts to reflect a workload that requires massive amounts of largely sequential storage throughput. These types of workloads are dramatically less common in the wild, but include applications that involve large amounts of high-bit-rate video and certain types of data warehousing. These tests result in a composite of large file processing, large database query, and video-on-demand figures that result in an aggregate throughput and price-to-performance figures reflected in raw megabytes per second.

Why is any of this useful?

On the face of it, it would seem that the only thing these benchmarks are good for is bragging rights for the vendors to say “we’re the fastest.” There’s certainly evidence along those lines, as this recent press release from Texas Memory Systems attests.

But the elephant in the room is that the vast majority of us in the enterprise storage landscape aren’t actually interested in performance — at least not primarily. Sure, raw performance is a big deal, but unless you’re running a high-frequency trading rig for a firm on Wall Street, it isn’t the biggest deal. In most cases, a storage vendor’s support reputation, the balance of available features such as snapshots and replication, ease of management, operational costs, and cost per capacity will play a much larger role in purchasing decisions.

Public SPC results can certainly give you an idea of the performance you can expect from a vendor’s solution as it was configured for the test, but it doesn’t really tell you anything about the other important factors you might be interested in. So the top-line SPC performance results might influence your purchasing decision to some extent, but they certainly aren’t a one-stop shop that will answer the often asked question: Which type of storage is right for me?

From my perspective, the real value of the SPC results is not in the raw IOPS or megabytes per second that they were able to post, but more in how the vendor actually orchestrated the test — since the tests are usually carried out by the vendors themselves (and then audited by other SPC members to ensure that no corners were cut). That might sound like it’s a recipe for a slanted result, but the configuration disclosure requirements go a long way toward preventing that.

In the results, you get to see not only how the test was carried out, but precisely which part numbers were included, how much they cost, and an acknowledgement of what the support costs were. That information by itself is worth its weight in gold when getting a feel for how different vendors price and sell their products. It’s very rare that you get that kind of visibility into what you’re actually paying for.

If you spend enough time reading through the test results, you can learn a lot. You’ll quickly realize that many vendors don’t include licensing for optional features such as snapshots, thin provisioning, and replication in their as-tested configurations. This result for a 3PAR F400 from a few years ago is a great example. If memory serves me, 3PAR arrays have six or seven different features that are unlocked through fairly expensive capacity-based software licenses. The test config included one of those — and the price would probably double were you to include all of them.

The truth is that SPC tests are no different than any other benchmarks — with the notable exception that they adhere to uniform rules and the testers have to show their math. They are much, much better than the usual vendor-sponsored head-to-heads or drive-by stats in marketing glossies. If you spend enough time reading them, you can learn a massive amount of information about how the largest storage vendors do business. And that’s more than worth the price of admission.

This article, “The good and bad of storage benchmarking,” originally appeared at InfoWorld.com. Read more of Matt Prigge’s Information Overload blog and follow the latest developments in storage at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.