VMware introduces VMmark 2.0 beta

If Mark Twain were alive today, he might have popularized yet another saying for those of us in the computer industry: “Lies, damned lies, and benchmarks.”

Benchmark results are often met with skepticism and controversy because of concerns that the results can be manipulated to make one vendor or platform appear better than another.

[ Also on InfoWorld: Dell, HP to certify and resell Oracle VM virtualization | Dell has also released a new virtualization-based secure browser, thanks to its KACE acquisition. ]

Yet when organizations are either implementing or evaluating virtualization platforms for their environments, they need a solution that helps them compare performance and scalability of these different platforms, make the right hardware choices, and then measure platform performance against some base level an ongoing basis. To do all of that, they need some sort of benchmark solution.

Unfortunately, the benchmark solutions that many IT shops use today in the physical world don’t translate well into a virtual machine environment. These physical server benchmark solutions operate on the old idea of a single workload per server, and therefore don’t have the proper understanding of server consolidation or simultaneously running multiple workloads on a single server.

VMware was the first to try its hand at addressing that challenge with the launch of its proprietary VMmark benchmark solution back in July 2007. Although extremely useful, it wasn’t necessarily the easiest thing to use and earned a reputation for taking too long to run and being too onerous on its test bed hardware requirements.

In addition, it was surrounded by a bit of controversy around the publishing and then the challenging of vendor and competitor benchmark results. Specifically, VMware and Citrix have had numerous public debates around published results and benchmark scores, challenging validity because of issues such as configuration or platform version used in the testing.

In October 2006, the nonprofit group Standard Performance Evaluation Corp. (SPEC) formed a working group in order to develop a new vendor-neutral standard benchmark for measuring virtualization performance. SPEC set out to measure the performance of data center servers used for virtualized server consolidation and to include options for measuring power consumption and power/performance relationships. It took nearly four years in the making, but SPEC finally released its vendor-neutral virtualization benchmark solution, dubbed SPECvirt_sc2010, in July 2010 — three years after VMware’s VMmark hit the scene.

SPEC’s virtualization benchmark was developed by the SPEC virtualization subcommittee, whose members and contributors included AMD, Dell, Fujitsu, HP, IBM, Intel, Oracle, Red Hat, Unisys, and VMware. Conspicuously missing from that list were the hypervisor platform vendors Citrix, Microsoft, and Parallels. While these three vendors are all members of the SPEC Consortium, they do not appear to have participated as subcommittee members or contributors to the benchmark according to the press announcement made by SPEC. If these companies did participate, why weren’t they mentioned? And are they throwing support behind this new SPEC virtualization benchmark?

And if they didn’t participate in the process, the question becomes, why not? If these three major hypervisor vendors had no say in the SPEC benchmark, can it really be trusted as vendor-neutral when comparing Hyper-V, XenServer, and Parallels hypervisors?

Even though VMware participated in the creation of the new SPEC virtualization benchmark, that didn’t stop the virtualization giant from also working on a new version of its own proprietary benchmark offering. VMmark 2.0, currently in beta, is being called a next-generation multihost virtualization benchmark that models not only application performance in a virtualized environment but also the effects of common virtual infrastructure operations.

So while SPECvirt_sc2010 and VMmark 1.x are both good at measuring the performance of a single virtualization host, VMmark 2.0 is being designed to benchmark the entire virtual data center. This implies measuring complex operations like manual (or user-initiated) vMotion, automated (or DRS-initiated) vMotion, Storage vMotion, as well as virtual machine cloning and deployment.

In moving from a single- to a multihost virtualization benchmark, VMware is constructing VMmark 2.0 with the following properties in mind:

• Multihost to model realistic data center deployments
• Virtualization infrastructure workloads to more accurately capture overall platform performance
• Heavier workloads than VMmark 1.x to reflect heavier customer usage patterns enabled by the increased capabilities of the virtualization and hardware layers
• Multitier workloads driving both virtual machine-to-virtual machine and external network traffic
• Workload burstiness to ensure robust performance under variable high loads

Organizations are at different places in their virtualization maturity, so benchmarking a single server still has a place in today’s virtualized data center. But VMware’s idea of looking at the data center as a whole definitely has its place as well. More mature virtualization shops will find this new benchmark inviting, as will infrastructure-as-a-service cloud hosting providers.

So in 2010, VMmark and SPEC virtualization benchmarks both have their place. But for the cost conscious, there is one other difference between these two virtualization benchmarking solutions: VMware VMmark is free, while the SPECvirt_sc2010 will cost you $3,000, though discounts are available for qualified nonprofit and educational institutions.

This story, “VMware introduces VMmark 2.0 beta,” was originally published at InfoWorld.com. Follow the latest developments in virtualization and cloud computing at InfoWorld.com.