As the virtualization seeds begin to sprout throughout IT organizations everywhere, there is a growing interest in being able to accurately measure and report on the performance of a virtualized platform. Back in June, we discussed how VMware was relaxing their benchmarking policy to allow users to conduct internal performance testing and benchmarking studies, the results of which could be published as long as V As the virtualization seeds begin to sprout throughout IT organizations everywhere, there is a growing interest in being able to accurately measure and report on the performance of a virtualized platform.Back in June, we discussed how VMware was relaxing their benchmarking policy to allow users to conduct internal performance testing and benchmarking studies, the results of which could be published as long as VMware reviewed and approved the methodology.Fast forward to September, VMware stated that plenty of benchmarks exist to measure the performance of physical systems, but they fail to capture essential aspects of virtual infrastructure performance. They said we need a common workload and methodology for virtualized systems so that benchmark results can be compared across different platforms. I agree with that sentiment wholeheartedly. VMware says there are a number of unique challenges in creating sound and meaningful benchmarks for virtualized systems: Capture the key performance characteristics of virtual systems Ensure that the benchmark is representative of end user environments Make the benchmark specification platform neutral Define a single, easy to understand metric Provide a methodical way to measure scalability so that the same benchmark can be used for small servers as well as larger serversAccording to VMware, having an industry standard way of comparing virtualized solutions will allow users to make more informed decisions regarding the entire stack of virtualization technology. Such a standard can also drive improvements in future hardware and software, again benefiting the industry. For these reasons, VMware is committed to solving this problem. For a while now they’ve been working on just such a benchmark. They refer to this benchmark as VMmark (Virtual Machine benchMark) and they plan on presenting it this month at VMworld 2006.Last month, the IDG news service reported that the Standard Performance Evaluation Corporation (SPEC) was forming a group to discuss whether virtualization benchmarks are needed and, if so, to set those benchmarks over an estimated three-month process. Walter Bays, president of SPEC, said “It seems like a number of our member companies have simultaneously agreed they are ready to move on it and form a working group.” Bays believes that because many IT managers at companies are still learning what virtualization is, benchmarks would help educate them. SPEC is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from their member organizations and other benchmark licensees.While VMware remains the leader in the virtualization space, benchmarks are needed to compare all types of virtualization software. These benchmarking standards should therefore be vendor neutral and be able to compare virtualization software from multiple vendors. The IDG news service reports that SPEC’s working group will include representatives from Intel, AMD, Fujitsu Siemens Computers, VMware and other major vendors. Additionally, SPEC would also like to invite end users to participate in the process. Software Development