In the light of VMware's apparent policy reversal on the issue of virtual machine benchmarking, the company's senior director of enterprise and technology marketing Andrea Eubanks explains the company's stand on VM benchmarking and the publication of results.
Q: Let's talk first about VMware's involvement with SPEC to help develop a public virtualisation benchmark. What's the latest? A: We've started working with open standard benchmarks as you know. Then in October 2006, at our request SPEC formed a working group to measure VM performance. By March 2007, we'd agreed design goals and plans for an industry standard benchmark. It's now being developed by a subcommittee, chaired by one of our people. Participants include VMware's major partners and competitors, including AMD Dell, Fujitsu Siemens, HP, Intel, IBM, Microsoft, Red Hat, Sun, and SWsoft.
It takes a broad view of virtualisation -- it's handling OS virtualisation and hardware partitioning. We expect it to generate a benchmark using an open philosophy. We've no timescale yet as we're at the design rather than the debugging stage.
Q: VMware has recently released a beta of VMmark, a free VM benchmark. What prompted that? A: It's about determining VM configuration, the number and size of VMs, such as their memory use. Unlike SPEC, it also specifies the software stack -- ie application and OS.
Our benchmark policy is orthogonal to VMmark. People care because it allows them to see the capacity of a set of hardware on a pre-determined hardware/software setup. It's not a capacity planning tool but it does show the mileage you'll get from a particular setup.
Q: Why does your EULA prohibit the publishing of benchmark results? A: We endorse benchmarking but we want people to follow proper procedures. There's not much known about benchmarking VMs.
For example, how do you make an industry standard benchmark map onto virtualisation? Take TPCC -- you modify it, run multiple workloads, how do you get meaningful numbers? Our EULA provides guidance on how to do this and we want to help people get the right numbers. We want people to use VMmark and learn how to benchmark virtual systems.
We're considering changing the EULA to make it more effective - for example, in education. We'll allow academics to publish benchmarks without restrictions because they're not commercial entities. Also, people can apply to publish a benchmark and we approve three to four benchmarks per week.
And the EULA will change. We'll make it easier to benchmark when using hardware virtualisation. We'll allow the publication of benchmark results as long as you disclose the environment and content of the test.
Q: Given how fundamental benchmarking is to the evaluation process, why has it taken so long to come up with a benchmark? A: It's timely with today's market maturity and adoption. We're leaders in the technology and are seeing strong adoption in VMware echo chamber -- there's huge market adoption.
We found early customers running benchmarks would run mixed environments when comparing physical to virtual. They made mistakes and we had to step in and help. For example, people would run clients in virtual machines, which perturbs the benchmark. We now have standard helpful hints and tips to help people simulate the loads without undermining their benchmarking.
It's about educating not controlling. We started in academia and are all about looking for the truth. That's the kind of company we are.