If you thought vendors couldn't get benchmark performance numbers right, just wait. Given the power and cooling issues data centres face, performance per watt will soon be the new buzzword. Intel's release of a new low-power Xeon, whose performance per watt Intel emphasises, underlines the trend. The problem is there are no uniform benchmarks or standards to make such numbers meaningful - and there may never be.
Jon Koomey, consulting professional at Stanford and staff scientist at Lawrence Berkeley Laboratory, says the problem is how to use metrics that are meaningful to real-world data centre managers. Even performance benchmarks are a problem, he says, let alone performance per watt. "Michael Patterson [a thermal engineer] at Intel said 'we cant even get the chip folks within our own company to agree on the metric,'" says Koomey, who worked on the early industry "watts per square foot" data center studies in 2000/2001 and participated last month at the EPAs Enterprise Servers and Data Centers: Opportunities for Energy Savings Virtual Conference.
But that won't stop vendors from trying. Last month at the EPA conference, Sun announced an initiative to develop performance per watt benchmarks. "Sun is pushing to bring industry players together to talk about metrics. We will have a meeting bringing together the key industry stakeholders and it will be a technical meeting [to develop] a set of draft benchmarks. Its on the agenda," Koomey says. That meeting will take place this spring.
Is it possible to come up with benchmarks that satisfy a broad set of users and give results that are meaningful in the real world? "On the one side there are the people who point to the complexities and say those will make it difficult to get meaningful metrics," says Koomey.
He's not in that camp.
"These questions of metrics are bigger than the energy question. They are questions that the computer industry always struggles with. Just because its difficult doesn t mean we can't do it," he says.
Many different metrics are used today to measure such things as the capacity of a server to serve up Web pages or perform other tasks. Koomey envisions taking a suite of such task-specific benchmarks and tweaking them to show performance per watt. "Users, all of whom have different requirements, can decide which of the metrics is right for their application," he says. "Those benchmarks would be likely to save energy in a lot of applications," he says.
Let the benchmark wars begin.