It started out as a way to save money when Gannett was adding new servers back in 2002. Now, says Eric Kuzmack, IT architect at the newspaper publisher, "we have a couple of hundred virtual machines in our data centre." Virtualisation technology has increased IT staff efficiency by allowing virtual servers to be deployed in days instead of the weeks required to provision physical machines.

But as virtual machine technology moves out of development labs and into production server environments in large numbers, some administrators are finding that the growth of virtual servers is getting ahead of the tools available to effectively manage them.

Existing server-monitoring tools are increasingly aware of virtual servers, but most aren't yet sophisticated enough to interpret feedback in a virtual machine context -- much less act on it. "They don't take into account the particulars of virtual machines," says Frank Gillett, an analyst at Forrester Research.

For example, a virtual machine may be running at 100 per cent utilisation but using only a fraction of the underlying server's resources. "Some of the things you monitor no longer mean the same thing," Kuzmack says.

"It would be nice if all of our standard tools worked in the virtual space, but they don't, and it doesn't look like they're going to anytime soon," says Norm Fjeldheim, CIO at Qualcomm. He is evaluating tools targeted at virtual machine management to fill the gap.

For many organisations, identifying the root cause of virtual server problems and rectifying them remains largely a manual process. As the number of virtual machines in the data centre increases, solving those problems in an automated way becomes more urgent.

Performance monitoring is just one aspect of virtual machine management. Other tasks include optimising the mix of virtual machines that should reside on each physical server to achieve the best possible performance; automating virtual machine provisioning, load balancing, patch management, configuration management and fail-over; and enabling policy-based orchestration to automatically trigger the appropriate responses to events.

For some functions, such as patch management, existing tools work fine, says Paul Poppleton, a Qualcomm senior staff engineer. In other areas, he says, "we're getting the best wins on the tools that take into account the fact that systems are virtualised."

Even organisations just starting virtual server projects can quickly run into management challenges. Once the decision is made to introduce virtual servers, the numbers can increase much more rapidly than expected because it becomes easier to procure new servers, says Poppleton. "Tack on 20 per cent or 30 per cent to what you planned on for growth, because it can really take off on you," he warns. Qualcomm has 1,280 VMware ESX Server virtual machines companywide that run a mix of Windows and Linux. About 850 virtual machines are running in Qualcomm's data centre, with each physical server hosting an average of 10 virtual machines.

Even fine-tuning the performance for as few as 10 virtual machines can be a challenge. "One place where we've had trouble is trying to manage the resources on a single physical host," says Poppleton. VMware's VirtualCenter 2 management software, which Qualcomm is beta-testing, should help with that, he says. The software is expected to ship in the first half of this year.

Finding the right tool

Kuzmack is also beta-testing Virtual-Center 2. One component, Data Resource Scheduler, aggregates servers into pools that can be assigned to groups and managed through policies that the administrator creates. "We can take a group of physical servers and carve them up into resource pools where we can set high-level and low-level limits and resource guarantees," he says. Another feature, Distributed Availability Service, automatically moves virtual machines to a new physical server and restarts them after a physical server fails.

Kuzmack is also working on basic monitoring. He wants to integrate VMware's VirtualCenter control software with Gannett's Microsoft Operations Manager (MOM) software. Other than basic performance metrics, "we haven't determined what we want to expose to the MOM console yet," he says.

Jim Ni, senior technical product manager at Microsoft, says the company is working on adding more virtual machine management capabilities to its management tools, but it's not there quite yet. For example, Systems Management Server can manage physical machine image libraries but can't differentiate between an image for a physical machine and a virtual machine image.

Poppleton is using VirtualCenter but says he also needs more cross-platform tools. "Right now, we're a VMware shop, but in the future that may not be true," he says. He's considering using VirtualIQ, a virtual machine management tool suite being developed by ToutVirtual. It supports automated provisioning, capacity management and security. He's also looking at tools for "grid-style management of physical and virtual systems," such as Platform Computing's VM Orchestrator. VMO optimises capacity by dynamically allocating and controlling virtual machine resources and utilisation levels based on user- defined policies.

Both ToutVirtual and Platform Computing currently support VMware, but both have also announced plans to support Microsoft Virtual Server and the open-source Xen virtual machine monitor. In the interim, Poppleton's staff has had to develop some of its own tools.

"We're in the first phases of automated provisioning," he says, adding that Qualcomm wrote its own utilities because it couldn't find a tool that met all its needs.

Christopher Ware, assistant vice president of technology services at a Wall Street brokerage house, recently deployed BMC Virtualizer, an orchestration and provisioning tool that's part of a suite of virtual machine management tools from BMC Software. "It gives us policy-driven responses to resource utilisation issues," he says. The tool also can support virtual machines created with Xen, VMware and Virtual Iron, all of which are running at the brokerage.

Like other cross-platform tools, Virtualizer integrates with APIs from VMware and other virtual machine software to automate the execution of those proprietary tools. For example, Virtualizer supports the automated movement of virtual machines between physical servers in VMware environments using VMotion.

Ware, whose company prefers to remain anonymous, says that getting BMC Virtualizer to work with all of his virtual machines wasn't exactly a plug-and-play experience -- especially since no common standards exist. In his environment, "there was a significant amount of coding and customisation required to do a lot of the virtual provisioning," he says, and that accounted for the bulk of the deployment costs. A BMC spokesman says the product works out of the box with more than 50 applications.

"Where standards would really help is these interfaces for controlling and manipulating virtual machines and getting feedback on them," says Forrester's Gillett, noting that basic virtualisation is being commoditised at the chip and operating system levels. VMware, Xen developers, Intel and others are discussing the need for higher-level standards but have yet to reach a consensus.

Stewart Hubbard, director of IT engineering at Coldwater Creek Inc., a clothing retailer in Sandpoint, Idaho, has about 60 production servers running in ESX Server virtual machines. He says resource allocation is a big issue.

"When a particular machine starts to get hammered and the resources aren't available, you'll see a noticeable dip, and the end users will notice it," he says. Right now, he uses VirtualCenter to address the issues, one machine at a time. And because not all application vendors support their software running in virtual machines, he uses a tool from PlateSpin to migrate virtual machines back onto physical servers for technical support purposes.

While virtualisation-specific management tools are a good option today, ultimately, "you need something that manages the virtual and the physical together," says Gillett. BladeLogic, which offers a virtualisation-aware configuration life-cycle management tool, contends that a single set of policies should control both worlds. Once the policy and "personality" of a server are defined, "every instance of the server, whether physical or virtual, is treated more as a compliance exercise," says Vick Vaishnavi, director of product marketing at Blade-Logic in Waltham, Mass.

Qualcomm's Fjeldheim says he sees no reason to wait for all the pieces of the management puzzle to come together. Integrated tools would be nice to have, but "we're happy to use the tools that are specific to the virtual environment. It's not that big a deal," he says.

Today, it's the early adopters in industries such as financial services that are encountering management issues with what vendors like to call "virtual machine sprawl," but the pain will spread as more companies ramp up the number of virtual servers in use between now and next year, analysts say.

"Implementing this stuff is the beginning of rethinking how you manage your IT infrastructure," says Gillett. "You won't get far without thinking about data centre automation."