When properly implemented and managed, virtualisation means cost savings -- for server hardware, support and administration -- as well as easier server deployment and reduced energy consumption. But for these benefits to come fully to fruition, the virtualisation layer of the stack has to be managed independently of either the application or the physical server.

Fortunately, that's not difficult if you do it right.

Sure, server provisioning and back-up become easier in the virtual world. But monitoring, especially checking the servers' underlying physical resources, becomes even more important. You have to balance the types and numbers of virtualised applications among the physical servers with an eye to best use of resources.

The consequences of getting it wrong can turn some -- or all -- of the virtualised applications running on a physical server into poorly performing slugs.

Successfully managing virtualised servers means understanding that virtualisation introduces a new layer into the server software stack. And you don't manage the virtualisation layer by managing the underlying physical server or the virtual machines or the VMs' applications.

Instead, virtualisation must be monitored and managed separately from the operating system, application or the (physical) server layer. (Note: This assumes you're using hypervisor products like Xen or VMware. There are also virtualisation products like Virtuozzo that virtualise on top of the operating system. That changes the picture somewhat, but the basic principle remains the same.)

Second, you have to keep in mind that while virtual machines appear independent to the applications running on them, they can and do interact through the physical server and its available resources. All the applications and -- to a lesser extent -- the VMs running on the physical server may act as if they're independent, which is part of the charm of virtualisation. But at the bottom, they're all drawing on the same pool of resources from the physical server. A large part of successfully managing virtualised servers is making sure all the VMs have the resources they need when they need them.

The third point flows from the first two. While managing virtual servers is in many ways similar to managing physical servers, they are not the same thing and the analogy stretches only so far.

The good news

Still, despite all the caveats, users agree that virtualisation is a win when it comes to managing servers.

"Virtual servers are a lot easier to manage," says Mike Carvallho, CTO of Radiator Express Warehouse The company, perhaps better known by its 1-800-Radiator moniker, is a chain of 200 franchised radiator shops in the US and Canada. Carvallho manages nine Dell servers that run VMware and are divided between two sites, supporting 55 virtual servers.

Not only are provisioning, capacity management and recovery easier, but there are fewer physical boxes to deal with. "One of the things you immediately lose is all the hardware that was a nightmare to manage," Carvalho says. Among other things, the physical servers tend to be of varying vintages from different vendors, which complicates patches and change management and means keeping on hand a supply of spare parts for the older servers.

When switching to a virtualised environment, many shops take the opportunity to limit the number and types of physical servers, often upgrading to newer technology at the same time. This makes it easier for data centres to keep their hardware in sync by limiting the variety of physical servers in the system.

The result of properly managed virtualisation is a lower-cost, more efficient operation. "It all comes down to cost at the end of the day," says Chris Stucker, manager of systems, networks and services at Applied Extrusion Technology (AET), a maker of plastic films. Stucker says his server support costs have dropped significantly. "Many of those were older servers, and we were paying a premium for support based on that," he says.

He could not provide specific savings statistics.

On a typical day, AET is running between 45 and 55 virtual servers on its three blade servers supporting company headquarters and three manufacturing plants in the US and Canada.

However, this rosy picture applies only if the virtualised servers are properly managed. There are several places where managing virtualised servers can go wrong.

Nonlinearity -- when 2+2 = 5 (or 2, or 10, or ...)

Part of the difference between physical and virtual servers is nonlinearity. In the world of virtualisation, two plus two sometimes makes three, or six.

"Nothing is really linear," says Derek Anderson, the lead developer for Enomaly's Enomalism product, a tool for Xen's virtualisation platform. "You get weird overlaps."

Different applications require different amounts of resources. Databases use a lot of RAM and processor cycles, Web servers make a lot of disk accesses, and so on.

Part of efficiently allocating resources depends on the timing of the loads. Anderson provides the following example, which he says comes from an actual case study involving a global financial services company.

A physical server running virtual servers to support users in the UK and Japan during the business day will probably be fine. The same server running virtual servers supporting users in New York and Florida is likely to have a problem -- because of their time zones. Japan and the UK are about 12 hours out of sync. Florida and New York are both on Eastern Standard Time. In the first case, the load profiles tend to cancel. In the second, they bump into each other.

So, each virtualised server needs a share of the resources of a physical server. For maximum efficiency, the virtualised servers' resource needs should complement those of the other virtual servers on the same physical box. In general you want to distribute applications with similar resource demands over different physical servers.

The physical server is where the contention occurs. Most of the time you're only running one application per VM because things are cleaner and easier to manage that way.

Think of it as trying to fit a lot of different-size blocks (VMs or application stacks) into a series of same-size boxes (physical servers). If you try to put all the blocks of the same size into one box, you'll probably need more boxes than if you mix up the size of the blocks to make the best use of the space in the boxes.

Managers should seek applications with complementary demands. This can be a problem because the natural tendency of someone brought up in the server environment of a data centre is to think in terms of putting the same virtualised applications on the same servers. In the traditional server world, it's common to have one or more servers devoted to mail, another group of servers devoted to the Web, another to databases and so on.

But this is exactly the wrong thing to do in the virtual world.

That means you want to be careful about putting two applications that require a lot of memory, or storage, or any of the same resource, on the same server.

It may work when neither application is doing a lot of work, but as the load gets heavier, you're going to run into problems of resource contention and degraded performance.

Balancing resources is trickier than it sounds, because you can't simply take two virtualised applications, each of which uses 50 percent of a physical machine's resources, load them both on the same physical machine and expect things to go well.

Even two applications that look like a perfect fit may interfere in minor -- but critical -- ways. An application that normally has low RAM requirements may occasionally spike its RAM demand. If it's loaded on a physical box with an application that constantly needs a lot of RAM, you've not only got a problem, you've got one that's probably weirdly intermittent.

The bottom line is that you can't predict absolutely if two applications will virtualise well on the same physical box. The only way to know for sure is to test the combination. "You have to see what works well and what doesn't," Radiator Express' Carvalho says.

"I would not take an unknown application whose features I have not thoroughly tested and expect all the features to work the first time out."

The good news is that virtualisation makes it easy to run the virtualised servers on a test system and then migrate them over and go live if the combination works.

The testing procedure has a lot in common with testing a physical server supporting a single application. That is, you run test loads until you're sure that it works, can characterise the virtual server's performance and can confirm that the level of performance is acceptable.

With a virtualised server, you run the proposed combination of VMs and applications on a physical server and monitor the results using monitoring tools from a third party or the virtualisation supplier. This is done instead of using something like Microsoft's Iometer, which works through the operating system. Remember that in products like VMware and Xen, the operating system is also running on a virtual machine. You need something that can go through the hypervisor layer to the underlying hardware. In both cases, you use a suite of test data, preferably drawn from your own enterprise.

One major difference comes when it's time to go live. Rather than installing the operating system and applications individually on the new server, you simply clone the whole stack over -- a process that usually takes a matter of minutes.

This concern for interactions still applies once the server is in production as well. Enomaly's Anderson cites the example of a DBMS and a Web server on the same physical platform. If the database can keep all its tables in RAM, it will have relatively little effect on a disk-intensive application like a Web server, Anderson says. But when a table gets large enough that the DBMS has to start paging it out to virtual memory, it can have a big impact on the Web server -- and both applications' performance can suddenly go sideways.

Keeping it all together

While you don't want virtual servers with the same resource demands on the same physical box, you probably want to keep at least some of your physical servers in close physical and network proximity.

The reason: to ease switching virtual servers among physical servers should the need arise. If a group of physical servers shares network connections and other resources, the time to switch a virtual server from one physical server to another can be greatly reduced, as can the configuration effort.

"If you have a bunch of servers in a rack, you can turn off the virtual machine [on one server] and come up on a server on the same switch," says Anderson, "and the time it takes is literally the amount of time it takes to copy the hard drive."

On the other hand, he says, if the machine you're transferring to "is on the other side of the data centre, plugged into different switches and different subnets, then the time to do the reconfiguration could be an extra five or 10 minutes. Or if there's other stuff, then that could take a lot longer.

"It's just in the way you laid out your data centre," he adds.

Of course, there are limits to proximity, either physical or network. You don't want the servers to overload the network connections, for example, and you may want to have physical servers some distance away -- or even in a different state -- for disaster recovery. It's a balancing act.

Keeping track of hardware

This non-linearity complicates another aspect of managing virtual servers. Administrators need to closely monitor resource demands on various physical servers.

This is not the same as the demands reported by the operating systems on the virtual machines. Administrators need to look at what's happening on the actual hardware as well. They need to keep an eye on trends to avoid a sudden resource starvation as applications' resource profiles change.

What's more, tracking the physical hardware has to be done in detail because of the different loads the virtualised applications put on the different kinds of resources the server supplies. Because the various virtualised applications have different demands, things like RAM, processor cycles and I/O bandwidth have to be tracked separately.

What you can't do is try to track servers' physical resources by monitoring the operating systems on the virtual machines. You must go below the hypervisor layer to track a physical server's resources separately from a VM's. You probably still need to track reported resource use by the VMs, but that's a separate issue and done for things like performance analysis on the apps the VMs are running. (Note: again, this assumes you're using hypervisor virtualisation like VMware or Xen.)

According to Anderson, memory bandwidth is usually the most critical resource on a physical server, simply because it is a hard limit. Unlike nearly everything else, from storage to processing cycles, you can't increase the memory bandwidth built into the box. When that saturates, you've got to move some of the VMs to a new physical server.

Bandwidth of all kinds is a particular problem because it is likely to be the place where problems show up. Not only do almost all applications use bandwidth, but the loads tend to be very variable. If two or more applications peak their I/O or memory demands at the same time, you're likely to have an intermittent problem that will be hard to track down.

This introduces, if not a new level of complexity, at least a new watch point in the virtualised data centre.

When the physical server supporting virtualised machines runs out of resources, a whole lot of applications are going to be affected. As Stucker puts it, "Anytime you miss your guess, there's going to be more delta in your peaks and valleys."

Monitoring

The need to assure the availability of resources is an additional reason why virtual/physical server stacks need to be monitored closely. Another reason is that a hardware failure will impact more than one application.

Because monitoring is so critical, virtualisation software like VMware and SWsoft's Virtuozzo include tools that help track both physical and virtual resources.

Because loads change over time, monitoring is a continuous process. You can't set it and forget it, any more than you can with a physical server. In fact, you have to pay even closer attention to monitoring, because each physical server is supporting several virtual servers and all of them can be affected if the physical server starts to run out of resources.

Third-party tools

One peculiarity of virtualisation management is that a lot of companies aren't invested in third-party tools to do it. According to Bob Gill, managing director for servers at TheInfoPro, a New York-based market research firm, surveys show that most virtualisation users rely on the tools supplied by the virtualisation vendors rather than investing in additional applications to help manage their environments.

In part, this is because companies like VMware and SWsoft already provide management tools with their products. But part of it is also due to the relative immaturity of the virtualisation market. At the recent Infrastructure Management Conference, Gill said he expects the market for third-party monitoring and management tools to grow as the market for virtualised servers grows.

Some customers are taking other tacks. At Radiator Express Warehouse, Carvalho employs what he calls a three-tier solution. He uses the tools that are already part of VMware's ESX, which he calls "awesome," but he backs them up with Dell Remote Access Controller, which monitors all the equipment in his physical servers, and WhatsUp Professional from IPswitch for managing his network.

What to virtualise

Holmes noted there are some myths about what you can and can't virtualise successfully. "Originally, when VMware was the first virtualisation product to really penetrate the enterprise market, you had people saying, 'Don't virtualise this or that,' " says Holmes. "That has somewhat changed." As users have become more comfortable with virtualisation, it's become clear that a lot more can be virtualised than some people originally thought.

For example, a lot of enterprises were reluctant to virtualise DBMS applications because they need so much memory, processing power and memory bandwidth.

But these days, advances in server technology, such as multi-core servers that can take a lot more RAM, have made database virtualisation more practical.

Ruven Cohen, CTO and co-founder of Enomaly, says that better hardware has a lot to do with the acceptance of virtualised databases. "On the early servers, when virtualisation was relatively young, your big stumbling block was what you could put through the server," he says. "With the latest chip sets from Intel and AMD, that is no longer an issue."

That said, in nearly every data centre, there are some applications it isn't a good idea to virtualise.

For example, Carvalho runs his production database on Dell eight-way 6850 servers. "I would never virtualise those, because I would lose too much performance," he says.

Similarly, Jamey Vester, a production control specialist for Subaru of Indiana Automotive, says his company doesn't intend to virtualise its SQL Server cluster. Also, he says, "some of the specialised production and automation applications are very sensitive to the physical hardware."

The critical point is not that these applications are databases, but that they are large, resource-intensive applications. For example, AET's Stucker doesn't virtualise his company's enterprise database, which is an ERP and data warehouse system running on Informix on HP-UX and which Stucker calls "huge." "But we have all kinds of ancillary databases we have virtualised," he adds. "We have a number of SQL Server databases that are virtualised."

Generally, applications that heavily load their servers in terms of bandwidth or capacity are not good candidates for virtualisation -- unless you get more powerful hardware.

Establishing and maintaining virtual machines absorbs resources (although how much depends on what kind of virtualisation scheme you use). Those resources come out of the same resource pool that supports the virtualised application. If an application is using 80 percent of a physical server's resources, it's not a good candidate for virtualisation unless you put it on a much more powerful server.

The licensing problem is stickier. Some vendors don't license their products to run on virtual machines. Sometimes this is done for business reasons, to keep control of the number of copies running in the enterprise. Sometimes, although more rarely, it is done because there are hard limits in the software that inhibit performance on a virtualised server.

In some cases, the vendor simply doesn't know how the product will perform in a virtualised environment and is trying to avoid possible performance problems. br> Generally, software vendors are becoming more virtualisation-friendly, but you need to check the license before trying to virtualise an application.

Do's and don'ts of managing virtualised servers

DO ...

  • Remember there's real hardware down there somewhere.
  • Plan your deployment.
  • Control virtual server sprawl.
  • Choose your tools carefully.
  • Balance your servers' loads.
  • Monitor physical resources and trends.

DON'T ...

  • Ignore physical server load.
  • Assume virtualised applications will scale linearly.
  • Put the same type of applications on the same machine.
  • Try to virtualise all your servers.