Security has been touted as one of the benign by-products of virtualisation – but according to a recent study, that’s no longer the case.

This is because technological developments mean that malware can detect that it's running inside a VM and so alter its approach. Hiding the existence of a VM is "fundamentally infeasible," says the report, in response to the mooting by some virtualisation and security developers of the desirability of developing undetectable VMs.

The white paper, called Compatibility is Not Transparency: VMM Detection Myths and Realities and published by virtualisation vendors VMware and XenSource, as well as academics from Stanford and Carnegie Mellon Universities, discusses the ability to use virtual machines (VMs) as way of trapping rootkit attacks - a technique known as honeypotting - as well as worm detectors. This has relied on malware’s general inability to recognise that it’s not attacking a real, physical machine.

The paper reckons that: "Anti-virus vendors are eager to cloak their use of VMs in identifying newly released exploits for similar reasons. Others have proposed offensive uses of virtualisation in the form of VM-based rootkits hoping to leverage the transparency of VMMs to cloak their presence and provide an ideal attack platform."

However, continues the report: "We believe the transparency these proposals desire is not achievable today and will remain so."

The problem according to the report, is that those developing security solutions start with the premise that, if the performance profile matches that of a real machine, its containment within a virtual environment is undetectable.

But there are big differences that betray the presence of a VM. The paper reports that: "Virtual implementations of this architecture differ substantially from physical implementations...Logical discrepancies are semantic differences in the interfaces of real and virtual hardware. Most current VMM [virtual machine management] detection methods exploit differences in the virtual CPU interface of VMMs such as VMware Player or Microsoft VirtualPC that violate x86 architecture."

It continues: "These and other discrepancies are unimportant to the vast majority of software, VMMs make no effort to hide them." Software can detect discrepancies in the CPU, in storage devices and in device drivers, according to the report, largely because VMM developers are more concerned about compatibility and performance than about detectability - or about overall security.

And the range of possible tell-tale signs is both wide and deep. It includes, for example, variability on a run-to-run basis of the time taken to query the PCI bus, which doesn't alter on a physical machine.

As a result, the authors go on to conclude that: "The ease of imagining new detection methods suggests that VMM transparency is difficult to the point of impracticality."

It would add large overheads and would be running to catch up with a moving target, concludes the paper: "Stealth and performance are thus, to some extent, in inevitable conflict."

Although "tomorrow’s VMMs will change, performance will remain paramount. Consequently, virtual and native hardware are likely to remain highly dissimilar, and thus amenable to discrimination."

Could this be a good thing, in that the spread of virtualisation will discourage rootkit-based malware? The paper reckons that: "Soon, malware that limits itself to non-virtualised platforms will be passing up a large percentage of commercial, military and institutional targets.

"To the degree that malware disables itself in the presence of VMs, VMs become even more attractive for production systems. In the long run, malware authors are motivated to operate regardless of the presence of a VMM."

The good news is that a VM-based rootkit is as detectable as its physically located counterpart, reckons the paper: "No matter how minimal the hostile VMM is, it must consume physical resources, perturb timings, and take measures to protect itself from the guest, leaving it no less susceptible to detection than other VMMs."

One conclusion which this suggests is that developers and IT managers need to ensure that security in their VMs is as tight as that in physical machines.