VMware says it's received a bad rap when it comes to security.

The company's problems started with a 2006 presentation at the Black Hat security conference by Joanna Rutkowska, CEO of Invisible Things Lab. Ironically, Rutkowska's "Blue Pill" talk had nothing to do with VMware. It was about creating undetectable malicious software using the virtualisation technology built into microprocessors.

But nevertheless, VMware is the world's best-known virtualisation company, so any questions about virtualisation and security "naturally became a VMware problem," said Nand Mulchandani, the company's senior director for security products.

"Blue Pill kind of set things off, but unfortunately it set things off on the wrong foot," he said. Soon VMware was fielding questions from worried customers. "They escalated it to our team and they said, 'Oh my God, we're going to get attacked by Blue Pill. What do we do?'"

Mulchandani has been trying to get the message across that the Blue Pill CPU virtualisation hack is not connected to VMware's software, which is widely used on datacentre servers to simultaneously run many copies of the operating system on a single computer.

It's one of several security messages that Mulchandani is trying to convey these days, as the company looks to repair its reputation in the security community while developing new products that will keep it one step ahead of rivals.

Critics say VMware must shoulder some of the blame for the Blue Pill confusion and that it harmed itself by attacking Blue Pill in company blog postings. "They took the easy route, which was to attack Joanna's research," said Tom Liston, a senior security consultant with Intelguardians Network Intelligence. "It was just a big brouhaha with VMware jumping in where they didn't belong."

The feud with Rutkowska flared up at a low point in the company's relationship with independent security researchers. Employees who had been working with researchers like Liston left, and by early 2007 the company had developed a reputation as being unresponsive to bug reports, something Mulchandani calls "Fortress VMware."

Mulchandani says the issue was simply that VMware didn't have the people in place to respond to the community. That changed, however, with the company's 2007 acquisition of intrusion-prevention software vendor Determina.

"With the Determina acquisition, a lot of the focus was on acquiring a team that had very fundamental and deep relationships with the security industry," said Mulchandani, formerly Determina's CEO. "We've really embraced the security community in a way we didn't before."

Since the acquisition VMware has restructured its bug response team, revamped its security portal and reached out to independent security consultants from I/O Active and the Metasploit team, to ask them to help hack their products and teach the company's engineering team.

Still, there have been some high-profile bugs found in the software. In April 2007, Liston demonstrated an attack on VMware Workstation that allowed him to run unauthorised software on a VMware system. And in February of this year, Core Security reported a similar flaw, also in VMware's desktop software.

Mulchandani says that these disclosures have further confused users, who wrongly assume that the bugs also affect the company's widely used datacentre product, called ESX.

ESX, he says, has a completely different architecture from the VMware Player, Workstation and Server products that have been hacked by security researchers. These products have many experimental features that may never get included in ESX, he said.

Intelguardians' Liston says the fact that a major flaw has not been found in ESX does not prove it is immune to bugs. "I would be willing to bet my paycheck that at some point in time, somebody's going to be able to find one of them," he said.

But the most intriguing part of the VMware security question may not relate to bugs at all. Nearly a year after the Determina acquisition, customers are still waiting to see what the company plans to do with its software, which scans the memory of Windows machines to block certain types of attack.

Mulchandani declined to comment on his company's product plans, except to say that his team is integrating the Determina software into the VMware platform.

But others say there is an obvious next step.

Because VMware ESX is already widely used in the datacentre to host Windows, it would be natural for the company to start selling a version of VMware that would secure Windows by default, according Thomas Ptacek, a principal with Matasano Security.

Liston agrees that Determina may help VMware stay one step ahead of Microsoft, which is readying its own virtualisation software.

"VMware is on a mission to tighten up their virtual infrastructure and to provide some things that they couldn't have provided before," Liston said. "They really sit in the perfect spot to do that kind of overall machine monitoring."