Microsoft may be looking to expand its AntiSpyware tool to detect the remote system monitoring tools known as "rootkits".

The feature would be a significant expansion of AntiSpyware, which has recently become the subject of controversy over Microsoft's decision to stop warning users to remove software from Claria and other adware vendors.

Microsoft, like other security software vendors, has begun devoting research efforts to the difficult task of combating rootkits, which have begun to pop up in combination with spyware, Trojan horses and other types of attacks. Rootkits have been around for years, but in the past few months attackers have begun using them to make spyware and worm infections nearly impossible to detect and remove.

Because rootkits modify the operating system kernel, they are able to disguise telltale signs that would ordinarily indicate the presence of malicious code.

Microsoft Research's Strider GhostBuster Project has developed three versions of a tool for detecting rootkits. Microsoft's Malicious Software Removal Tool got rootkit-detection features in April. AntiSpyware may be next, according to a report from industry journal eWeek. Microsoft declined to comment on the report.

AntiSpyware, still in beta-test stage, has recently suffered from blows to its reputation. Earlier this year Microsoft admitted it had mistakenly listed a Dutch MSN competitor as a source of malicious code, disabling users from setting the site as their homepage. Earlier this month, the company faced strong criticism for its decision to change the status of software from Claria (formerly Gator) and several other software makers, no longer advising users to remove the components.

Microsoft is reported to be in talks to acquire Claria, but denied it had given the company's software any special treatment. Regardless of Microsoft's motives, many industry analysts said the move means AntiSpyware is no longer trustworthy, since Claria's software clearly meets several of Microsoft's own criteria for identifying spyware.

Rootkits are still relatively scarce, but their proliferation could mean serious problems for enterprise security, since they are nearly impossible to detect with many current security tools. They are often implanted as the result of a virus infection or a system compromise. Rootkit techniques have begun migrating to viruses and spyware, possibly under the influence of organised online criminal groups, security experts believe.

Windows is particularly vulnerable to rootkits because it is so widely used, and because its application programming interface (API) makes it easy to mask behaviours on a system.

Other security vendors are also bringing in anti-rootkit features. Sana Security recently added rootkit detection to its Primary Response product. F-Secure has released Blacklight, a tool specifically designed for rootkit detection, and Sysinternals Freeware offers RootkitRevealer.

Microsoft's competitors in the anti-spyware business include Trend Micro, which in May Trend Micro acquired anti-spyware startup InterMute, as well as other major anti-virus vendors such as McAfee and Symantec.