Not one single intrusion-protection system (IPS) is able to protect against all attacks claims testing house, NSS Labs. The company evaluated 15 different network IPS products from seven vendors and showed none were fully effective in warding off attacks against Microsoft, Adobe and other programs.

NSS Labs found that the Sourcefire IPS showed 89 percent effectiveness against a total of 1,159 attacks on products such as Windows, Adobe Acrobat and Microsoft SharePoint, while the Juniper IPS scored lowest at only 17 percent effectiveness. NSS Labs, which conducted the test without vendor sponsorship of any kind, also evaluated the 15 network IPS offerings for their capability in responding to "evasions," attacks delivered in an obfuscated and stealthy manner in order to hide. In that arena, the McAfee and IBM IPS held up particularly well.

Rick Moy, president of NSS Labs, said he was disappointed overall that none of the 10Mbit/s to 10Gbit/s IPS products tested achieved 100 percent effectiveness in detecting and blocking the attacks, including buffer overflow exploits.

Products tested came from Cisco, IBM, Juniper, McAfee, Sourcefire, Stonesoft and TippingPoint. Check Point, Enterasys, Nitro Security, Radware, StillSecure, Top Layer and Trustwave declined to participate in this round of tests, which were conducted in October and November.

"The threats are continuing to get worse and everyone says they're keeping up with them, so we wanted them to prove it," Moy says.

The vendors that did participate were allowed to tune their equipment in one round of tests designed to find out how long it took to make changes to the default settings in order to try and improve performance based on policy. Under this measurement, McAfee, IBM and Stonesoft did well. The Sourcefire IPS, however, took the most time, which Moy said would translate into time needed for professionals to manage it in an enterprise.

McAfee, which is set to announce major enhancements to new network-security gear, was left at a loss to explain why the its IPS didn't achieve 100 percent effectiveness in the NSS Labs tests.

"There are a variety of reasons you might not achieve 100 percent," said Greg Brown, McAfee's senior director of products marketing, who added that he hadn't read the NSS Labs report yet. Sometimes lab tests simply "don't look like a real attack" to equipment. He said McAfee focused its efforts on "very new exploits."

Details on the IPS effectiveness, evasion attacks, tuning, performance and cost-of-ownership issues are included in depth in the 50-plus page report Network Intrusion Prevention Group Test that NSS Labs is selling for $1,800. NSS Labs also expects to conduct a round of tests for host-based IPS products in the near future. In September, NSS Labs rated reputation-based antivirus systems and found them very effective.