With the appearance of a new set of malware testing guidelines, it looks as if the hitherto erratic industry of software security testing could finally be about to grow up. But there is still a long journey ahead.

The <a href=" http://www.amtso.org/documents/cat_view/13-amtso-principles-and-guidelines.html" target="_blank"> important documents </a> that start to map out the future of this sometimes controversial area can be found on the website of the cross-industry organisation, <a href=" http://www.techworld.com/security/news/index.cfm?newsid=106220"> AMTSO </a> (Anti-Malware Testing Standards Organisation), which is backed by most of the world's leading AV companies.

The fundamental principles document has a series of bullet points on best practice that I don't, in all honesty, think solve much, useful as it is to state them in black and white. For instance, point 2, ‘Testing must be unbiased', sounds like the absolute bare minimum for carrying out a test of any product on a competitive basis. The problem has been that bias is often hard to detect or define, especially in an industry where tests are sponsored or paid for by parties who might in other fields be said to represent a conflict of interest.

The second document on dynamic testing makes more interesting reading and forms a brief but worthwhile introduction to some of the imponderables of anti-malware testing in the field. To date, the easiest and therefore most popular way to test anti-malware suites is to subject them to an identical family of malware samples, measuring detection rates, false positives, and accurate identification and remediation.

The problem is that this approach is years out of date, and everyone knows it. Today's malware evolves in days or even hours, comes in huge numbers of variants, and open back doors with a bewildering array of payloads. Many others have no specific payload at all, and just function to compromise an attacked PC the better to run any one of a menu of dynamically-chosen payloads. Detecting these using signatures is a resource-hungry process, tied into blocking the web, email and IM lures used to get the code on to a PC in the first place.

The answer is to expose AV programs to real-world threats, in real time, but which ones? How does one ensure that each program has been tested against the same threats so that the tests are fair and not misleading? The AMTSO's dynamic testing methodology reads like a first draft in trying to solve this issue.
As the paper itself says:

"An interesting variation on this is to obtain the malware by causing the machine to visit a large number of suspicious web sites, in the hope of its being infected, for example by using a script to launch the browser repeatedly. After the sites have all been visited, the machine can be analyzed to see how successfully the vendor's product protected the machine from infection.

This test is a good simulation of a common infection vector (drive by downloads). One caveat of this approach is that, partly due to the efforts of malware researchers, malicious servers may not serve the same (or any malicious code) if multiple requests are made. In this test it is hard to guarantee that each vendor's product will be exposed to the same threats so it is important to both use a diverse list of suspicious sites, and to repeat the test a number of times to see trends in performance."

AMTSO is better than nothing, but one suspects that the politics of its members (security vendors) and the complexity of its challenge will slow down its evolution. The truth is that all AV programs will miss real-world threats. Whether the world wants - or will be allowed to hear that - is open to debate.