The rising popularity of custom malware and the inability of antivirus software to keep pace poses potent challenges for enterprises trying to keep their systems secure.

It's no secret that the goal of modern malware writers is to create attack software that is stealthy and flows undetected for as long a period of time as possible. However, what's increasingly startling is how pervasive custom malware has become as part of traditional attacks.

"The advanced attack is getting more pervasive," says Shawn Moyer, managing principal at security services firm Accuvant Labs. "In our engagements and my conversations with peers we are dealing with more organisations that are grappling with international infiltration," he says. "Every network we monitor, every large customer, has some kind of customised malware infiltrating data somewhere. I imagine anybody in the global 2,500 has this problem."

Not all of the code slipping on by security defenses is customised malware. Consider the Prioxer backdoor Trojan discovered by Symantec. The Trojan does many of the things a typical Trojan would do, such as drop a .dll file, operate command and control through IRC channels. However, because of some trickery between the cached version of the infected file on disk, the Trojan that resides in memory is essentially invisible to the system.

All of this supports a report released by NSS Labs last week that found many antivirus applications fail to find malware that attempts to infect systems through multiple entry points, such as email or USB drives. The independent testing company evaluated the effectiveness of 10 popular antivirus applications against multi-vector attacks (malware delivered from the web, email, network file sharing and USB flash drives), memory-only attacks and anti-evasion techniques.

The report, titled Socially-Engineered Malware Via Multiple Attack Vectors, found that antivirus applications that find malware at one point-of-entry may not detect it in another. For example, a web download could be missed if downloaded from a USB drive or network file server.

The report found that antivirus products miss between 10 percent and 60 percent of the evasions most often used by attackers. Furthermore, less than a third of the tested vendors had protection for memory-only malware, leaving a significant evasion gap in their products.

Typically, according to the report, these evasion techniques include placing a wrapper or a disguise that are applied to exploits and malware in an attempt to thwart detection. "An exploit that is detected by a security product can be modified by an evasion technique to still get through to the target - if the intermediary security product does not have the appropriate anti-evasion capability," the report stated.

None of that surprises Moyer. "It's fairly trivial to customise an exploit to bypass 70 percent of the time. I do it all of the time on engagements," Moyer says.

How important is it for antimalware software to be able to stop each and every attack? Stewart has some sobering news about what the attackers do once they gain any kind of foothold. "They'll conduct a lot of network probing. They'll look for vulnerable network servers, web servers, SQL servers and other areas where they can gain another foothold: weak passwords. File shares, and they'll go from there. Moving point to point and extract whatever data they can," he says.