To stop worms and malware, first you must know about them. In today's rapidly evolving networks, where attackers are often one step ahead of the products designed to thwart them, anomaly detection is an important innovation. Many vendors rely on signature detection to find network-borne threats. Customers often have to wait days to get a working signature for a new worm, leaving their networks vulnerable in the most critical period during a worm's release.

Network behaviour analysis is one of the most robust and scalable security technologies classified recently by Gartner. At the core of network behaviour analysis are anomaly-based algorithms used to identify emerging threats. Three types of anomaly detection are used in network behaviour analysis:

  • Protocol - detects packets that are too short, have ambiguous options or violate specific application layer protocols. Most useful for detecting host-level attacks.
  • Rate-based - detects floods in traffic using a time-based model of normal traffic volumes. Most useful for detecting denial-of-service attacks.
  • Relational or behavioural - detects changes in how individual or groups of hosts interact with one another on a network. For example, a normally quiet host that starts connecting to hundreds of hosts per second on the SQL port indicates a worm. Useful for a variety of threats, from worms and malware to insider misuse.

    By applying anomaly algorithms best suited to the attacks they are designed to detect, anomaly detection can proactively identify zero-day worms, malware, acceptable-use policy violations and insider misuse. Because anomaly detection looks for substantial changes in network behaviour, it is less prone to false positives, and requires less configuration and ongoing maintenance than many other security methods.

    However, network behaviour analysis doesn't end with detection. Once a threat has been identified, this technology allows operators to visualise good and bad or suspicious traffic, and contextualise it in relation to other traffic and historical roles.

    A network behaviour-analysis system can take preemptive action, blocking a port on a switch, quarantining traffic to a separate virtual LAN, or applying a filter or access control list to lock down propagation. Behaviour modelling is equally applicable to mitigation and detection.

    For example, network behaviour analysis systems can generate a filter to block traffic on a suspect port and simultaneously generate a white list for known good traffic, freezing the network in a known good state.

    There's both an art and a science to applying anomaly detection. Effective use of the technology by security vendors requires deep experience with networks, threats and the appropriate anomaly-detection algorithms for a given threat model. When done well, anomaly detection is extremely effective in finding and foiling network-borne threats and should be part of everyone's security tool set.

    Paul Morville is director of product management at Arbor Networks.