For network managers, it's not so much a case of New Year's Resolutions as New Year's Higher Resolution, argues Adam Powers, the CTO of network analysis specialist Lancope. He predicts that this will be the year when flow-based technologies finally overtake existing packet-based methods of network monitoring, giving managers a deeper insight into what's actually going on in the network.

Chief among these technologies will be the behaviour-based techniques developed by the likes of Arbor Networks, Lancope, Mazu Networks and Q1 Labs, he adds. Built on top of deep packet inspection (DPI) technology similar to that used in IDP (intrusion detection & prevention) systems, these look for anomalous behaviour on the network that could signal the presence of a hacker or Trojan, or a virus propagating itself, for example.

"Our technology came out of research by Dr John Copeland of Atlanta University, who was looking for faster ways to move ATM cells. He developed flow analysis technology and discovered you could see patterns in flow data. We now have something like 140 algorithms looking for patterns," Powers says.

He adds that as well as using a probe to collect network traffic for DPI, systems such as his are also beginning to make use of flow data generated elsewhere in the network, in particular NetFlow and sFlow data.

So why are so many network performance specialists - Packeteer recently hooked in NetFlow, too - getting interested in flow data? Partly, it's the realisation that there is no point reinventing the wheel - there's already an awful lot of devices out there generating information that could be very useful if it were correlated.

NetFlow gets Flexible

That's been helped along by the development by Cisco of Flexible NetFlow, and of switches and routers with enough processor power to run NetFlow without it affecting their ability to do their real job, says Powers.

"Flexible NetFlow lets you put whatever information you want into NetFlow," he says. "We're getting to the point where, instead of being afraid of it, people will turn NetFlow on and use it. Moore's Law has helped there."

It can even save money, by removing the need to deploy monitoring hardware to remote networks, he adds. "In big distributed or meshed networks, you need to be able to see and track and audit communications between remote sites, and the only way to do that is to deploy a probe on each site. A really nice way to do that is to use NetFlow, so your switch or router becomes a virtual probe - after all, you already have the equipment there."

And what of sFlow, the flow reporting technology preferred by the likes of Extreme, Foundry and HP Procurve? Powers says it works in a subtly different way, and obviously is a much smaller market, but does have some notable advantages.

"With NetFlow, the router has to maintain a cache - though its overhead is less than a network engineer would expect - but sFlow has no memory consumption," he explains. "Most sFlow implementations use an ASIC in the linecard that only has to send a UDP packet with the datagrams, so it can be done at much higher speeds and is easier to adjust as well, you just change the sample rate.

"sFlow works great for traffic analysis or DoS detection, but my opinion is if you need specific information on transactions - to audit a server, say - sFlow may not be the best."

All this extra information is allowing the behavioural analysis companies to broaden the ranges of behaviour that they can look for.

"The next thing is behavioural modelling of users, not just IP addresses. It's particularly useful where your IPs change regularly," says Powers. "In our latest release we're modelling the behaviour of countries, too - we have a database containing the IP address blocks of countries and how they interact with your routers, allowing you to set your tolerance by country. The tolerance-based engine is new and significant.

"And right now, people are also very interested in botnets and extrusion detection or data leakage, so we are working on new algorithms for those."

He adds, "It's only in the last two years or so that both NetFlow and the tools to analyse it have matured to this point. With Cisco's fixed-format versions 1, 5 and 7 of NetFlow, it was layer 3 and 4 only. What they've done recently is build Flexible NetFlow - NetFlow v9 - which gives you the ability to send other information - payload data, TTLs, even application-level intelligence.

"We - the analysis tools companies such as Lancope, Mazu, NetQoS, Arbor, Crannog [now owned by Fluke], the people who figure out how to take in NetFlow and use it creatively - have really blossomed over the last year or so. Now it's not only security stuff such as worm reporting, but also network operations reporting, for example to understand how packet-shaping or rate-limiting is working."

Caveat monitor

However, there are caveats - for users, network analysis companies and network hardware vendors alike. For a start, not all equipment supports either sFlow or NetFlow - even some Cisco kit does not support NetFlow - and even if it does, you may have to pay the hardware supplier a licence fee to enable it.

"So do an inventory of your network and discuss your list of network equipment with your analysis software vendors," Powers says.

Not all of those software vendors support it, as well - indeed, Powers claims that company take-overs are likely, as the major players strive to acquire behavioural analysis technology.

"Three or four vendors do this sort of thing well - it takes real-world experience, and anyone else that wants it would have to buy it. I wouldn't be surprised if a large company added this technology to its portfolio," he says.

And useful as it may be, flow-based analysis cannot do everything. Both NetFlow and sFlow are statistical (albeit in different ways), and do not gather all the network traffic, yet for some tasks you will need to do just that.

"There's still room for IDP systems, Sniffers and other traffic recorders - there comes a time when you need to capture all packets, and NetFlow is only the first 128 bytes," Powers explains. "Sometimes you do need to record the whole session and replay it.

"But pick the areas where you need DPI - datacentres, Internet ingress points and so on - and deploy those IDPS and packet sniffers there. For everything else, you can leverage flow-based technologies."