Vendors of IT security products frequently release information about vulnerabilities associated with their software and hardware. I receive 15 to 20 such advisories every day and have to examine each one to determine whether it's applicable to our infrastructure. If the advisory relates to a product we use, I must figure out whether it affects us.
That's not always easy. Before making a decision, I need to know if there have been any reported compromises and if the exploit code actually works.
I also review who released the advisory, the potential effect on my company and what resources I'll need to mitigate the vulnerability. Getting that information requires additional research, but as my recent experience shows, that can be time well spent.
Recently, someone released code that exploits a remote denial-of-service vulnerability in any network device from Cisco Systems that runs Versions 11.x or 12.x of the vendor's Internetworking Operating System (IOS).
Since there was a significant amount of information both from Cisco and on message boards, and the advisory pertained to versions of IOS that we run, I needed to do some research about the potential of this vulnerability.
Reconfiguring or upgrading routers to cope with the problem isn't easy, so I decided to find out more about the risks we faced and ways we could protect ourselves without causing too much disruption to our operations.
Just as I started investigating the advisory, a colleague sent me two emails that included code exploiting the Cisco IOS weakness. Rather than relying on third-party opinions, I decided to use this code to test the vulnerability myself.
To exploit this vulnerability, the program must send a series of specially formatted IP packets with specific protocol types to the interface of a vulnerable router.
When executed properly, the transmitted packets can cause the router to flag the input queue as full. When this happens, the input queue refuses any more packets, causing the denial-of-service condition.
Network administrators must then reboot the router to reset the device and clear the problem. If the router isn't accessible from the network, someone must go to the data centre and manually reboot the device - which isn't much fun at 3 a.m.
The code I received by email wasn't compiled. I transferred it from my desktop to a Linux workstation I have in the lab and reviewed it for other malicious content.
Once I had it compiled, I needed to find a router to run tests on. I talked one of the network engineers into letting me borrow a spare Cisco router, then asked him to install one of the vulnerable versions of IOS and configure it in a way that should allow the exploit to work.
For the code to succeed, for example, the Time to Live (TTL) setting for network traffic has to be less than 2. TTL is a value in an IP packet that tells a router whether the packet has been in the network too long and whether it should be discarded. I was sure that some of our routers had TTL settings within the vulnerable limits.
I ran a continuous string of packet traffic through the lab router while launching the attack. Within seconds, it stopped processing the legitimate packets and failed, just as the advisories had warned.
By running the "show interfaces" command on the router, I was able to see that the input queue had indeed filled up. And sure enough, the only way to reset or clear the input queue was to reboot the router.
I immediately sent out an advisory to the network team suggesting that they schedule an emergency inventory of all Cisco routers, switches and other devices running vulnerable versions of the IOS.
At this point, it looks like about 40 percent of our routers and switches are vulnerable.
It's difficult to get an exact number because many of the routers and switches across my organization aren't managed by our core network engineering team. Tracking all of those devices and determining the person responsible for each one would be a time-consuming process, especially when the devices are in different countries.
Solving the problem
Mitigation for this vulnerability requires two actions. The networking team must first patch IOS or upgrade it to a non-vulnerable version and then configure the access-control lists to block the protocols that the code uses to attack routers. The latter could be accomplished either on the device itself or on the firewall, which protects the device.
As I write this, no IOS patches are available, but I'm sure that by the time this column sees print Cisco will have released one. That's good, because upgrading IOS on a production router isn't a trivial task. It normally requires hours - sometimes days - of lab testing before our engineers feel confident in upgrading.
While we're preparing to upgrade or patch the systems, we've been able to take other steps to protect ourselves. For example, we obtained enough information about the exploit to create intrusion-detection system rules to watch inbound traffic for signs of attempts to launch this particular denial-of-service attack.
It has been a few days now, and we haven't seen any traffic indicative of this attack. So far, so good.
This week's journal is written by a real security manager, Mathias Thurman, whose name and employer have been disguised for obvious reasons. Contact him at [email protected]