So you have a secure perimeter around your network, your policies prevent anybody opening strange files and your mail system scans all emails and removes anything dodgy. Yet you still get hit with a virus, or even worse a worm. What do you do?
This is pretty much the position my team found itself in last week. Despite extensive precautions, the notorious Blaster worm had managed to get round our defences and launch itself inside our network.
My company (which I’m not going to name) runs a large, multi-national network encompassing over 2,000 users in locations across the USA and Europe. It has been designed in such a way as to minimise the number of entry points for unfriendly code, namely by keeping the Internet access points down to an absolute minimum. Unfortunately even with a hardened perimeter, we could not avoid this attack. Fortunately our WAN design managed to localise the attack, although it had happened at our HQ. This brought our network down for a complete morning, and it took all our IT resources for the best part of three days to get back to normal.
Fortunately, we had considered the possibility that something like this would happen one day and had formulated a ‘plan B’. Here is the first lesson you must learn: the first 30 minutes are critical in dealing with any emergency and in the case of dealing with a worm, even more so. This is when you are most vulnerable. You cannot afford to waste time wondering what to do and who had to do what.
There’s a worm on the loose…
As soon as we realised we had a worm on the rampage, our first action was simply to raise the barriers. Nothing fancy or overly technical, we simply unplugged ourselves from the outside world. We pulled the DS3’s (our connection to the outside world) from the router. It isn’t advisable to go playing with configurations as it takes time and also needs to be redone later.
At this stage we didn’t know the exact nature of the threat and we wouldn’t take the chance that the worm was trying to broadcast essential data. But if you are going to take this approach, make sure you have a safe machine with some form of Internet access. Otherwise you will be working blind.
A communications centre was established in the main meeting room. In events such as these, tech managers are rarely useful, so having them handle all the communication was appropriate. Someone needs to manage the senior management and disgruntled user base. From this centre, voicemail messages were broadcast and flyers pinned to doors letting people in the company know there was a problem. Good quality communication is vital in managing expectations so don’t assume it’s not important.
In the Data Centre, there should be a core team of engineers analysing the problem. Get on the news groups; someone will have an announcement out there that can help. Within an hour of the network going down, we had our culprit. We also had our fix and people off applying the patch.
Our remote offices were brought up to speed and patches applied there. Our decision to unplug the Internet had also been prudent as Blaster wanted to broadcast information. In this case, we would have been safe as the port it communicates on is 4444, which is blocked outbound, but it may in other instances have been port 80. We would then have been in trouble.
As soon as we knew the scope of the external threat, we were able to make the decision to connect back the DS3 links. By the time the outside access had been re-established, the core servers had been patched and our VLANs were doing a good job of keeping the rampant worm off the server networks. We then started restoring services.
Don’t forget the Workstations. Blaster infected everything it touched. We have a standard email template which was distributed with details of the patch and how to deploy it. Automatic software distribution would have helped significantly here, but we aren’t scheduled to install that for another 6 months, although that is now being accelerated. It took several days to patch all workstations, but once the core was done, the worm lost its ability to replicate at pace and the network stabilises.
Once we were clean it was time to work out how were attacked in the first place. We have a solid infrastructure. We have a very secure perimeter and our security committee is comprised of exceptional individuals and our processes are followed meticulously. Yet we were still compromised. How? Because it is the real world out there. At the moment it looks like it was an errant laptop, infected on a home network then brought into the office.
Could we have prevented the outbreak? Yes. We could have patched everything in time. Can we afford the time to test and install every new patch, within two to three weeks of it being released. No. So what’s the answer? Let me come back to you on that one...!