Someone has been poking about on one of your servers. Maybe it’s obvious: defaced web pages, emails that couldn’t have come from the assumed sender, or worrying logs on your firewall. Or maybe there’s just something that doesn’t feel quite right. But where do you start, and what do you look for?

Isolate and Capture
The first thing you should do, which a lot of people don’t, is to disconnect the server from the network. There is always the possibility that someone is accessing it right now, who shouldn’t be, and they could be working behind you, changing things as you check them. Or something could be configured to react if you find it (best to isolate the suspect device to avoid contaminating anything else).

In all your investigations, make sure that you are aware of, and follow, your company security policy (assuming there is one). This includes anything you need to do to ensure that the correct procedures are followed to allow the matter to be pursued through the legal process later if required.

Back up the system - it sounds strange, as why would you want a backup of a compromised system, but this will be needed if legal procedures are subsequently followed up (in which case you will need to put it away where it cannot be tampered with). It will also be useful to see just what was done - or even as a training exercise for colleagues.

What’s Been Done?
The first and most obvious place to start looking is in your log files. If you don’t know where they all are on your server, look at the /etc/syslog.conf file. Remember that it’s possible that anyone attacking your system may have also managed to access, delete or change logs, so if you don’t see anything unusual that doesn’t necessarily mean you are safe. Look also at other logs: from firewalls, routers and IDSs, to see if you get any hints there.

Check all system binaries against the original media (not backups, as last week’s backup could have been infected without you suspecting). Trojan horses are commonly installed by intruders to replace binaries such as ftp, ls, telnet, netstat and others. Often these can have the same timestamp and sum values as the original, so you’ll have to use cmp for a proper comparison against known good files.

Check the /etc/passwd file for new or strange entries, particularly any with no password set. And look for setuid and/or setgid files anywhere on your system that may have been left to make life easier for someone the next time they decide to access your system.

Pay particular attention to system and network config files. Are there any that weren’t there before, or have permissions been changed on any to allow write access where it shouldn’t be permitted, for instance? In fact, you’ll need to thoroughly investigate all files and directories, to ensure user data hasn’t been modified and to search for hidden directories and files that aren’t immediately apparent. You’ll need to use find rather than ls, as it won’t normally show files and directories beginning with a period that have been tucked away under some valid user’s directory structure.

Investigate also any cron jobs, and network services that might have been enabled in /etc/inetd.conf, looking for entries that execute a shell program. Check all programs that are specified to make sure that they are correct and haven't been replaced by Trojan horse programs.

Stuff left behind
You may be surprised at what you find deposited on your system. Amongst hidden files, companies have found details of other targets, their password files, or files to crack passwords. So-called hacker tools, designed to take advantage of known software vulnerabilities, may well have been copied onto your system for ease of use, or a network sniffer placed to catch your users’ login information. The outputs from these tools may generate themselves large log files, so you could look for large files that you don’t expect. If you do find a sniffer and some output, it may be worth having a look at the traces to see if they shed any light on what the attacker was particularly interested in.

Remember that if you are using Unix tools on your server to look for anomalies, those tools themselves may have been compromised. Best to get hold of a known, clean copy of whatever you need to use. It may sound paranoid but this isn’t the time to take risks.

Rectifying the situation
Once you have identified what has been tampered with, and can judge the damage that has been done, you will have to rectify the situation. Often the only way to do this is to rebuild the system from a clean copy of the OS, with all the up-to-date patches. Just deleting the affected parts doesn’t guarantee you’ve caught everything. There may be a backdoor left, someplace, that will cause you more problems in the future.

Once you are sure you’ve cleared that server, check the other systems on your network. It’s quite unlikely that if someone has managed to gain unlawful access to one server, they’ve stayed on it - unless you’ve caught them early in their investigations. So repeat these checks on other servers until you are satisfied you know the extent of the intrusion.

This feature can be read in conjunction with Security: keeping Unix servers secure