If an illegal hacker wants to do something to your system, such as plant a virus, a Trojan horse program or spyware, he has to gain access to the system's root directory and the unlimited power that goes with that access.
Once established as root, the intruder can modify system commands to hide his tracks from the systems administrator and preserve his root access. The easiest way to do this is via a rootkit.
Generally, a hacker obtains normal, user-level access to a computer or network by guessing or stealing a password or exploiting some known vulnerability. Then he finds a way to collect user identities and passwords to other machines on that network while simultaneously erasing all evidence of his activity. Years ago, the hacker would have done this by exploiting his direct knowledge of and experience with the system and his personal programming skills. Today the job is simplified - the hacker can use one of many available rootkits that pretty much automate the process.
Originally, the term rootkit referred to a set of modified and recompiled Unix tools (typically including ps, netstat and passwd) designed to hide any trace of the intruder's presence or existence. David O'Brien has traced the lineage of rootkits back to the early 1990s, when Solaris and Linux operating systems were the primary targets. Rootkits are no longer limited to Unix-like systems; similar tools are available for other operating systems, including Microsoft Windows.
The name rootkit may suggest a set of canned attack scripts for obtaining root access, but this is not really the case. A rootkit may include programs to monitor traffic, create a back door into the system, alter log files and attack other machines on the network. In almost all cases, a rootkit itself causes no direct damage. Instead, its function is to mask the presence of other types of (usually malicious) software, such as keylogging Trojan horses, viruses or worms. Rootkits do this by hiding or removing traces of log-in records, log entries and related processes.
Some rootkits replace the binary files for system commands with modified versions designed to ignore attacker activity in order to escape detection. For example, on a Unix or Linux system, the rootkit may replace the list files command (ls) with one that ignores files located in specified directories. Or it may replace the ps command, which lists processes running on the system, with a similar command that ignores any processes that the attacker has started. Programs that log system activities can be similarly modified, so that when the systems administrator checks the logs, everything looks normal despite the fact that the system has been compromised.
Both rootkits and computer viruses modify core software components, inserting code to hide their presence and perform some additional function (what is called the payload). The key difference is that the computer virus attempts to spread itself to other systems, whereas a rootkit generally limits itself to a single system.
The rootkit's payload attempts to maintain the integrity of the rootkit itself -- i.e., to ensure that the target system remains compromised. For example, every time a computer runs one of the rootkit's commands, the rootkit also checks to see that other system commands on that machine are still compromised and reinfects them as necessary. The rest of the payload generally involves back doors, hidden command-line switches or "magic" environment-variable settings that circumvent normal access controls.
A rootkit sitting inside one of your systems is prima facie evidence that your system has been hacked, and it's something you want to know about. One of the rootkit's main goals is to hide its very existence, but you can detect user-mode rootkits, which accomplish their task by replacing binaries, by looking for changes in the size, date and checksums of key system files.
Kernel-mode rootkits are harder to find, because they take advantage of Unix's (or Linux's) ability to load kernel extensions on the fly. These rootkits sit deep inside the operating system, intercepting system calls from legitimate programs and returning only the data the attacker wants you to see. The fundamental problem in detecting rootkits is that you can't trust your operating system. You can't believe what the system tells you when you request a list of running processes or files in a directory.
One way to get around this is to shut down the suspect computer and check its storage after booting from alternative media that you know are clean, such as a rescue CD-ROM or a dedicated USB flash drive. A rootkit that isn't running can't hide its presence, and most antivirus programs will find rootkits by comparing standard operating system calls (which are likely to be altered by the rootkit) against lower-level queries, which ought to remain reliable. If the system finds a difference, you have a rootkit infection.
How do you get rid of a rootkit infection? Removing rootkits presents two distinct problems: removal of the rootkit itself, then removal of the payload the rootkit was hiding. Because rootkits change the operating system, you might not be able to remove the rootkit without causing the system (especially a Windows machine) to become unstable.
Russ Cooper, founder of the NTBugtraq mailing list, notes that "only a person with very little knowledge would try to remove a rootkit." Ultimately, the only safe and foolproof way to handle a rootkit infection is to reformat the hard drive and re-install the operating system.