Tech Tip: Recover from rootkits on your Linux system

Learn how to recover from rootkits on your Linux system.

By Jonathan Yarden

A rootkit is a collection of replacement programs for essential UNIX system commands that facilitates cleaning up after an exploit, and it allows an attacker to maintain access to a compromised system. You can use commercial or open source programs, such as Tripwire or chkrootkit, to detect rootkits on your system. However, finding out someone has compromised your UNIX system is one thing—cleaning up the resulting mess is entirely another.

Getting rid of rootkits can be a complicated matter. UNIX doesn't have a consistent package standard, which means that reinstalling the entire operating system is the usual method of recovery.

This isn't good news for many people, especially when the compromised system contains irreplaceable data. Backups help, and you can clean many systems without taking them offline.

Linux systems that use the Red Hat Package Manager (RPM) can quickly verify the integrity of system files by running the rpm -Va command as the root user. The command's output is a listing of files and directories that have changed in some way from their original versions, but this doesn't mean that there's an installed rootkit. Be advised that changes to command files such as netstat, ifconfig, ps, and chsh are typical with rootkits, regardless of the UNIX operating system in use.

If you suspect or know that someone has installed a rootkit, the first thing to do is unplug the network connection and try to shut down and restart the compromised machine in single-user mode. If you have a "rescue" boot disk or CD-ROM, this is the time to use it.

Once you've brought up the machine in single-user mode, you can begin the recovery process. If you don't have a "rescue" disk available for your Linux system, try searching for one on Google.

Intruders typically change the attributes of rootkit files on Linux systems to prevent someone from replacing them. Running the lsattr command on suspect files or directories will show differences when compared to other executable program files.

Files marked with the "immutable" flag ("i") are always suspect. This flag implies that someone has changed the rootkit files using the chattr command to prevent you from deleting or replacing the files.

The key to recovery is making sure that you don't change the wrong files. Rootkits frequently replace files in /bin, /sbin, /usr/bin, and /usr/sbin, which are some of the main file system locations that UNIX uses for its commands. Rootkits also place directories and files outside of these areas, but unless you're absolutely sure that these files belong to a rootkit and not to a legitimate application, you should leave them alone.

If you find a suspected rootkit file, you need to replace it with a legitimate copy. If your Linux distribution uses RPM, use rpm -q -f {file} to find the packages that provide the compromised files. Using RPM, the best way to replace the compromised files is by reinstalling the entire package, so you'll need your installation CD-ROM or the package files from some source.

But before you can replace packages, you have to "unlock" the compromised files that were marked immutable. On Linux, the chattr -i command unlocks files so you can delete or replace them. (Consult the manual pages for lsattr and chattr for more information on these commands.) After unlocking the compromised files, you can use the rpm -Uvh {package} command to reinstall the package, which replaces the compromised files.

Replacing the compromised files doesn't mean that you're out of the woods yet, but if you need to get a machine back online in a hurry, this is one way to do it. Sometimes it's more important to get a machine back online quickly than it is to figure out how the intruder compromised the machine in the first place. But this investigative work definitely should be your next step, or someone will likely compromise the machine again.

Jonathan Yarden is the senior UNIX system administrator, network security manager, and senior software architect for a regional ISP.

Editor's Picks

Free Newsletters, In your Inbox