What to do if your server is compromised

If you think someone else is on your server, don't panic, and don't start deleting things. Work in order: contain it, look around, save what you need, rebuild, and close the hole they came in through.

Cleaning a compromised system in place almost never works out. Someone with root can hide nearly anything, and you'll never be sure you got it all.

What it tends to look like#

• CPU pinned at 100% by a process you don't recognise. Very often a cryptocurrency miner.
• New users, unfamiliar keys in authorized_keys, cron jobs you didn't write.
• Outbound traffic you can't explain, or an abuse report from us or anyone else.
• Your site's files changed, odd redirects, or spam going out from your domain.
• Logins in last from places you've never been.

A quick look around#

These only read, they don't change anything. Run them as root:

last -n 20
who
ps auxf --sort=-%cpu | head -n 20
ss -tupn
crontab -l; ls -la /etc/cron.* /var/spool/cron/crontabs
cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys
awk -F: '$3 == 0 || $3 >= 1000 {print $1}' /etc/passwd
journalctl -u ssh --since "-2 days" | grep -i accepted

You're looking for anything you can't explain: an unknown process with outbound connections, a second account with UID 0, a key you didn't add, a login from an address you've never used.

One caveat. A compromised system can lie to these very tools, so a clean result doesn't prove the server is clean.

Contain it#

First, shut the attacker out without shutting yourself out. Allow SSH only from your own address, and block outgoing traffic (replies still get through) so the server can't go on attacking others:

ufw allow from 203.0.113.7 to any port 22 proto tcp
ufw delete allow OpenSSH
ufw default deny outgoing
ufw enable

Use your own public address in place of 203.0.113.7. Then run ufw status numbered and delete any other rule that still lets the world reach port 22.

Next, change every secret the server held: database passwords, API keys for other services, any VPSNine API token stored on it. Assume everything on that disk has been read.

And check your VPSNine account itself: the sessions and activity log, your SSH keys, your tokens. If you used the same password on the server, change your account password too.

Save what you need#

Copy off your data, and the logs as evidence, before you rebuild:

rsync -a [email protected]:/var/www/ ./rescue-www/
ssh [email protected] 'tar czf - /var/log' > logs.tar.gz

Treat all of it as untrusted. Restore data, never scripts, binaries or cron jobs, and look at what you restore.

Rebuild#

Reinstall, or create a new server, from a fresh image. Then go through the first-hour checklist and the SSH hardening checklist. Install your software from where it originally came from, not from the old disk. Restore data from a backup made before the break-in if you have one (Back up your server). If your old private key was ever on that server, make new keys.

Work out how they got in#

Otherwise it happens again next month. The usual suspects: password logins left on with a weak or reused password; an out-of-date web app or plugin; something listening publicly that shouldn't be, like a database, Redis or a Docker port; a leaked key. The logs you saved and last usually point the way.

Then open a support ticket, especially if the server sent spam, attacked anyone, or you got an abuse report. Telling us early lets us help, and makes it clear the server was broken into rather than misused on purpose. If we spot abuse from a server first, we may warn you or suspend it to protect others, as the Acceptable Use Policy explains.