Backstory for securing my SSH server

What had happened

My server got hacked, and I received an email from my ISP regarding that. They said that my server was doing port scans and tried to brute force login into a honeypot. Immediately after I finished reading the mail, I checked AbuseIPDB for a database listing about my server, only to be faced with 18 abuse reports.

Pasted image 20221025135527.png

Locating the server

I was almost certain that the malware had infected one of my virtual machines running on my server at home. I logged into my Proxmox instance web UI and sorted hosts by CPU usage (I thought that maybe I got some mining malware too).

The first host that popped up is not supposed to have that high CPU usage. The VM is used only for programming and I was not doing any work at the moment. When I logged into the server and ran htop I was greeted with multiple processes by user temp.

The running programs were originating from /tmp/.X2s9-unix folder. In the folder there was a lonely dota3.tar.gz file and a hidden folder named .rsync (not visible in the picture).

Running the dota3 file through VirusTotal resulted in 39 malware detections. From the detections names I could figure out that the high CPU usage was due to a crypto mining process running in the background.

I found an article from CounterCraftSec that had information about the malware and an SSH public key on my server that matched the one in the article. 

root@fi-op-terminal-1:/home# cat temp/.ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

How this happened

I had been experimenting with cloud-init and decided to build my own image using it for my Proxmox server. That was the first mistake. I should have just used an image from https://cloud.debian.org/images/cloud/.

I was able to find the problem very fast thanks to my documentation of the cloud-init template. The documentation had the following lines. 

Other user
Name: `temp`
Username: `temp`
Password: `temp`

Deleted later

That was it. The user was never actually deleted, and someone had just found my server and tried to log in with user temp and password temp. The article from CounterCraftSec describes that the malware is distributed via weak SSH servers and I had exactly one like that.

It is based on a botnet that attacks weak SSH servers using dictionaries with default or common credentials

Preventing this from happening in the future

With a proper SSHD config, I would have avoided this incident completely. The following rules would have stopped an attacker:

In the end, I had to reformat the server completely, since the malware could have left some hidden hooks to the OS. Thankfully, I used that server only for programming and did not lose anything important. I do have daily backups of important services in case something similar happens again.