What had happened

My server got hacked, and I received an email from my ISP regarding that. They said that my server was doing port scans and tried to brute force login into a honeypot. Immediately after I finished reading the mail, I checked AbuseIPDB for a database listing about my server, only to be faced with 18 abuse reports.

Locating the server

I was almost certain that the malware had infected one of my virtual machines running on my server at home. I logged into my Proxmox instance web UI and sorted hosts by CPU usage (I thought that maybe I got some mining malware too).

The first host that popped up is not supposed to have that high CPU usage. The VM is used only for programming and I was not doing any work at the moment. When I logged into the server and ran htop I was greeted with multiple processes by user temp.

The running programs were originating from /tmp/.X2s9-unix folder. In the folder there was a lonely dota3.tar.gz file and a hidden folder named .rsync (not visible in the picture).

Running the dota3 file through VirusTotal resulted in 39 malware detections. From the detections names I could figure out that the high CPU usage was due to a crypto mining process running in the background.

I found an article from CounterCraftSec that had information about the malware and an SSH public key on my server that matched the one in the article.

root@fi-op-terminal-1:/home# cat temp/.ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr

How this happened

I had been experimenting with cloud-init and decided to build my own image using it for my Proxmox server. That was the first mistake. I should have just used an image from https://cloud.debian.org/images/cloud/.

I was able to find the problem very fast thanks to my documentation of the cloud-init template. The documentation had the following lines.

Other user
Name: `temp`
Username: `temp`
Password: `temp`

Deleted later

That was it. The user was never actually deleted, and someone had just found my server and tried to log in with user temp and password temp. The article from CounterCraftSec describes that the malware is distributed via weak SSH servers and I had exactly one like that.

It is based on a botnet that attacks weak SSH servers using dictionaries with default or common credentials

Preventing this from happening in the future

With a proper SSHD config, I would have avoided this incident completely. The following rules would have stopped an attacker:

• Public key is required for login
• Password login is not allowed
• SSH access is allowed only for users in a specific group

In the end, I had to reformat the server completely, since the malware could have left some hidden hooks to the OS. Thankfully, I used that server only for programming and did not lose anything important. I do have daily backups of important services in case something similar happens again.

Keyring

During the migration period from the old to the new, I was using a simple paracord loop to hold my keys together. I wouldn't recommend this to anyone who cares about their access to the home. The knot on the paracord is just too flimsy and has to be retightened constantly. One could also fix this by learning how to tie a knot.

On the previous keychain, I had a sturdy ring repurposed from an old Kevlar key retractor that was broken. It was made to last and wouldn't bend ever (too many keyrings look like hardened butter).

Tracking

I would have never thought about needing my keychain to know its location and telling it to me until I lost mine. Choosing a way to track the keys was easy. I had an iPhone and Apple had announced two years ago (in 2021) its competitor to Bluetooth trackers. Their tiny coin-sized AirTags were able to transmit the location of your lost stuff to your iPhone by utilizing a large network created by other Apple devices.

The Apple pricing though is ridiculous for the keyring attachment. I wouldn't ever pay close to 50 euros for a piece of leather (even the Airtag costs less). I found quickly "Belkin Secure Holder for Airtag" that seemed to fit my use case exactly and cost less than the official Apple variant.

Multitool

I had experience with Victorinox tools attached to my keys but had been eyeing a Leatherman for quite some time. When choosing a replacement for the new keychain, I settled on the red Leatherman Squirt PS4. I was surprised at how small it is, but I don't mean that it is bad. It is the best size for something to carry alongside keys in your pocket every day.

USB flash drive

I have a 128 GB Kingston USB drive. Ventoy is a tool that simplifies the creation of live USBs. Normally you would have only one operating system on a live USB drive, but with Ventoy, you can fill all the storage space on the drive with ISO files. Ventoy has a single partition where you can copy your Linux, Windows, and other ISOs. I have downloaded:

• Arch
• Ubuntu (Desktop & Server) 22.04 LTS
• Windows 10
• Kali Linux
• Clonezilla

Keys

What are those!?

NOTE: Attach pictures later.