Background:
OpenVZ is a (stripped down) free, open-source version of Virtuozzo linux virtualization software. The modified OpenVZ kernel allows server operators to partition their servers into multiple Virtual Environments running a different Linux distribution.

Outdated Kernel:
As of now (Oct 10th, 2006) the latest Kernel listed on the RHEL4 download page (version 2.6.9-023stab016.2) is vulnerable to a root exploit that was first reported in July of 2006. That means that OpenVZ has had the vulnerable kernel available for download for around 3 months!

Response from OpenVZ: (*UPDATE*)
The response from OpenVZ was quick & effective – we contacted them at around 10PM on Oct 10th and by 6AM on October 11th (~ 8 hours) they released an updated version (2.6.9-023stab030.1). This does not negate the fact that a vulnerable kernel was left available for download for ~3 months, but I am quite pleased with their response.

update 2: OpenVZ sent an email to their list today (October 11th) at around 1PM EST saying “Everybody using 023 kernel is advised to upgrade.” – perhaps they should have mentioned the root exploit in the email as a reason to drive people to upgrade.

Damage:
This only effected the OpenVZ kernels, not the Virtuozzo kernels. Our paid Virtuozzo installations were in the 2.6.8 branch which was not affected.  A handful of our OpenVZ servers running 2.6.9 were vulnerable – we’ve updated them immediately. Unfortunately we became aware of this because one of the servers was actually exploited.


Server Security & Incident Tracking:
It goes without saying that if an attacker manages to get root access to a server, somewhere a sysadmin will forgo a night of sleep trying to recover.

‘root’ access to a server is absolute – root is the ultimate Unix user. Once an attacker gains root access, he/she can do anything. Cleaning a box that has had a root exploit is a nightmare, and many will argue not even possible.  Because the ‘root’ user has the ability to modify anything on the system, any system binary can be replaced with a trojan’d version. Any configuration file can be changed to allow an attacker access through an unexpected port, ssh keys can be added to let an attacker in and cronjobs can be put in place to ensure that their exploits will stick around even if a sysadmin deletes them. An attacker can add a new user to /etc/passwd with uid ‘0’ (root). The list goes on (and I don’t want to give malicious people any more ideas!)

Having a malicious entity gain ‘root’ access to a server is a worst-case scenario for any system administrator.

How do you know if you were rooted?
There are many obvious signs:

  • log files disappear
  • suspicious processes are running on the server
  • programs with names like ‘sendmail’ are running on a non-standard port
  • files will be modified

Many system administrators will just know when something does not feel right.

What can you do?

Arguably the most important thing that must be done after an attack is finding the source of the exploit; what php script was exploited? what kernel bug was exploited? etc    If you don’t close the security hole, the hackers will just jump back in.

There are many ways that you can diagnose your system for changes and unusual activity:

  • Check the logs (assuming they weren’t deleted)
  • use the unix ‘find’ command to search for files that have been modified or created in the last X days
  • use RPM –verify (if you are running an RPM-based distribution)  to verify that binary files are not replaced malicious ones
  • Use ‘netstat -apn’ to look at incoming and outgoing sockets and inspect the output for unusual items.
  • hire someone who has experience in these situations

Most of the time attackers don’t clean up after themselves – while they will delete the server logs to cover their tracks, they will leave behind the scripts that they use – these will be invaluable tools to discover how they exploited your system. Time stamps are also keys to finding out what was changed or added to your system.

“Why did the hacker choose me!”
This is a common question that we get from shared hosting customers who have vulnerable PHP scripts or forums. The answer is, these low-lifes have automated tools that search the internet for vulnerable scripts & forums – and then they notify the attacker of the vulnerabilities so that attacker can proceed.

Most of the time (especially in mass-defacing situations) attacker doesn’t have a grudge against your personal website and they are not targeting your website for any reason other than it is vulnerable.

Most of the attackers that we have dealt with have 1 goal: replace all website files with their own political or religious messages…. and to gloat to their underground, hacker friends.

What is even worse is that you have websites with archives of hacks and records of what hacker defaced what website in the form of a competition – which hackers have defaced the most websites today? Websites shouldn’t be encouraging hackers to increase their hack count!


Hacking in a Hosting Environment
In the context of a web hosting situation, there are 2 important types of exploits:
* ‘localized’ Exploits
* Server-Wide Exploits

An example of a “‘localized’ exploit” would be when a customer who is running an outdated PHP script gets attacked. The customer then gains access to the customers username and overwrites their files, can read their emails & confidential files, etc.  For a web hosting company, this is expected and of ‘minor’ significance. For a customer, this may be the end of the world – files are gone, data is missing or modified and they feel victimized.

What scares system administrators is the server-wide exploits. This can be a direct attack (perhaps an SSH deamon has a vulnerability?) or this can be the result of an attacker who used a ‘localized’ exploit to escalate his/her privileges to ‘root’ level. A server-wide exploit is terrifying for web hosts. While web hosting companies will always tell customers that it is the customers responsibility to backup their files, the web hosting company has a job to do: keeping customer files online & accessible 24×7.

Backups
When the worst possible scenario becomes a reality, the web hosting company will usually turn to its backups. Backups come in many shapes and forms – local harddrives to store backups,  remote backups and RAID (though that’s not really a backup method… it’s a redundancy method to protect against drive failure) are just 3 examples. Many hosts employ combinations of local & remote backups.

The problem is: If you store backups on a local server, an attacker can delete them. But, the cost of storing backups on a remote server is measured in additional administrative time & coordination, the cost of more bandwidth and the cost of the external storage space – this can add up to be an expensive proposition, especially if you are backing up to a remote datacenter at fast speeds – the bandwidth toll is expensive. In a web hosting environment, backing up dozens of servers with data retention spans of 1-3 months can require many TB of storage.

Another important decision is the backup schedule: will you backup everything each night or backup important things each night etc. Backing up an entire server each night would increase the CPU load, require much more storage and more bandwidth. Another option is backing up website files (the bulk of the data) once a month and everything else each night. This will help reduce the storage, bandwidth & CPU requirements, but the result will be that you may have to settle for a 1 month old backup if your files are removed.


The moral of the story:

Customers: Keep your scripts updated! Help provide a first line of defense for the server that your website is on. ALWAYS keep backups of your website data on your computer.

Web Hosting Companies: Keep your servers updated! Make sure that you update nightly & that you have good practices in place to help detect, quarantine and recover from an attack.

Comments are closed.