Hosting nearly 13,000 domains on our shared hosting and reseller hosting platform means that we investigate dozens of compromised accounts each month. Once we find a hacked account we could just notify the account owner that their account has been hacked and restore the account from a backup taken before it was hacked with a warning to the customer that they must update the scripts in their account. However the security of each account is closely related to the security and stability of the entire server. A hacked account can be used to launch an attack on other websites, or it can monopolize CPU/memory/bandwidth resources and degrade service for other customers or worse, the hacking of a non-privileged account on a server (ie a customer account) can lead to a server-wide root exploit if the attacker stays quiet until a privilege escalation in the operating system is found. For these reasons, we investigate each and every security incident and hacked account that we find to ensure that the attack vector is closed and the account is secured. Thankfully our backup system, centralized logging systems and other security systems allow us to investigate and recover from such attacks on our hosting servers. The ultimate goal of any hacking-related investigation is to locate & fix the attack vector (how the hackers got in) and restore the customer files from a time before the attack (we can usually go back up to a year or so thanks to our R1Soft Backup System!).
Since we do not offer SSH access, generally there are 2 ways through which accounts on our web hosting platform are compromised – both of which will allow attackers to take full control of the compromised account:
- FTP login credentials (username and password) have been stolen by a trojan or virus installed on the computer of someone who has FTP access to the account -or- if the FTP usernames and password, which are sent unencrypted over the Internet, were intercepted by an eavesdropper or hacked computer on an unsecured WIFI network, hub-based network or unsecured Internet connection.
- A Perl or PHP script uploaded by the account owner is outdated or vulnerable to exploitation or if the account owner leaves a file manager (with the ability to edit, create or upload files) on their website without password protection
With that in mind, let’s investigate a hacked account! Today I received the following notification from one of our cPanel servers (the username ‘bob’ has been changed to hide the identify of the account):
This is an automated status warning from server.
The process (24133) has exceeded defined resource limits, as such a kill signal was invoked from the process resource monitor.
– Event Summary:
PID : 24133
CMD : /usr/local/apache2/bin/httpd
CPU%: 97 (limit: 65)
MEM%: 0 (limit: 25)
PROCS: 1 (limit: 150)
What caught my eye here is the command of the running process (CMD): /usr/local/apache2/bin/httpd – that is suspicious because I happen to know that the server in question runs Apache 1.3 with the Apache web server httpd binary in /usr/local/apache/bin/httpd After logging into the server I found no running processes by the hacked user to investigate. I immediately suspended the account and copied the account website access logs (so that they are not rotated out by cPanel). If there were processes running by the hacked account, I would change into the /proc/$processID directory and look at where the process executable file really is (/proc/$processID/exe links to the running process file), look at the contents of the /proc/$processID/cmdline file to see what command line arguments where used to execute the file, I’d look at the contents of /proc/$processID/environ to see what environment variables were set and I’d look at the contents of the /proc/$processID/fd/ directory to see what files the process was reading and writing. I did a quick search of the FTP logs (/var/log/messages and /var/log/messages.*) for any FTP activity on the account. Nothing. So, at this point, confident that FTP was not the attack vector I was fairly sure that the account owner had left an outdated or vulnerable PHP or Perl script in their account which was exploited by hackers. I’ve seen this thousands of times. I then ran the following command to find any files that the hackers may have created or modified recently in the account:
find /home/bob/public_html -mtime -30 -o -ctime -30 -ls
Back to the hacked account, bob. So, I knew that the hackers were running processes, but they hadn’t left any files in the users home directory, so the last places to check were the temporary directories (/tmp and /var/tmp) on the server.
root@server [~]# find /tmp -user bob
Bingo. Checking the file, it is indeed a Perl backdoor script (MD5 checksum: 1960d67656bdca390b93d2827872b7e2). So, now we can check the timestamp of the file and see when the file was modified. We can then use that timestamp to see what script ran on the hacked website at that time.
root@server [~]# stat /tmp/back.txt
File: `/tmp/back.txt’Size: 1246 Blocks: 8 IO Block: 4096 regular file
Device: 3fh/63d Inode: 300518823 Links: 1
Access: (0644/-rw-r–r–) Uid: (32239/bob) Gid: (32238/bob)
Access: 2010-05-26 05:52:40.000000000 -0400
Modify: 2010-05-04 03:05:24.000000000 -0400
Change: 2010-05-26 05:52:40.000000000 -0400
The output of ‘stat’, specifically the ‘Change:’ line, tell us that the file inode was created or changed at 2010-05-26 at 05:52:40 EST. Now we can use that time to lookup entries in the web server logs:
root@server [~]# grep ’26/May/2010:05:52:’ /usr/local/apache/domlogs/bob/*
It looks like cPanel has already rotated out the logs, bummer. Happily, there is a suPHP log which contains a log of PHP scripts that were executed (notice that I don’t search for the exact second since the file timestamp might vary from the suPHP log timestamp by a second or two):
root@server [/usr/local/apache/logs]# grep ‘May 26 05:52:’ /usr/local/apache/logs/suphp_log | grep bob
[Wed May 26 05:52:39 2010] [info] Executing “/home/bob/public_html/blog/pingserver.php” as UID 32239, GID 32238
Now we know that the hacked script is /home/bob/public_html/blog/pingserver.php which, upon examination, is “pMachine Publishing Software Version 2.3”. Copyright 2003 (7 years old). A quick google search revealed that this software is vulnerable (http://secunia.com/advisories/9045/) – when register_globals is enabled. That is because the variable $pm_path which is used in include() statements can be overridden by passing a variable with that name via GET, POST or cookies. The authors of the software were aware of this vulnerability in earlier versions and released 2.3 to fix that – however their fix, in a file named pm/inc.lib.php is:
While that would appear to be a viable fix, the script is still vulnerable. Their fix stopped GET requests with pm_path set, but register_globals will also read from POST and cookies – so if an attacker set pm_path in a cookie or a POST request to the script they will be able to include arbitrary files and execute any code as the account owner. At this point I thought I had found the attack vector, but I was wrong. Why? Because, while this customer did have register_globals enabled, they did have allow_url_fopen disabled – so the call to include() with a remote URL failed. That means that attackers had to include a local file on the server to run malicious code. Since, prior to the attack, there were no malicious files in the users home directory or the temporary folders I concluded that they must have used a different method.
So how did the attackers exploit this script? I noticed a php error_log file in the directory and decided to read through it. In the error_log file I found:
[26-May-2010 05:52:39] PHP Warning: Unterminated comment starting line 2 in /home/discover/public_html/blog/pm/xmlrpc/xmlrpcs.inc(402) : eval()’d code on line 2
That time corresponds exactly to the timestamp of /tmp/back.txt and the access to pingserver.php. And the error log is indicating that there is an eval() statement being called with some unexected or syntactically incorrect input. If the hackers were able to manipulate the data being eval()’d then they could easily execute arbitrary code and take control of the account. A quick google search for ‘xmlrpc exploit eval’ shows that there was a huge vulnerability in xmlrpc (Slashdot: PHP Blogging Apps Open to XML-RPC Exploits). Mystery solved and we now know how the account was hacked :)
Since cPanel had rotated the access log for the domain before we were notified of the attack we are unable to pinpoint the exact IP address responsible for the hack, but we did find these POSTs to the script on logs for the remainder of the day:
188.8.131.52 – – [26/May/2010:21:24:03 -0400] “POST /blog/pingserver.php HTTP/1.1” 200 330 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6” 184.108.40.206 – – [26/May/2010:21:25:59 -0400] “POST /blog/pingserver.php HTTP/1.1” 200 323 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6”
220.127.116.11 resolves to staff.antisecurity.net – so whether the guys at antisecurity.net had some something malicious or if they were investigating something, I don’t know – but I would like to find out.
PS – while investigating another hacked account a few hours later, I *did* find a way (two ways actually) that hackers could have used to manipulate the local file include attack to run arbitrary PHP code. While it was not used in the attack that I documented in this post, it was used on another account and it is very clever. In my 10+ years of experience I’ve never seen this type of attack used first hand, but I’m sure some of our customers have been hit by it in the past. I’ll post the details of that in a future post.