Web Hosting Blog

Warning to all: 'latest' RHEL4 OpenVZ Kernel has a root exploit!

Wed, 11 Oct 2006 13:24:01 -0400

Background:
OpenVZ is a (stripped down) free, open-source version of Virtuozzo linux virtualization software. The modified OpenVZ kernel allows server operators to partition their servers into multiple Virtual Environments running a different Linux distribution.

Outdated Kernel:
As of now (Oct 10th, 2006) the latest Kernel listed on the RHEL4 download page (version 2.6.9-023stab016.2) is vulnerable to a root exploit that was first reported in July of 2006. That means that OpenVZ has had the vulnerable kernel available for download for around 3 months!

Response from OpenVZ: (*UPDATE*)
The response from OpenVZ was quick & effective - we contacted them at around 10PM on Oct 10th and by 6AM on October 11th (~ 8 hours) they released an updated version (2.6.9-023stab030.1). This does not negate the fact that a vulnerable kernel was left available for download for ~3 months, but I am quite pleased with their response.

update 2: OpenVZ sent an email to their list today (October 11th) at around 1PM EST saying "Everybody using 023 kernel is advised to upgrade." - perhaps they should have mentioned the root exploit in the email as a reason to drive people to upgrade.

Damage:
This only effected the OpenVZ kernels, not the Virtuozzo kernels. Our paid Virtuozzo installations were in the 2.6.8 branch which was not affected. A handful of our OpenVZ servers running 2.6.9 were vulnerable - we've updated them immediately. Unfortunately we became aware of this because one of the servers was actually exploited.

Server Security & Incident Tracking:
It goes without saying that if an attacker manages to get root access to a server, somewhere a sysadmin will forgo a night of sleep trying to recover.

'root' access to a server is absolute - root is the ultimate Unix user. Once an attacker gains root access, he/she can do anything. Cleaning a box that has had a root exploit is a nightmare, and many will argue not even possible. Because the 'root' user has the ability to modify anything on the system, any system binary can be replaced with a trojan'd version. Any configuration file can be changed to allow an attacker access through an unexpected port, ssh keys can be added to let an attacker in and cronjobs can be put in place to ensure that their exploits will stick around even if a sysadmin deletes them. An attacker can add a new user to /etc/passwd with uid '0' (root). The list goes on (and I don't want to give malicious people any more ideas!)

Having a malicious entity gain 'root' access to a server is a worst-case scenario for any system administrator.

How do you know if you were rooted?
There are many obvious signs:

log files disappear
suspicious processes are running on the server
programs with names like 'sendmail' are running on a non-standard port
files will be modified

Many system administrators will just know when something does not feel right.

What can you do?
Arguably the most important thing that must be done after an attack is finding the source of the exploit; what php script was exploited? what kernel bug was exploited? etc If you don't close the security hole, the hackers will just jump back in.

There are many ways that you can diagnose your system for changes and unusual activity:

Check the logs (assuming they weren't deleted)
use the unix 'find' command to search for files that have been modified or created in the last X days
use RPM --verify (if you are running an RPM-based distribution) to verify that binary files are not replaced malicious ones
Use 'netstat -apn' to look at incoming and outgoing sockets and inspect the output for unusual items.
hire someone who has experience in these situations

Most of the time attackers don't clean up after themselves - while they will delete the server logs to cover their tracks, they will leave behind the scripts that they use - these will be invaluable tools to discover how they exploited your system. Time stamps are also keys to finding out what was changed or added to your system.

"Why did the hacker choose me!"
This is a common question that we get from shared hosting customers who have vulnerable PHP scripts or forums. The answer is, these low-lifes have automated tools that search the internet for vulnerable scripts & forums - and then they notify the attacker of the vulnerabilities so that attacker can proceed.

Most of the time (especially in mass-defacing situations) attacker doesn't have a grudge against your personal website and they are not targeting your website for any reason other than it is vulnerable.

Most of the attackers that we have dealt with have 1 goal: replace all website files with their own political or religious messages.... and to gloat to their underground, hacker friends.

What is even worse is that you have websites with archives of hacks and records of what hacker defaced what website in the form of a competition - which hackers have defaced the most websites today? Websites shouldn't be encouraging hackers to increase their hack count!

Hacking in a Hosting Environment
In the context of a web hosting situation, there are 2 important types of exploits:
* 'localized' Exploits
* Server-Wide Exploits

An example of a "'localized' exploit" would be when a customer who is running an outdated PHP script gets attacked. The customer then gains access to the customers username and overwrites their files, can read their emails & confidential files, etc. For a web hosting company, this is expected and of 'minor' significance. For a customer, this may be the end of the world - files are gone, data is missing or modified and they feel victimized.

What scares system administrators is the server-wide exploits. This can be a direct attack (perhaps an SSH deamon has a vulnerability?) or this can be the result of an attacker who used a 'localized' exploit to escalate his/her privileges to 'root' level. A server-wide exploit is terrifying for web hosts. While web hosting companies will always tell customers that it is the customers responsibility to backup their files, the web hosting company has a job to do: keeping customer files online & accessible 24x7.

Backups
When the worst possible scenario becomes a reality, the web hosting company will usually turn to its backups. Backups come in many shapes and forms - local harddrives to store backups, remote backups and RAID (though that's not really a backup method... it's a redundancy method to protect against drive failure) are just 3 examples. Many hosts employ combinations of local & remote backups.

The problem is: If you store backups on a local server, an attacker can delete them. But, the cost of storing backups on a remote server is measured in additional administrative time & coordination, the cost of more bandwidth and the cost of the external storage space - this can add up to be an expensive proposition, especially if you are backing up to a remote datacenter at fast speeds - the bandwidth toll is expensive. In a web hosting environment, backing up dozens of servers with data retention spans of 1-3 months can require many TB of storage.

Another important decision is the backup schedule: will you backup everything each night or backup important things each night etc. Backing up an entire server each night would increase the CPU load, require much more storage and more bandwidth. Another option is backing up website files (the bulk of the data) once a month and everything else each night. This will help reduce the storage, bandwidth & CPU requirements, but the result will be that you may have to settle for a 1 month old backup if your files are removed.

The moral of the story:

Customers: Keep your scripts updated! Help provide a first line of defense for the server that your website is on. ALWAYS keep backups of your website data on your computer.

Web Hosting Companies: Keep your servers updated! Make sure that you update nightly & that you have good practices in place to help detect, quarantine and recover from an attack.

Virtuozzo / OpenVZ networking broken for Redhat 7.X VEs

Sun, 01 Oct 2006 14:23:11 -0400

The problem:
A few months ago we noticed that after updating either Virtuozzo or OpenVZ utilities, we would no longer be able to reboot Redhat 7 virtual environments (VEs - or VPS [virtual private servers]).

We tracked this down to the fact that Virtuozzo and OpenVZ have the code:

CP='/bin/cp -f --preserve=mode,ownership'

in the files: dists/scripts/redhat-add_ip.sh & dists/scripts/functions

The above scripts are executed in the VEs to setup networking - the problem is that RedHat 7 supports only 'cp --preserve' and not 'cp --preserve=...' and therefore the startup scripts can't run and setup networking in the VEs.

The Solution:
The solution is easy:

Just modify: dists/scripts/redhat-add_ip.sh & dists/scripts/functions (in /etc/vz for OpenVZ or /etc/sysconfig/vz for Virtuozzo) and remove the "=mode,ownership" text. That will fix it.

Important: You must manually make the above changes after any time that Virtuozz or OpenVZ releases a new version of the virtuozzo tools because they will override this.

(update) Bug Report submitted to OpenVZ:
We opened a bug report with OpenVZ and they've responded, however we're unsure if they will fix it - we asked for clarification but we've received no response.

It's time to punish spammers

Tue, 19 Sep 2006 12:45:26 -0400

This post focuses on the S-word. SPAM.

How can ISPs & Web Hosts stop spam? How can we fight back? What tools can we use to fight back? What methods can be used on the server-level to protect end-user inboxes?

There are some things in life that just make me smile:

Ice cream
Watching Borat sing "Everybody dancing now"
Brokeback Mountain becomes Spongebob brokeback

But this makes me smile from ear to ear:
Earthlink awarded $11Million from Spammer

Spam is a problem that is plaguing not only end-users but web hosts, ISPs, backbone providers and network administrators as well.

While the CAN-SPAM legislation is weak, it provides an essential first-step towards setting up the battle in the legal arena to fight spam. Making it illegal to forge headers and return addresses provides companies with a legal basis for prosecuting spammers in the United States. Obviously more needs to be done, but the CAN-SPAM act is better than nothing.

It is obviously every good netizen's (net citizen?) dream to eliminate SPAM. SPAM has turned one of the quickest and far-impacting methods of communication into a daily hassle and waste of time.

SPAM is damaging the internet community in many ways. A few of the main problems caused by spam:

End user frustration. End-users are frustrated by the amount of SPAM in their inbox and eventually, instead of experiencing a life-changing method of communicating with relatives in another country or engaging in commerce, end-users are forced to sift through myriads of messages to weed out the ones that they want to read. In extreme cases, this deluge of spam may even cause light-weight users to simply stop using E-mail.

Hijacked computers. A large portion of bulk email is sent from hijacked and compromised computers. While there are many spammers who rent their own servers, there are networks of hijacked PCs which are sold in blocks of thousands for use by spammers. Ignoring the fact that such behavior is illegal, anyone who has used or tried to disinfect a hijacked PC knows that they often slow to a crawl, crash or they consume an entire house-holds worth of bandwidth which will result in degraded performance of other computers. Just like the point above, this frustration will lead many users to abandon use of their computers or waste money on having their computers repaired.

Lost emails. A direct result of SPAM is the loss of legitimate and valuable emails.
1. Accidental Deletion. Legitimate emails are often lost in the process of a user repeatedly clicking 'Delete' while clearing their Inbox of SPAM.
2. Spam Filters. To combat SPAM, many E-mail service providers filter incoming email for SPAM & Virii. It is unrealistic to believe that SPAM filters will never accidentally tag a legitimate email as spam. When this happens, either the email will be discarded by the E-mail providers servers or the message will wind up in the Spam-folder where it may get discarded before the end-user can review it and realize that it was not spam.

Wasted time & productivity. According to a Linux News article, Spam costs $20Billion - yes billion - each year in lost productivity and that spam costs enterprises between $600 and $1,000 each year for every user.

ISP, Backbone, Network Administrator & Servers effects. Unquestionably, SPAM has caused a huge headache for network operators of all types. Throughout the internet chain, from the Email service provider to the backbone providers SPAM is costly.
1. Web Hosting providers & E-mail service providers are forced to setup spam & virus filtering systems to protect their end-users. For large web hosting companies, ISPs and enterprises this means the additional overhead of purchasing dozens, hundreds or even thousands of spam scanning servers as well as the additional overhead & staff time of managing all these systems.
2. ISPs (ie residential DSL/Cable modem providers as well as enterprise connectivity providers) and network backbone operators are forced to expand their network to carry the new deluge of spam & virii. Hijacked computers can easily send out thousands or tens of thousands of SPAM/Virii emails each day. Having to deal with this extra (unwanted) network traffic, ISPs will raise their rates - because even if you are not using your internet connection, your hijacked computer sure is.
3. System / Network Administrators. In addition to end-user or employee assistance, help desks & system administrators now have to worry about finding infect computers, cleaning infected computers and educating end users about safe computer practices. It takes valuable time for a network administrator to locate infected machines (most often because of inadequate logging or firewall policies).
4. Web Hosting Companies. Insecure sendmail scripts are now exploited thousands of times on a daily basis to send spam. Often times this results from customers running an old version of a publicly available PHP or Perl script. What is alarming is that spammers are now starting to exploit custom written mailing scripts - using search engines to find email forms and then testing them each individually for vulnerabilities. This shows extreme dedication on the part of the spammers - they are testing custom written, unknown scripts form vulnerabilities and exploiting them. It has now become the web hosts job to location insecure scripts and notify customers. Large-scale exploitation of insecure scripts can endanger a web hosts standing with ISPs or even get their email servers listed on public black lists such as SPEWS or SORBS - resulting in a portion of the internet not accepting email from customers on those servers.

A real life analysis

Figure 1.

The above graphic is an image of data collected from just one in-bound spam filtering server that we run for our web hosting customers. This one server was processing over 100,000 spam emails per day at its peak. Spam is not a minor issue for ISPs, email service providers or web hosts - the infrastructure required to support the weeding out of spam and viruses is expensive and time consuming to operate/update.

Later on in this post I will go into detail about how we cut down the amount of spam that our servers had to process by checking a connecting computers IP address before accepting email from them.

Fighting Back
There are many ways that we can fight back against the spammers.

End User Education. There would be no spam if end-users didn't actually open them or respond to them. The economics of running spam networks and operations require that at-least a small portion of the spam recipients act on or open spam emails.
Network administrators and computer-savvy 'nerds' must start to educate the end users as to the danger of clicking on unknown popups, downloading unknown files and not updating their computer because these direct actions result in the compromising of their computers. A single click by an uneducated end user can cause havoc for system and network operators. End-users must also be taught never to reply to spam - replying will only let them validate your email address. It is also important to note that simply by opening a spam email, spammers can validate your address. Spammers will often place unique hidden (or visible) images or code in emails that will identify you as having seen the images or run the code. [tip: set your email reader not to show images in emails unless you specifically allow it.]

Spam Filtering. Email providers and web hosts are increasingly successful at filtering spam from ever reaching their end-users inboxes. Personally, I've seen the results of AOL's spam filtering and I'm very impressed. Probably one or two junk emails will get through each day - for someone who checks their email once a month, that could mean 30-60 junk emails - and that would be terrible.
Email service providers are becoming increasingly efficient at keeping spam out of their end-users inboxes. Software such as SpamAsassin can be run server-wide and help filter out unwanted emails. DNS Black lists such as Sorbs, SpamCop and spamhaus can help your server to filter out connections from unwanted IP addresses (computers which are hijacked, vulnerable etc). The problem with these methods for combating spam is simply that it requires more cpu & processing power. Checking black lists and running SpamAssassin or ClamAV (free Linux virus scanner) requires a large amount of resources, which cost money.

A powerful & effective spam & virus Protection system can be built at the Web Host / E-Mail provider level with free software. Incoming email servers should always run a combination of:

DNS Black list checks - SpamCop(.net), Sorbs(.net) etc.
SpamAssassin(.org) Spam-filter
ClamAV(.net) Virus Scanner

These 3 tools can be integrated separately into common email servers such as SendMail or you can use a package such as MailScanner to tie all3 systems together.

If you look at Figure 1. (above), you can see that when Elite Hosts started to implement RBL (real time block lists - aka DNS Block lists, a few examples given above), the amount of spam that just one of our many incoming email servers processed fell from around 60,000 messages per day to around 2,000 messages per day! If that is just from one server, imagine how much processing power we saved across all of our incoming email servers.

Another 3 useful, but lesser used technique for combating spam on the server-level are:

Require reverse DNS Requiring connecting machines to have reverse DNS will allow you to easily identify connecting servers. You can then use these results to block certain domains from sending to your email servers. For example, with the exception of their email servers, we block connection attempts from any computer with a hostname ending in comcast.net that is not a static email server. Figure 1 (above) shows that requiring reverse DNS reduced the spam messages per day processed on one server from around 2,000 to 1,000
Sender Call-Back. This is one anti-spam mechanism that is used by CPANEL. The mail server will connect to the email server of the domain in the From: address and check whether or not the sending address is a valid email address. This can help to filter out spam from those spammers who are just too lazy (or dumb?) to send you email from non-existent email addresses or domains.
SPF. This is controversial technique for verifying the sender of an email. SPF has become controversial because it uses the DNS TXT record of a domain to specify valid servers which are allowed to send email for a domain. In my opinion, loss of the (rarely used) TXT record is a good exchange for the benefits. SPF is more of an anti-fishing tool than an anti-spam tool. SPF simply looks up the domain of the sender and verifies that emails are coming from an accepted server. This means that if you receive an email from billing@paypal.com, an SPF capable email server will ask PayPal if the computer sending spam is in-fact a valid email server that is allowed to send email from the domain. The thought behind SPF is excellent - verifying that connecting servers are sending email from domains that they are responsible for - however some argue about the implementation and the use of the TXT record.
Bottom Line: If SPF stops phishing attacks from reaching end users (which it does!), then it is a step in the right direction. This will not eliminate spam, but at-least it will protect uneducated users from replying to billing@paypal.com with their PayPal passwords.

In order for SPF to be really effective, it needs to see more wide-spread usage - major companies (AOL, eBay, PayPal) already publish SPF records, but email servers need to start checking those records.

The Future of Spam
Spammers are not stupid. Like many other things in life, Spammers will adapt and change with the times.

Spammers are not oblivious to spam filters - to the contrary, spammers will work day & night to craft email messages that receive low scores on spam filtering systems. Spammers are increasingly shifting to image-based spams so that Spam-Filtering software can't find the key words and phrases that it needs to tag the messages as spam.

Anti-Spam companies are also fighting back - working to develop software that will recognize spam text in image-based spam emails and other advanced modules to help stay in-step with the spammers.

Spammers are also diversifying, experimenting with new mediums such as Instant Messages and Blogs. I have yet to receive an IM Spam (I hope that this doesn't Jinx it), but I can imagine how upset I will feel when my private IM space is invaded by a new IM window from a spammer. Spammers are also moving to blogs, forums & community systems. Spammers are creating automatic scripts that post their customers products URLs in blog comments, forum posts and more. Just like the email anti-spam companies are fighting back, these communities are coming up with methods to block this automated spam - image captchas, audio passwords and other interesting methods.

Conclusion & Summary
Unfortunately, it looks like spam is here to stay - wasting our valuable time, money and resources. There are many things that we can do to retake our inboxes and put spammers out of business:

Better end-user education
Better legislation to prosecute spammers
Better programmer education - teaching programmers how to write secure scripts that send email
Residential & Small Business ISPs should block outbound SMTP port 25 by default (with the option to enable it upon request)
Microsoft, the provider of over 95% of the worlds desktop system, has to have stronger security built into their products. How is it acceptable that 1 click can cause a program to download an take over your computer? How is it acceptable that an un-updated Windows installation can be compromised within minutes of being connected to the internet? It is excellent that Microsoft has now decided to focus on security, but it might be too late. For the last few years, Microsoft has been handing their customer's computers to hackers.
Web Hosts & Email service providers should install spam scanning software, use spam black lists, check for phishing via SPF & implement other common-sense methods for reducing spam, fishing & virus attacks. If web hosting companies & email providers could stop spam from being delivered to their end-users, spammers would make no money and simply disappear.

Hopefully we can take back the Internet and provide an amazing, worry-free & frustration-free experience for our end-users.

cPanel = CRUD PANEL

Thu, 14 Sep 2006 19:49:25 -0400

In today’s web hosting world there is a 'de-facto' control panel called cPanel. There is a large segment of reseller hosting and shared hosting customers who look for cPanel hosts. To a certain extent, many of those looking for reseller web hosting accounts are looking for cPanel hosts.

Because cPanel is one of the most established control panels in the web hosting market, if a customer transfers to a new host, choosing a host with cPanel will make it easy for them to migrate their settings and will minimize the learning curve with the new host.

Of course there are other competitors (DirectAdmin, Plesk & H-Sphere to name a few), but cPanel is simply the most wide-spread.

cPanel has become a force in the market - they have easily past the critical mass of customers that they need to be a dominant market power and they can charge whatever price they want, they can be slow with bug fixes, they can be slow with new features and they cna be slow with updates.

There are many problems with cPanel... a very breif list would be:
* While some of cPanel is open-source, there are a lot of encoded, compiled routines that are vital to its functioning. If you find a bug (and believe me there are many), you have to wait for cPanel to decide that they want to fix it.
* A lot of the cPanel code is compiled Perl - this makes extremely large and extremely slow binaries that need to run each time or whm is called.
* cPanel offers no clustering support (I don't call distributed name servers 'clustering')... scalable hosts need the ability to have separate email servers, MySQL servers, email list servers, etc). Because some vital routines are hard-coded into cPanel, it can't even be ported, upgraded or patched to do distributed hosting without major problems
* cPanel tries to offer everything to everyone (and run on over a dozen Linux/Unix platforms [and windows!]) you wind up with an installation that is simply bloated well beyond what most hosts will need. Can you fathom cPanel + windows? It's a sysadmin nightmare. What sane web hosting system administrator would want this burden on their shouldiers?

My advice to cPanel is simple: Stop trying to support dozens of operating environments, choose an OS, support it, fix it and maintain it.

There are simply so many bugs that are confirmed by cPanel but not fixed. For example this bug report was reported by us in November of 2005, confirmed by cPanel on Dec. 1st 2005 and it is still unresolved as of Today, Sept 14th, 2006.

Instead of spending their time fixing known (and confirmed) bugs and improving their software, cPanel decided to work on their own script-deployment system (cPAddons).. that'd be a very useful feature except that Fantastico for cPanel provides around 50 pre-installed scripts, blogs, message boards and more. *shock* - cPanel has wasted their time.

Reseller hosting customers have expectations from their providers: speed and reliability from the servers and quick resolution from the hosting company. cPanels compiled binaries & bloating have slowed our servers down, bloated them down with useless software and their (extremely) slow response times have simply forced us to give responses such as "this is a cPanel bug, our hands are tied until cPanel resolves this issue".

The above is an excellent summary as to why our shared web hosting system runs on our own in-house developed control panel, SimpleCP. Running our own control panel on our shared hosting servers gives us power, flexibility, scalability & performance that we could never dream of with cPanel. It is for those reasons as well that we will be creating a fast, clustered/distributed and responsive replacement for cPanel for our reseller customers.

SupportSuite fixes blank Yahoo email problem... finally?

Fri, 01 Sep 2006 09:22:57 -0400

After being first reported in April of 2006 (in this forum post), Mahesh Slaria from Kayako told the community how to fix this problem by changing some code (it is not fixed in the current stable build).

For a while now we've had customers with @yahoo.com email addresses complaining about receiving blank emails from our ticket system... after looking at the Kayako forums (we use Kayako support-suite) we found that other Kayako customers were experiencing this as well.

Finally, around 4-5 months later we get a solution:
"Blank email issue is just due to HTML encoding, you can change 'html_encoding' to '7bit' or '8bit' in en-us.php at locale folder of SupportSuite."

Sure enough, we tried it out and it fixed the problem!