PHP FTP + Passive FTP Server Behind NAT = nightmare.

My daily routine involves helping customers with programming / scripting issues. Normally support for applications written by anyone outside of our company are not supported by us, however I don’t mind going through people’s code to help out when I get a few spare minutes.

One of our Linux managed hosting customers was creating a PHP script to connect to an FTP server at their office. That should be simple and trivial, right? The important code that they were using looked like:

// create a basic connection
$ftpconn = ftp_connect($ftp_ip, $ftp_port);

// login with username and password
ftp_login($ftpconn, $ftp_user, $ftp_pass);

// Switch into passive mode (required for servers behind firewalls / NAT devices)
ftp_pasv($ftpconn,true);

// Print a directory listing
print_r ftp_nlist($ftpconn, "/");

Unfortunately the code kept timing out at the ftp_nlist() command. After a few minutes the script would timeout and return a warning:

Warning:  ftp_nlist() [function.ftp-nlist]: php_connect_nonb() failed: Operation now in progress (115) in /home/script.php on line 27

My first thought was that there must be a firewall issue with the FTP server that was blocking certain ports. I decided to check if I could connect to the server directly from my local computer using FileZilla. To my surprise, I was able to connect, browse, upload and download without any problems. Now I was intrigued. I decided to see if I could implement the FTP client using PHP’s curl extension – and again I was able to connect, browse, upload and download without any problems. So, given that I was able to connect to the FTP server with FileZilla and curl I knew that the issue was somewhere in PHP’s ftp implementation and not with the FTP server. But I still had no idea what the problem was.

I turned my attention to the FTP logs in FileZilla and noticed:

Command:  PASV
Response: 227 Entering Passive Mode (10,1,2,3,79,212)
Status:   Server sent passive reply with unroutable address. Using server address instead.

The ‘Status’ line clearly hinted to us that FileZilla knew that something was wrong – the FTP server has sent back a reply containing a private RFC1918 IP address (10.X.X.X 192.168.X.X etc) which is not routable over the internet. To understand what is happening here, it is helpful to know about Active FTP vs Passive FTP. The important thing to know is that when using FTP in passive mode, the FTP client sends a ‘PASV’ command to the server. When the server receives a ‘PASV’ command it opens up a random network port for the client to send further data to. The FTP server returns to the client an IP address and port that the client should use for further communications. The problem is, that when connected behind a NAT devices, the FTP server only knows about it’s local RFC1918 IP address (10.X.X.X, 192.168.X.X 172.16.X.X etc) and not it’s public internet IP address – therefore the server responds with the only IP address that it knows about, the RFC1918 IP address. This is problematic because RFC1918 IP addresses are not routable over the internet so any FTP client using the unroutable IP address returned by the FTP server will be unable to send further data packets to the server.

So, I was pretty confident that the problems this customer was seeing were due to PHP attempting to communicate with the FTP server on the private 10.1.2.3 RFC1918 IP address. To test that theory, I fired up tcpdump to take a look at packets going through the network interface and sure enough I found the server attempting twice to send packets to 10.1.2.3:

04:49:42.134381 IP X.X.X.X.36202 > 10.1.2.3.20426: S 1619271447:1619271447(0) win 5840 <mss 1460,sackOK,timestamp 3080816617 0,nop,wscale 7>
04:49:45.133886 IP X.X.X.X.36202 > 10.1.2.3.20426: S 1619271447:1619271447(0) win 5840 <mss 1460,sackOK,timestamp 3080819617 0,nop,wscale 7>

And yes, the time stamp says 4:49AM – I have much more fun debugging than I do sleeping :)

So now we understand why the problem was happening, but how do we solve it? Happily FileZilla told us exactly what it did to solve the problem – ‘Using server address instead’. So FileZilla was basically ignoring the IP address returned by the FTP server in response to the PASV command.

I scoured the PHP ftp extension documentation and source code in a fruitless search for any hints of a setting that would instruction PHP to ignore the IP address returned by the FTP server. Going through the C code that makes up the FTP extension code in PHP, I realized that it would be relatively simple to modify the code and add an option that causes PHP to ignore the unroutable IP addresses returned by an FTP server.

I’ve created a patch for PHP’s FTP extension that adds the USEPASVADDRESS option. When this option is set to TRUE (which is the default setting), PHP will behave as it does now – continuing to use the IP address returned by the FTP server in response to the PASV command.  When this option is set to FALSE, PHP will ignore the IP address returned by the FTP server in response to the PASV command and instead use the IP address that was supplied in the ftp_connect() (or ftp_ssl_connect() call)

This option can be set and retrieved using the ftp_set_option() and ftp_get_options()  functions:

ftp_set_option($ftpconn, USEPASVADDRESS, true);
echo "USEPASVADDRESS Value: " . ftp_get_option($ftpconn, USEPASVADDRESS) ? '1' : '0';
ftp_pasv($ftpconn, true);

The option should be set before calling ftp_pasv($ftpconn, true);

You can download the patch here – it was written for and tested on php 5.3.8 however it also works against PHP 5.2

To apply the patch:

cd /usr/src/php-5.3.8
patch -p0 < ftp_usepasvaddress.patch

Bug #55651 has been submitted to the PHP developers to hopefully include in future releases.