#1473 Incomplete FTPS transfers through HTTP proxy

[Marco De Vitis]

When doing a FTPS connection through a HTTP proxy (Blue Coat ProxySG) and downloading a file using a clear data channel through the same proxy, the downloaded file is truncated.
As far as I can see it looks like curl is considering the "Connection established" header from the proxy as part of the file, and so closes the file transfer some bytes too early.

This happens with curl 7.40.0 on RedHat 6.3:

curl 7.40.0 (x86_64-redhat-linux-gnu) libcurl/7.40.0 OpenSSL/1.0.1e zlib/1.2.3 c-ares/1.10.0 libidn/1.18 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets Metalink

This is the command line I use:

curl --trace-ascii - -k --ftp-ssl-control --disable-epsv --proxy <proxyIP>:3128 --proxytunnel ftp://<ftpUser>:<ftpPassword>@<serverIP>//testpub.xml

The same operation works fine with older curl versions. It has been tested to work with versions 7.19.7, 7.25.0, 7.26.0 and 7.27.0, while it does NOT work with versions 7.29, 7.31, 7.38, 7.39 and 7.40, although I can't confirm the behaviour was always exactly the same (for example, I recall that with 7.29 the transfer just timed out).

I attach a zip file containing a sample XML file I was downloading from the server, and the full --trace-ascii logs from 7.19.7 and 7.40.0 containing also curl -V output and the used command line.
In the trace for version 7.40 you can see that the transfer is interrupted before the complete XML file is downloaded.

[Marco De Vitis]

Sorry, the 7.40 log inside the ZIP attachment contains a wrong command line. The correct command line used is the one in the ticket text and in the 7.19 log.

[Daniel Stenberg]

This is likely similar to the FTPS-over-proxy problems (and solutions) being discussed on the curl-library list.

[Daniel Stenberg]

Can you please try the current dev version and see if the problem remains or is fixed?

[Marco De Vitis]

Sorry, I'm new to curl development process... Where can I download the dev version from? Is there an RPM for RHEL6?

[Daniel Stenberg]

The dev version only exists in git (http://curl.haxx.se/dev/source.html) and in daily snapshots (http://curl.haxx.se/snapshots/)

[Marco De Vitis]

Unfortunately I'm a bit limited in what I can do on the systems behind the HTTP proxy, so building/installing from source is complicated. Testing would be much easier with a RHEL 6 binary package.

[Daniel Stenberg]

I'm sure it would but I don't have any RHEL machines nor do I have any scripts to build such packages with, so...

[Marco De Vitis]

The problem is still present.
I downloaded http://curl.haxx.se/snapshots/curl-7.42.0-20150331.tar.gz, built it with default configure, make, make install on another RHEL 6.3 machine, then copied the /usr/local/* curl-related content on the target machine which is behind the proxy, and launched the same test command using /usr/local/bin/curl, but I get the exact same result, the downloaded file is not complete.

I hope the process I used is OK. I do not have complete freedom of what I can do on the target machine. Here is what I get with curl -V after installing the daily snapshot this way:

# /usr/local/bin/curl -V
curl 7.42.0-20150331 (x86_64-unknown-linux-gnu) libcurl/7.42.0-20150331 OpenSSL/1.0.0 zlib/1.2.3 libidn/1.18
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IDN IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets

[Daniel Stenberg]

Can you please re-resubmit this problem in the new bug tracker we've moving to? We're closing this down and I would like to get all "active" bugs moved over and yet I don't have any way to do it automatically.

You can still link to this bug report as it won't vanish, we'll just focus all activities going forward on the new one.

The new bug tracker: https://github.com/bagder/curl/issues