[privoxy-users] [ ijbswa-Support Requests-1573878 ] Privoxy and HTTPS - Requests without ":443" die

[SourceForge.net <no...@so...>]

Support Requests item #1573878, was opened at 2006-10-09 10:03
Message generated for change (Comment added) made by sf-robot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=211118&aid=1573878&group_id=11118

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: other
Group: 3.0.x
>Status: Closed
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Fabian Keil (fabiankeil)
Summary: Privoxy  and HTTPS  - Requests without ":443" die

Initial Comment:
You can email me at pri...@sp...

I am running Privoxy 3.1.5 on a CentOs 4.3 machine.  I
found a problem in the way that v3.1.2 handled chunked
responses, so I was forced to upgrade to 3.1.5-Beta.  I
am now seeing a problem when I try to run some
LWP::UserAgent scripts through Privoxy, but I don't
know if the fault lies with Privoxy or LWP::UserAgent. 

When I try to access a site over HTTPS using a web
browser (Firefox running on Linux or Windows, IE on
Windows), I can see that Privoxy sees the Request as
follows...

Oct 08 23:08:13 Privoxy(b7536bb0) Request:
someSecureSite.com:443/
Oct 08 23:08:13 Privoxy(b7536bb0) Writing: 
Oct 08 23:08:13 Privoxy(b7536bb0) Writing: HTTP/1.0 200
Connection established
Proxy-Agent: Privoxy/3.0.5


And after this line in the logfile, encrypted traffic
shows up, the page is fetched, life is good.

When I try to write a perl script to automate this
transaction, the request appears as follows...

Oct 08 23:10:01 Privoxy(b7536bb0) Request:
someSecureSite.com/
Oct 09 09:53:15 Privoxy(b7f37bb0) Writing: 
Oct 09 09:53:24 Privoxy(b7f37bb0) Writing: GET / HTTP/1.1
TE: deflate,gzip;q=0.3
Host: someSecureSite.com
User-Agent: X11; U; Linux i686; en-US; rv:1.8.0.7)
Gecko/20060915 CentOS/1.5.0.7-0.1.el4.centos4
Firefox/1.5.0.7 pango-text
Connection: close


The :443 is not appended to the end of the Request by
Net::HTTP::Methods within the format_request method. 
This causes Privoxy to fail with 500 errors.  

Using a browser, everything seems to work for ports 80
and 443 requests.  Using LWP::UserAgent HTTP works but
HTTPS seems completely broken.  Can someone offer some
suggestions?  Is this a Privoxy bug or a
Net::HTTP::Methods bug?


----------------------------------------------------------------------

>Comment By: SourceForge Robot (sf-robot)
Date: 2006-12-25 19:20

Message:
Logged In: YES 
user_id=1312539
Originator: NO

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Fabian Keil (fabiankeil)
Date: 2006-12-11 08:19

Message:
Logged In: YES 
user_id=875547
Originator: NO

CONNECT is described in RFC 2817 "Upgrading to TLS Within HTTP/1.1"
and it is the standard HTTP method to get non-HTTP
content through HTTP proxies.

Basically the client asks the proxy to open a
connection to a specific host and get out of the
way when it's done.

The proxy doesn't have to care what the user is
transmitting, and in the SSL case the content
passes the proxy encrypted anyway.

The proxy knows the target host and the port,
and that's it. I doesn't even see the URLs the
user is requesting. 

Just because a program supports direct SSL connection,
that doesn't mean it supports SSL connections through
proxies as well. Of course most do and I find it rather
strange that libwww doesn't.

There is a short note about it in perldoc Crypt::SSLeay:

"At the time of this writing, libwww v5.6 seems
to proxy https requests fine with an Apache mod_proxy
server.  It sends a line like:

         GET https://www.nodeworks.com HTTP/1.1

to the proxy server, which is not the CONNECT request
that some proxies would expect, so this may not work
with other proxy servers than mod_proxy.  The CONNECT
method is used by Crypt::SSLeay's internal proxy support."

I tried WWW::Mechanize yesterday and it not only uses
GET instead of CONNECT, but it also send the complete
client headers unencrypted. I don't know what it expects
as response (as usual for Perl modules the documentation
has room for improvements), but I assume that it wants the
proxy to silently do all the encrypting work, decrypt the
server's answer and send it in clear to WWW:Mechanize.

Privoxy currently can't do that and I wouldn't be
surprised if Crypt::SSLeay's perldoc was right and
Apache's mod_proxy is the only one that does.


I think the best solution for your problem would
be to keep Privoxy out of the loop entirely,
to configure WWW::Mechanize for direct connection and
have these connections intercepted and redirected into Tor:
http://wiki.noreply.org/noreply/TheOnionRouter/TransparentProxy


While you could put stunnel behind Privoxy to get
it working with WWW::Mechanize's pseudo SSL requests,
that's a rather painful setup and you have to change
Privoxy's and stunnel's config files for every host
you want to connect to. I also think you would need
additional patches to convince stunnel to make the
connections through Tor (unless you run Tor as
intercepting proxy in which case you wouldn't need
Privoxy and stunnel in the first place).

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2006-12-07 22:14

Message:
Logged In: NO 

Hello Fabian,

Thank you very much for your response.  I appologzie for not checking for
a response on this web site, I was under the (mistaken) impression that
any response would be sent to the email address listed in the detailed
description of the post. 

I share your puzzlement over the versions of Privoxy that I quoted.  I am
using 3.0.5 Beta, not 3.1.x.  I appologize for the confusion.

I am intrigued by your comment about CONNECT.  Forgive my ignorace, based
on the W3 HTTP 1.1 spec that I referenced, this method has not been
implemented.  
http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

I do not know the status of CONNECT in LWP.  I have read the docs on this
package and can not recall seeing any reference to CONNECT.  I can rule
out the possibility that LWP does not handle SSL.   I have the following
trivial perl script....

<test.pl>

#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use HTTP::Cookies;

my $bot = WWW::Mechanize->new;
# $bot->proxy(['http', 'https'], 'http://localhost:8118/');
my $url = "https://login.yahoo.com";
my $response = $bot->get($url);

</test.pl>


(WWW::Mechanize extends LWP and should be for the purposes of this script
is same).

Bypassing Privoxy by leaving the line that specifies proxy commented out,
I can retrieve Yahoo's SSL login page without any problem.     The
$response object show a 200 result and the html is indeed Yahoo's login
page.  Repeating the above experiment with the proxy enabled, I get a 503
response and the "503 - Connect failed (Privoxy@localhost)" HTML page from
Privoxy.  

My motivation for using Privoxy is so that I can use Tor to anonymozie the
scripts.  I have been successful in configuring Privoxy and Tor to work
together and I can direct browser traffic through them which effectively
obscues the IP address.  As I've tried to implement the LWP scripts, I've
removed Tor by commenting out the "forward-socks4a" line of the config
file so that I can simplify the problem.  

I have not yet tried the stunnel that you have suggested.  I'm still open
to trying this, but I just don't think that the SSL is the problem since I
could retrieve secure web pages from a script as long as I don't try to use
Privoxy.  But I could be convinced otherwise and welcome your suggestions.








----------------------------------------------------------------------

Comment By: SourceForge Robot (sf-robot)
Date: 2006-10-23 19:20

Message:
Logged In: YES 
user_id=1312539

This Tracker item was closed automatically by the system. It was
previously set to a Pending status, and the original submitter
did not respond within 14 days (the time period specified by
the administrator of this Tracker).

----------------------------------------------------------------------

Comment By: Fabian Keil (fabiankeil)
Date: 2006-10-09 10:56

Message:
Logged In: YES 
user_id=875547

I'm somewhat puzzled by your Privoxy versions.
The latest stable version is 3.0.3, the latest
beta is 3.0.5.

The Privoxy Team didn't release 3.1.2 or 3.1.5,
so maybe you are using a third party package
done by a more than clueless packager?

At least from your log it looks as if you were
running something based on Privoxy 3.0.5.

To answer your question:

Are you sure that your LWP::UserAgent version
supports SSL and CONNECT requests? From your log
it looks like Privoxy is seeing the actual GET
request, which means the connections isn't encrypted
and CONNECT isn't used. You can enable header debugging
(debug 8) to see what's going on here.

Either LWP::UserAgent doesn't support SSL and
CONNECT requests, or you aren't using it correctly.

The easiest way (I know of) to create encrypted requests
with Privoxy and applications that don't support SSL and
CONNECT requests is to chain it with stunnel.

If you chain it like:

client -> Privoxy -> stunnel -> web

Your application only needs to speak HTTP,
and stunnel does the SSL stuff for you.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=211118&aid=1573878&group_id=11118