[OpenSTA-devel] [ opensta-Bugs-897856 ] Recording HTTPS through Gateway fails in newer browsers

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Bugs item #897856, was opened at 2004-02-16 02:43
Message generated for change (Settings changed) made by dansut
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110857&aid=897856&group_id=10857

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: HTTP Capture
Group: Serious
>Status: Open
>Resolution: Accepted
Priority: 5
Private: No
Submitted By: Alain Ribault (ribault)
Assigned to: Daniel Sutcliffe (dansut)
>Summary: Recording HTTPS through Gateway fails in newer browsers

Initial Comment:
HTTPS recording :
With my browser (IE6, latest patches), when the opensta
receives an http page with https links, it transforms
the link like http://{ ..., which is correct by opensta
implementation. The problem is that IE escapes '{' by
'%7D' and the gateway does not understand http://%7D,
so the capture stops by 'page not found'.

Hisashi Iwashimizu wrote that it is after applying
MS04-004 
 security patch to IE6. 

I did some tests with Mozilla and it worked fine
(because no escape code). 

----------------------------------------------------------------------

>Comment By: Daniel Sutcliffe (dansut)
Date: 2007-07-03 16:14

Message:
Logged In: YES 
user_id=19748
Originator: NO

I am re-opening this bug with the possibility that it will never be closed
in the future!

Nearly all of the newer browsers (IE7, Firefox 2, Opera 9) fail recording
HTTPS for various reasons.  In IE it seems that the curly bracket gets
multiply URL encoded and Firefox/Opera refuse to even attempt to attempt a
HTTP connection to URLS whose hostname contain this 'invalid' character. 
Instead of attempting to apply bodge after bodge to keep this mechanism
working I am proposing that no more work is done on the Gateway mechanism
for recording HTTPS because it is really too much of a headache and any fix
is liable to be broken again (for valid reasons) by any updated browser
releases - instead I propose that any effort in this area instead be put
into other mechanisms for recording HTTPS traffic.

I am commiting to the CVS HEAD a fix that has been mentioned on the Users
list a few times (patching %7b to %7B) because it does help certain
situations where IE6 gets confused - but I will state again that this is
NOT because the gateway was doing a case dependent comparison but more
because IE was relying on the case to be in upper case.  This might break
things again in future versions but it helps the situation for a small
amount of users now.  This workaround will become generally available in
the OpenSTA 1.4.4 release.
gwhttp.dll: 1.4.4.3
OpenSTA: 1.4.4.5

More info on this whole area in the FAQ here:
http://portal.opensta.org/faq.php?topic=RecordingHttps

If you want to discuss this whole area then please do so by posting to the
OpenSTA Users Mailing List:
http://portal.opensta.org/faq.php?topic=UserMailingList - and not by
posting more info to this bug report.

----------------------------------------------------------------------

Comment By: Daniel Sutcliffe (dansut)
Date: 2005-12-18 13:54

Message:
Logged In: YES 
user_id=19748

This problem had NOTHING to do with script playback - it was
entirely a recording issue.  This fact does not matter
anyway, as the problem is fixed in the current release of
OpenSTA (1.4.3).

Also, please do not use this bug database as a type of
support forum - this just clutters up bug reports and makes
them more confusing.

----------------------------------------------------------------------

Comment By: kikibug (kikibug)
Date: 2005-12-07 04:42

Message:
Logged In: YES 
user_id=1314392

Should recording the script OK with '%7D' instead of '{'
affect playback? I have a problem that I do not know for
sure to be a bug, I have entered it in the mailing lists - I
cannot play back HTTPS script (which was recorded OK and
shows https:// in the requests) when going from the second
to the third HTTPS page in a row. So I am wondering if this
occurence with IE6 may affect playback in any way?

----------------------------------------------------------------------

Comment By: Lance (stampy)
Date: 2005-10-19 14:16

Message:
Logged In: YES 
user_id=1137295

hm, i too also get the %7B when using IE6. Im gathering from 
your previous response that its a problem in IE. Any other 
ideas?

im using osta 1.4.3.20

----------------------------------------------------------------------

Comment By: Daniel Sutcliffe (dansut)
Date: 2005-06-15 12:18

Message:
Logged In: YES 
user_id=19748

mvemers wrote:
> This problem seems to have re-occurred - IE is showing %7B
> in the location bar again:

This actually doesn't mean that this bug has re-occurred,
the fix to this problem was to allow the gateway to
understand the urlencoded form of the { character as well. 
It is only a problem if you cannot record HTTPS because of
this...

----------------------------------------------------------------------

Comment By: Mark Vevers (mvevers)
Date: 2005-06-15 11:26

Message:
Logged In: YES 
user_id=13500

This problem seems to have re-occurred - IE is showing %7B
in the location bar again:

gwhttp.dll: 1.4.3.6 
IE Version:  6.0.2900.2180.xpsp_sp2_gdr.050301-1519

Any suggestions?

Thanks

----------------------------------------------------------------------

Comment By: Daniel Sutcliffe (dansut)
Date: 2005-01-24 13:37

Message:
Logged In: YES 
user_id=19748

And addendum to this fix has been merged into the CVS HEAD.
The case were a Location: header was specified from the root
during a HTTPS redirect was not dealt with.  Thanks to Thierry
for spotting this.  This will be included in the OpenSTA 1.4.3
release.

gwhttp.dll: 1.4.3.5
OpenSTA: 1.4.3.18

----------------------------------------------------------------------

Comment By: Daniel Sutcliffe (dansut)
Date: 2004-12-20 17:29

Message:
Logged In: YES 
user_id=19748

The most recently mentioned fix has been merged into the CVS
HEAD.  It will become generally available in the OpenSTA 1.4.3
release.

gwhttp.dll: 1.4.3.4
OpenSTA: 1.4.3.13

----------------------------------------------------------------------

Comment By: Daniel Sutcliffe (dansut)
Date: 2004-12-20 14:33

Message:
Logged In: YES 
user_id=19748

I now have a "proper" fix for this that treats the URL encoded
version of { (which is %7b not %7d as Alain initially reported)
as equivilant to the {.

The fix had to be extended with a little fudge because it
uncovered and issue with IE and redirects.  This is best shown
by an example:  If I go to record https://osta.lan/mail/ I'll 
get http://{osta.lan/mail/ which the new "fixed" IE will rewrite
to http://%7Bosta.lan/mail/ - this will work fine, but if the
Web server then returns with a HTTP 302 and "Location:
src/login.php" then IE will get confused and try to redirect to
http://%7Bosta.lanan/mail/src/login.php - obviously the extra
2 chars and the domain name changing length confuse IE.  It
is unlikely that anyone in the "real world" will find this IE
buglet so I worked around it by having the gateway rewrite
the "Location:" header so that it is always fully specified for
SSL recordings (the recording .all gets the original value).

I was convinced this was something the gateway was doing
wrong for a long while but hours spent with a sniffer and
tests with other browsers showed that it was IE.

The fix also "enhances" the console and trace output from
the gateway.

This gateway code is ugly...

----------------------------------------------------------------------

Comment By: John L (johnleij)
Date: 2004-09-21 13:34

Message:
Logged In: YES 
user_id=999135

Hi all,

As I wasn't able to build OpenSTA due to lack of access to 
required third party tools, I binary patched the product. I'm 
not sure whether the license gives me the rights to distribute 
the patch, but I gather that it is for everybodys' best if I at 
least inform you of what I did.

I binary patched gwhttp.dll to make it work with SSL even 
after security patch MS04-004 on Internet Explorer 6.

The idea is to replace the ordinary "http://{" sequence 
with "http://-" so that IE doesn't destroy the URL when it is 
UrlEncoded. Note that this quick and dirty fix will make 
OpenSTA stop working with sites that begin with '-' (not very 
common though).

In gwhttp.dll I patched:
- Changed "http://{" to "http://-" (address ~00029380h)
- Changed "{" to "-" (address ~00029c40h)

I could write a long disclaimer here to inform you that I can't 
guarantee that the patch works, but I think you already know 
that. The patch surely seems to work for me.

----------------------------------------------------------------------

Comment By: jmrwnd (jmrwnd)
Date: 2004-09-21 09:23

Message:
Logged In: YES 
user_id=1125756

Has johnleij's fix been implemented anywhere, and is there 
any way to get this alternate version of the code?  I am 
unable to alter and build the source code myself and need a 
workaround for this problem ASAP.  I absolutely must test 
with IE6 or Netscape 6, but both escape the '{' character.

If all else fails, is there another similar tool that could record 
https traffic with these browsers that could run off of a 
laptop?  Any help is greatly appreciated.  Thanks.

----------------------------------------------------------------------

Comment By: John L (johnleij)
Date: 2004-03-16 09:28

Message:
Logged In: YES 
user_id=999135

A possible quick-fix could be to change the implementation of 
OpenSTA so that instead of using "http://{" to identify SSL, 
we could use "http://-". The latter isn't common on the 
internet and is not tampered with by MSIE.

A search through the source code gave me the idea to chage 
the value of constants CYRANO_SSL_URL and 
CYRANO_SSL_STR in HttpGateway\gwhttp\context.hpp to 
reflect this.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110857&aid=897856&group_id=10857