Re: [OpenSTA-devel] https problem related to %7B

[Daniel S. <da...@Op...>]

Vikram Bartakke wrote:
> > I was trying to use openSTA 1.4.3.20.  I came across a problem
> > that whenever I try to record and playback https traffic, I get
> > page not found error.  When I read the documentation, I realised
> > that the OpenSTA modeller sends a special request to Gateway to
> > handle all HTTPS requests.

I'd probably let you get away with this statement on the users list;
but this is the developer list and I feel it's more important we talk
about things in more exact and correct terms here and not try to
"dumb things down" to a users level.  You should understand the
toolset properly before you try and fix or enhance it ;)

The Modeler DOES NOT send any sort of special request to the Gateway,
the Modeler doesn't care or know whether SSL is being recorded or not.

> > It does so by http://{ and  "{" corresponds to the value "%7b";

The way it 'works' is that the Gateway translates all occurences of
'https://' in the HTTP traffic from Web server to browser into
'http://{'.  In the browser to server direction the Gateway takes
any requests for 'http://{' URLs and establishes a HTTPS connection
as if it was the browser, plus some reverse translation of URLs is
in traffic is done.  There's also some trickery that has to be done
with the hostname, for cookies etc.,  but I sure you get the idea.

This hack/trick worked OK because the URL never changed length - but
it has still a dodgy mechanism and is prone to failures but the only
other options for recording HTTPS (a secure protocol) are in the
browser or server code and outside the encryption.

Then some security fix in IE6 decided that the { in a URL had to be
url encoded (to %7B) and this broke the recording mechanism for any
IE6 with that security fix applied.  I fixed this problem in the
OpenSTA 1.4.3 release by allowing the gateway to spot the URL encoded
version of the '{' as well.  Although it wasn't that simple as the
URL and hostname now changed length ...

> > however in IE 6, that value "%7b" gets somehow translated as "%7B"

It's more probable that 'fixed' IE6 URL encodes to %7B always - the
code in the Gateway uses the lowercased version (for comparisons
and replacement) ... whatever, I don't think there is any "somehow
translated" going on.

> > which is not recognizable from gateway.

This is simply not true - although it is what everyone who has
spotted this problem has theorized...  Since the first time this
problem has been reported I have checked the code and tested the
IE6 versions I have and this is just NOT what is happening.  The
code does a case independent comparison and I tested URLs that
were typed with the %7B and %7b and the Gateway spotted them both
correctly.   Everytime the problem has been reported I have replied
with this information and asked for more information to help track
down the reason for this problem - I have yet to reproduce it.

Jerome Delamarche wrote:
> As far as I know, a lot of us are using MS-IE 6 in HTTPS mode
> without any problem.

And it looks like Jerome can't reproduce it either. ;)

It has also been reported that patching the code to change the
references to %7B (instead of %7b) fixes the problem - I am quite
willing to believe there is *a* problem and that making this change
fixes *the* problem... but the actual bug *isn't* what a few people
have decided it is!

Given the reported and known facts, here are my thoughts on what
could be causing the (roughly) described symptom:
 - Specific C library (DLL) on problem machines can't do case
   independent comparisons - this is highly unlikely.
 - The code I wrote to cope does not cope with specific Web sites
   (or their coding technique) and 'somehow' breaks.  My guess this
   would be specifically to do with use of Cookies and/or Javascript
   on a HTTPS site.  If this was the case then you would still be
   able to record many HTTPS Web sites but specific ones would be a
   problem - not what people have reported when they get the problem.
 - A very specific verion of IE6 exists (one that Jerome, myself and
   others don't have) that causes the problem to occur because of a
   mismatch between %7B and %7b in HTTP headers and request.  This
   must be the case if *every* HTTPS Web site cannot be recorded on
   the problem machines.

> > After reading a further more about this problem on mailing
> > archives, I found that gwhttp.hpp is the file where such
> > declaration is done and it needed to be changed to "%7B" in order
> > to make it work in IE 6.

> You're right about the part of code that translates the URL.
> But the code make case insensitive comparison.
>
> I think your problem lies somewhere else.

I can confirm this, and my reasoning above should show the problem is
elsewhere.  Even if changing case of the code constant causes the
problem to go away - IHMO this is not really a fix because the
problem has never really been understood and diagnosed properly, the
patch may just be masking a problem which could be fixed properly if
only it was only understood fully.

As you obviously have a machine that is exhibiting the problem and
you have taken the time to get involved in building OpenSTA then I
urge you to investigate this problem properly and scientifically -
if you can give us clues as to what is really happening (or even
what is not) it would be most appreciated.

Whilst investigating this problem you should make sure you are
aware of, and understand, the other problems that could cause the
symptom to are seeing:
  http://portal.opensta.org/faq.php?topic=RecordingHttps
Specifically the CSS related issue: bug#1389081

> > However in order to build the OpenSTA completely I need
> > "CODEMAX 2.1.0.24" which I am not able to find anywhere.  I even
> > tried to search for http://groups.yahoo.com/group/codemax/ but
> > could not locate the yahoo group.

The group has had its membership closed for a long time and it was
never publically viewable.  The author of Codemax was the owner of
the group and when he decided to make a version of Codemax commercial
he always said that the open source version and the facilities that
he maintained might be removed at any point - I guess it is now
officially dead.  The licensing of Codemax always disallowed anyone
other than the author from distributing the source - so while it was
available only through the group it was almost dead, now the group
has been deleted it is completely dead.  I, obvioulsy have a copy of
the required package, and if you work out what is causing the HTTPs
failure you are seeing then I'll tell you how to find a copy ;)

It's a good job Jerome has started work on the Scintilla port of the
Modeler :)

Cheers
/dan
-- 
Daniel Sutcliffe <Da...@Op...>
  OpenSTA part-time caretaker - http://OpenSTA.org/