#197 TCPProxy recording fails at re-creating urls with tokens

[Anders Storsveen]

I have a case where a token is (correctly and automatically) discovered by the recorder via httpUtilities.valueFromLocationURI(). It also attemtps to create a later url which contains that token. However the token is never url encoded.

I confirmed this was the case by using urllib and the urlencode() method. This created a working url.

[Philip Aston]

Please post an example of an incorrectly script. Thanks.

[Anders Storsveen]

This script is auto-generated:

def page27(self):
"""GET verifyLoggedIn (request 2701)."""

# Expecting 302 'Found'
result = request2701.GET('/verifyLoggedIn')
self.token_SAMLRequest = \
  httpUtilities.valueFromLocationURI('SAMLRequest') # 'nZLbTsMwDIZfpcp9l7Tr2hFtkwYTYhKHaRtccINC...'
self.token_RelayState = \
  httpUtilities.valueFromLocationURI('RelayState') # 'https://staging.comoyo.no/verifyLoggedIn'

return result

def page32(self):
"""GET sso (requests 3201-3208)."""

# Expecting 307 'Temporary Redirect'
result = request3201.GET('/id/sso' +
  '?SAMLRequest=' +
  self.token_SAMLRequest +
  '&RelayState=' +
  self.token_RelayState)
self.token_FlowState = \
  httpUtilities.valueFromLocationURI('FlowState') # 'cmVxdWVzdElkPV83NGMyMDYwNDkwOGE1NzQ3Y2Q5...'

Here is how I had to fix it:

def page32(self):
"""GET sso (requests 3201-3208)."""

# DEFINES FLOWSTATE HERE!
print "Using Samlreq: " , self.token_SAMLRequest , " and relaystate: ", self.token_RelayState

samlReqEnc = urllib.urlencode(self.token_SAMLRequest)

# Expecting 307 'Temporary Redirect'
result = request32011.GET('/id/sso' +
  '?SAMLRequest=' +
  samlReqEnc +
  '&RelayState=' +
  self.token_RelayState)
self.token_FlowState = \
  httpUtilities.valueFromLocationURI('FlowState') # 'cmVxdWVzdElkPV83NGMyMDYwNDkwOGE1NzQ3Y2Q5...'

[Anders Storsveen]

Was the code ok?

[Philip Aston]

It certianly helps. I'll put this on my queue, thanks.

[Philip Aston]

Analysis: The recording currently unencodes tokens. It relies on the HTTPRequest method to re-encode URLs as required

Unfortunately, the generated script passes query string and path tokens by concatenating up a single the path parameter. HTTPRequest does no further processing on this parameter.

[Philip Aston]

I'm trying to decide the right way to fix this.

1. We could pass the query string parameters as NVPairs.This would require handling these tokens differently from path parameters, and a reasonable amount of change to the XSL scripts. We'd still need to fix the urlencoding of path parameters. IIRC, there's also some issue with method overloading and making sure Jython binds to the right GET(..)/POST(..)/... method. On the other hand, this would result in a reasonable script.

2. We could URL encode the whole path argument. This would look ugly in the script.

3. We could URL encode each query string value in the path. This would look ugly in the script, and potentially still be broken because the query string keys need encoding too.

4. We could capture the token string keys and values in encoded form. This would solve another lurking bug: values captured from response body <input> tokens are not urldecoded. We could still use the urldecoded key to generate a reasonable token ID.

Currently, I'm favouring 4.

[Anders Storsveen]

I'm liking fixes that make the script look good, however I don't necessarily like to much magic if it could introduce unexpected errors for the users later. I don't however have enough knowledge to be very helpful in this. Keep up the good work though!