proxy/firewall authentication

[Charles Lynn Walls]

I just ready Paul's article on Finance::Quote in TPJ #19 and noted the comment that the proxy server setup may be accomplished with:

  $quoter->user_agent->proxy('http', $MY_PROXY);

However, I am behind a firewall that requires authentication as well.  For my own direct internet connections, I code a "proxy" line similar to what is shown above, but IN ADDITION I must code the following to get authenticated to the firewall:
 
   my $req  = new HTTP::Request GET => $stock_url . join('+',keys(%QUOT));
   $req->proxy_authorization_basic(split /\:/, $MY_LOGIN);

where $MY_LOGIN is a userid:password pair.

Would there be a similar specification to set up the proxy userid:password authentication in Finance::Quote? If not, might that capability be added to the development task list?

Thank you,
Lynn Walls

[Paul Fenwick]

G'day Lynn,

    Wow, you've read the article already?  I'm still waiting for mine. :)

    You've asked a very good question, one that I don't immediately know the answer to.  If LWP::UserAgent has a way to set proxy-authentication on the UserAgent object, then this should be an easy task -- however I can't find an immediately obvious way of doing this.

    Looking through the LWP::UserAgent source-code reveals that the appropriate headers need to be set on the request object we're passing in (as you've done in your example).  However, Finance::Quote doesn't allow you to do any sort of special configuration for requests except for what's provided by LWP::UserAgent.

    There are probably ways around this, like hacking the HTTP::Request package directly, but this certainly isn't recommended.

    I'll add proxy authentication (and other arbitary HTTP headers support) to the development task-list.  If you'd like to help out in the coding, please let me know.

    Cheers,

        Paul

[Charles Lynn Walls]

Paul,

Thanks for you very quick response.

After a brief look at the "Finance::Quote" code, the "LWP::UserAgent"
code and "HTTP::Request" code, I have concluded that the proxy-server
authentication problem should most properly be fixed in LWP::UserAgent
and HTTP::Request, as you have suggested. But I am not technically ready
to tackle that one.  It does seem to me that BOTH the proxy server URL
and any necessary authentication userid/password should be captured at
the UserAgent object level -- not partially at the UA level and
partially at the Request level.  Both sets of data can be thought of as a single spec object
that is needed to get through the proxy server.  It seems to me that Request should
inherit from UA not only the proxy server URL specification (as it
already does), but eveything that goes with it: like the associated
authenticating userid and password, when required.  Ergo: design
oversight! (?)

BTW: I do indeed see the logic behind having the proxy URL specified at the UA level
and the userid/password specified at the Request level.  For example, one might have a
general purpose server using LWP::UserAgent and HTTP::Reuqest to manage internet
access for a NUMBER of different users.  Such server  will always have to go through the
same proxy server, and hence the same proxy server URL.  But it may be serving a whole
bunch of different users with different levels of userid/password controlled access on its
back end.   Thus each "Request" may have to have its own user dependent proxy server
authentication.  So, I am only suggesting that LWP::UserAgent and HTTP::Request be given the ability to have
an OPTIONAL, inheritable userid/password object as a default.  That way users of UA and
Request on single user platforms (like Windows PCs) would find life happier!

But, given all that, and given my unwillingness at this point to tackle the problem at that
level, what can be done about Finance::Quote at the "local" level?

It looks to me like it would not be a difficult thing to capture BOTH
the proxy server URL as well as the authenticating userid/password pair
as part of the "Quote" object (see my suggestion below), and then pass them along to
UA and Request where needed.  That would be the easiest thing for the
users of Quote to deal with. ...Yes, I know they could continue to feed
to UA the proxy URL directly as they can now. But why make the user have
to deal with specifying one single concept -- getting through his proxy
server -- to two different object levels -- Quote and UA? Let's just
combine the whole get-through-the-proxy-server garbage in one
place!...i.e., "Quote" (since as I noted previously, UA and Request are
technically incapable at present, and we [read: "I"] don't want to mess
with them).

You could just create a new Quote method to capture this data...maybe
along the lines of "set_currency", since the proxy server settings are
global to everything the user is doing:

  $q->set_proxy($proxy_url, $proxy_userid, $proxy_password);

Then Quote would do the

  $ua->proxy('http',$proxy_url);

thing, and the

  $req->proxy_authorization_basic($proxy_userid, proxy_password);

thing at all the various appropriate places.

If someone subsequently gets around to fixing LWP::UserAgent and
HTTP::Request along the lines suggested, they will probably do it in
such a way that the userid/password specification made directly to
Request will not break. So the above changes to Quote should NOT have to
be remediated.

If this sounds reasonable to you, do you want me to make a stab at it?
If so, do you want to point me to a development instance of
Finance::Quote that I can hack on?

Sincerely,
Lynn Walls

[Paul Fenwick]

G'day Lynn,

    I agree entirely that the users of F::Q shouldn't have to worry about proxies being defined on the user-agent but proxy-auth being defined on the request.  I'd actually like to take this one step further and say that writers of F::Q modules shouldn't have to worry about it either.

    I actually sat down and had a good think about this problem last night, and I think it would be very convienient to define an arbitary set of user-defined headers that should be included with HTTP requests.  This covers us for instances such as proxy-authentication now, but also covers us for things in the future as well.

    The best way I could think of doing this was by creating our own special F::Q::UserAgent module, which either inherits from or otherwise forms friends with the standard LWP::UserAgent module.  Since all the F::Q code calls $q->user_agent rather than creating its own user-agent, we only need to modify one place to put our new F::Q::UserAgent to work.  This can all be transparent to the end-user, but means that F::Q module-writers don't need to play games with request code.

    The special UserAgent will keep track of a special "template" request, which will be used in generating all new requests.  Placing our proxy headers in the template means they will automatically be added to all new requests.

    All this can be wrapped up nicely so that programmers can use
a simple interface like you've already suggested to handle negotiations through
the proxy.  I'd also like an environment interface so that users can specify proxyauth information (possibly in an already-digested form) similar to how proxies can be specified using the http_proxy environment variable.

    If you do want to tinker with F::Q for any reason at all, you can always grab the most recent development copy from the CVS repository (follow the CVS link at the top-right of this page).  If you don't have access to CVS, there's also a nightly tarball which can be downloaded from http://cvs.sourceforge.net/cvstarballs/finance-quote-cvsroot.tar.gz , but CVS is much more useful.

    If/When I get the new F::Q::UserAgent working I'll commit it to CVS and ask for your opinions.  I'm interested in seeing any code you have/will develop, as I may not be writing the optimal solution.

    Cheers,

        Paul

[Paul Fenwick]

Good news!

I managed to get my modified LWP::UserAgent running.  I discovered that most of my problems were caused by me having my http_proxy environment variable set incorrectly, and not actually with my code.  Ooops.  :)

The F::Q::UserAgent allows the user to define a template HTTP::Headers object, which will be automatically used in the generation of new requests.

This means you can now do something like this:

use Finance::Quote;
my $q = Finance::Quote->new;
$q->user_agent->default_headers->proxy_authorization_basic("foo","bar");

$q->user_agent->default_headers is a HTTP::Headers object, and hence accepts all HTTP::Headers method calls.

Mind you, $q->user_agent->default_headers->do_stuff(@args) is a fairly unwieldly thing to type.  We might put in a few short cuts so people can type $q->http_headers->proxy_authorization_basic (or even something shorter).

The ironic thing is that having done all this, I discovered that there actually was a possibly "more correct" way of doing things that had been staring me in the face for a while.  The 'credentials' and 'get_basic_credentials' methods are used when challenged with a basic proxy or http authorisation request.  Rather than going to the trouble of modifying headers (which will cause a slight performance hit on every request), we could just over-ride the get_basic_credentials method to return the appropriate information when it's required.

This approach doesn't give us the chance to slap on arbitary headers to each request, but it does seem a little cleaner.

In any case, I've checked the code that I have done into CVS.  If you want to do a test, and don't mind putting your proxyauth username and password into an environment variable, then you can do this:

export http_proxy_auth_clear=username:password
perl Makefile.PL
make test

You should get a 100% success rate on all the tests in the development test suite.  (Obviously you need to replace the "username:password" string with your proxy username and password.)

These methods should still be considered flexible right now.  Behaviour may change if we find a better way to do things.

Any questions or problems, then please let me know.

Cheers,

    Paul