Menu
▾
▴
cpp-netlib-devel
|
From: Dean M. B. <mik...@gm...> - 2008-08-15 00:17:12
|
Hi Guys, Does anybody know how I can get the HTTP Headers from the request that invokes the CGI script? Help would be most appreciated. TIA -- Dean Michael C. Berris Software Engineer, Friendster, Inc. |
|
From: Divye K. <div...@gm...> - 2008-08-21 17:11:08
|
Hi Dean, I hope this response isn't too late. > > Does anybody know how I can get the HTTP Headers from the request that > invokes the CGI script? > Try LiveHTTPHeaders. It's an add-on for Firefox that allows you to see the headers that are sent by the server in response to a particular request. https://addons.mozilla.org/en-US/firefox/addon/3829 You can make a GET request directly by accessing http://localhost:8000/cgi-bin/requestinfo.py?var=value and see the headers. To make a POST request, create a simple HTML form <html> <head> <title></title> </head> <body> <form action="cgi-bin/requestinfo.py?arg=argument" method="POST"> <input type="text" name="test" /> <input type="hidden" name="hiddenparam" value="hiddenvalue" /> <input type="submit" value="submit" /> </form> </body> </html> and save it in the test\server directory. Type in some text and press the submit button. View the headers using LiveHTTPHeaders. Divye -- An idealist is one who, on noticing that a rose smells better than a cabbage, concludes that it will also make better soup. H. L. Mencken (1880 - 1956) My official web site: http://people.iitr.ernet.in/shp/061305/ Webmaster: http://www.drkapoorsclinic.com Blog: http://divyekapoor.blogspot.com |
|
From: Dean M. B. <mik...@gm...> - 2008-08-22 07:14:00
|
Hi Divye, On Fri, Aug 22, 2008 at 1:11 AM, Divye Kapoor <div...@gm...> wrote: > Hi Dean, > I hope this response isn't too late. > >> >> Does anybody know how I can get the HTTP Headers from the request that >> invokes the CGI script? > > Try LiveHTTPHeaders. It's an add-on for Firefox that allows you to see the > headers that are sent by the server in response to a particular request. > https://addons.mozilla.org/en-US/firefox/addon/3829 > Right, but I'm more interested in something that can be made part of the unit tests so that I can verify whether the http::client we have has really sets the User-Agent header. -- Dean Michael C. Berris Software Engineer, Friendster, Inc. |
|
From: Divye K. <div...@gm...> - 2008-09-06 18:37:32
|
Hi Dean, On Fri, Aug 22, 2008 at 12:43 PM, Dean Michael Berris < mik...@gm...> wrote: > Right, but I'm more interested in something that can be made part of > the unit tests so that I can verify whether the http::client we have > has really sets the User-Agent header. > I've committed a change (changeset 85) that allows you to see the HTTP headers in the response from cgi-bin/requestinfo.py There are a couple of bugs in the library that put in non-existent headers in the response. However, it should be sufficient to test for setting of http headers. I'm thinking of refactoring the entire file to make parsing of the response easy. What would be the ideal response from the CGI script? CSV, XML or something else? Currently, the response is rather kludgy to parse. Comments would be helpful. Divye -- An idealist is one who, on noticing that a rose smells better than a cabbage, concludes that it will also make better soup. H. L. Mencken (1880 - 1956) My official web site: http://people.iitr.ernet.in/shp/061305/ Webmaster: http://www.drkapoorsclinic.com Blog: http://divyekapoor.blogspot.com |
|
From: Dean M. B. <mik...@gm...> - 2008-09-08 05:25:16
|
Hi Divye, On Sun, Sep 7, 2008 at 2:37 AM, Divye Kapoor <div...@gm...> wrote: > Hi Dean, > > On Fri, Aug 22, 2008 at 12:43 PM, Dean Michael Berris > <mik...@gm...> wrote: >> >> Right, but I'm more interested in something that can be made part of >> the unit tests so that I can verify whether the http::client we have >> has really sets the User-Agent header. > > I've committed a change (changeset 85) that allows you to see the HTTP > headers in the response from cgi-bin/requestinfo.py > There are a couple of bugs in the library that put in non-existent headers > in the response. However, it should be sufficient to test for setting of > http headers. > When you say "there are a couple of bugs in the library", which library are you referring to? And what exactly is the bug? > I'm thinking of refactoring the entire file to make parsing of the response > easy. > What would be the ideal response from the CGI script? CSV, XML or something > else? > Currently, the response is rather kludgy to parse. > Tab/space separated values work -- they can easily be parsed through std::ostringstream's -- as well as doing a '\n' delimited lines for pseudo-CSV. > Comments would be helpful. HTH (BTW, I'll take a look at your code to see if anything breaks in cpp-netlib) -- Dean Michael C. Berris Software Engineer, Friendster, Inc. |
|
From: Divye K. <div...@gm...> - 2008-09-08 12:58:48
|
Hi Dean, On Mon, Sep 8, 2008 at 10:55 AM, Dean Michael Berris <mik...@gm... > wrote: > Hi Divye, > > When you say "there are a couple of bugs in the library", which > library are you referring to? > The python CGI library. > > And what exactly is the bug? A simple get request sets the content-type header to be application/x-www-form-urlencoded. This is an issue with the Python CGI library. Other than this minor bug, I haven't found any major issues with the code and I don't think any of the tests break. > > > I'm thinking of refactoring the entire file to make parsing of the > response > > easy. > > What would be the ideal response from the CGI script? CSV, XML or > something > > else? > > Currently, the response is rather kludgy to parse. > > > > Tab/space separated values work -- they can easily be parsed through > std::ostringstream's -- as well as doing a '\n' delimited lines for > pseudo-CSV. > Ok. > > > Comments would be helpful. > > HTH > Sure did. > > (BTW, I'll take a look at your code to see if anything breaks in > cpp-netlib) > Sure. Divye -- An idealist is one who, on noticing that a rose smells better than a cabbage, concludes that it will also make better soup. H. L. Mencken (1880 - 1956) My official web site: http://people.iitr.ernet.in/shp/061305/ Webmaster: http://www.drkapoorsclinic.com Blog: http://divyekapoor.blogspot.com |
|
From: K. G. <kim...@gm...> - 2008-09-08 20:12:35
|
Dean, Divye, On Mon, Sep 8, 2008 at 14:58, Divye Kapoor <div...@gm...> wrote: >> >> Tab/space separated values work -- they can easily be parsed through >> std::ostringstream's -- as well as doing a '\n' delimited lines for >> pseudo-CSV. > > Ok. I'm currently playing with a HTTP header-style format to echo headers for the post tests. I have a new post_echo_headers.py, that emits the request headers on the canonical "name: value\n" form. It gets harder if you want to combine headers and other data on the same page, though... Maybe we can re-group once we find something that works for everybody? - Kim |
|
From: K. G. <kim...@gm...> - 2008-09-09 07:40:33
|
Hi guys, On Mon, Sep 8, 2008 at 22:12, Kim Gräsman <kim...@gm...> wrote: > > On Mon, Sep 8, 2008 at 14:58, Divye Kapoor <div...@gm...> wrote: >>> >>> Tab/space separated values work -- they can easily be parsed through >>> std::ostringstream's -- as well as doing a '\n' delimited lines for >>> pseudo-CSV. >> >> Ok. > > I'm currently playing with a HTTP header-style format to echo headers > for the post tests. > > I have a new post_echo_headers.py, that emits the request headers on > the canonical "name: value\n" form. It gets harder if you want to > combine headers and other data on the same page, though... I've looked into this more closely, and it appears that only selected headers make it through to the CGI scripts. So, as far as I can see there's no reliable way to dump all request headers to the response from a CGI script, via the Python server. The ones that make it through are listed in the run_cgi method in Python's CGIHTTPServer.py. Content-Length and Content-Type are among them, so I thought I had it working for a while :-/ Grr. - Kim |
|
From: Dean M. B. <mik...@gm...> - 2008-09-09 09:22:16
|
On Tue, Sep 9, 2008 at 3:40 PM, Kim Gräsman <kim...@gm...> wrote: > > On Mon, Sep 8, 2008 at 22:12, Kim Gräsman <kim...@gm...> wrote: >> >> On Mon, Sep 8, 2008 at 14:58, Divye Kapoor <div...@gm...> wrote: >>>> >>>> Tab/space separated values work -- they can easily be parsed through >>>> std::ostringstream's -- as well as doing a '\n' delimited lines for >>>> pseudo-CSV. >>> >>> Ok. >> >> I'm currently playing with a HTTP header-style format to echo headers >> for the post tests. >> >> I have a new post_echo_headers.py, that emits the request headers on >> the canonical "name: value\n" form. It gets harder if you want to >> combine headers and other data on the same page, though... > > I've looked into this more closely, and it appears that only selected > headers make it through to the CGI scripts. > > So, as far as I can see there's no reliable way to dump all request > headers to the response from a CGI script, via the Python server. The > ones that make it through are listed in the run_cgi method in Python's > CGIHTTPServer.py. Content-Length and Content-Type are among them, so I > thought I had it working for a while :-/ > Is there no way to define which headers (or if all headers) should be preserved? > Grr. That should be alright... We may find a better way to go about things if we write our own HTTPServer extension which does what we want instead of relying on the CGIHTTPServer that Python defines. I don't have enough Python kung fu to be able to pull it off yet though so if you find other ways, I'm all ears. ;-) Thanks for looking into this! :-D -- Dean Michael C. Berris Software Engineer, Friendster, Inc. |
|
From: K. G. <kim...@gm...> - 2008-09-09 11:29:45
|
Hi Dean, On Tue, Sep 9, 2008 at 11:22, Dean Michael Berris <mik...@gm...> wrote: >> So, as far as I can see there's no reliable way to dump all request >> headers to the response from a CGI script, via the Python server. The >> ones that make it through are listed in the run_cgi method in Python's >> CGIHTTPServer.py. Content-Length and Content-Type are among them, so I >> thought I had it working for a while :-/ > > Is there no way to define which headers (or if all headers) should be preserved? Not as far as I can tell... I think maybe this problem is part of the CGI concept as well; since it uses environment vars to pass on selected headers, there's a risk that custom headers overwrite env vars that the shell depends upon. Unless you jump through hoops to escape them, etc, or encode all headers into one long string, somehow. But then it's no longer CGI per se. > That should be alright... We may find a better way to go about things > if we write our own HTTPServer extension which does what we want > instead of relying on the CGIHTTPServer that Python defines. > > I don't have enough Python kung fu to be able to pull it off yet > though so if you find other ways, I'm all ears. ;-) Me neither, I'm just guessing my way ahead :) We could potentially derive from one of the requesthandlers and implement our own, but then we'd have to figure out a good way to get all the request data from the web server to the handler script, essentially reinventing CGI. Unless we want to do it all in one mega-script, in-process. Oh well... - Kim |
|
From: Allister L. S. <all...@gm...> - 2008-09-09 22:05:23
|
Hi folks, I've created a new python server that provides all the headers (as defined in RFC 2616). I created a handler based on CGIHTTPRequestHandler (which already provides HEAD, POST, and GET). I've also placed methods for PUT, DELETE, TRACE, etc., although for the moment all they do is return a 405 error. Once I get to reading the HTTP specs, I might have a better idea on how to implement them. But if you know what to do, go ahead and try your python kungfu on the methods I've prepared. :-) The new server is in server/http_test_server.py and it is now called from server/http_test_server.cpp. In server/cgi-bin/requestinfo.py, I also added sample code on how to get these headers -- although it should be pretty obvious ;-) Cheers, Allister On Tue, Sep 9, 2008 at 1:29 PM, Kim Gräsman <kim...@gm...> wrote: > Hi Dean, > > On Tue, Sep 9, 2008 at 11:22, Dean Michael Berris > <mik...@gm...> wrote: > >> So, as far as I can see there's no reliable way to dump all request > >> headers to the response from a CGI script, via the Python server. The > >> ones that make it through are listed in the run_cgi method in Python's > >> CGIHTTPServer.py. Content-Length and Content-Type are among them, so I > >> thought I had it working for a while :-/ > > > > Is there no way to define which headers (or if all headers) should be > preserved? > > Not as far as I can tell... I think maybe this problem is part of the > CGI concept as well; since it uses environment vars to pass on > selected headers, there's a risk that custom headers overwrite env > vars that the shell depends upon. Unless you jump through hoops to > escape them, etc, or encode all headers into one long string, somehow. > But then it's no longer CGI per se. > > > That should be alright... We may find a better way to go about things > > if we write our own HTTPServer extension which does what we want > > instead of relying on the CGIHTTPServer that Python defines. > > > > I don't have enough Python kung fu to be able to pull it off yet > > though so if you find other ways, I'm all ears. ;-) > > Me neither, I'm just guessing my way ahead :) > > We could potentially derive from one of the requesthandlers and > implement our own, but then we'd have to figure out a good way to get > all the request data from the web server to the handler script, > essentially reinventing CGI. Unless we want to do it all in one > mega-script, in-process. > > Oh well... > > - Kim > > |
|
From: Dean M. B. <mik...@gm...> - 2008-09-10 03:34:06
|
Hi Allister! On Wed, Sep 10, 2008 at 6:05 AM, Allister Levi Sanchez <all...@gm...> wrote: > Hi folks, > > I've created a new python server that provides all the headers (as defined > in RFC 2616). I created a handler based on CGIHTTPRequestHandler (which > already provides HEAD, POST, and GET). I've also placed methods for PUT, > DELETE, TRACE, etc., although for the moment all they do is return a 405 > error. Once I get to reading the HTTP specs, I might have a better idea on > how to implement them. But if you know what to do, go ahead and try your > python kungfu on the methods I've prepared. :-) > > The new server is in server/http_test_server.py and it is now called from > server/http_test_server.cpp. In server/cgi-bin/requestinfo.py, I also added > sample code on how to get these headers -- although it should be pretty > obvious ;-) > Sweet! Thanks. :-D Would love to see more tests in localhost_test that take advantage of this new testing server. :-D > Cheers, > Allister > Thanks Allister! :-) -- Dean Michael C. Berris Software Engineer, Friendster, Inc. |
|
From: K. G. <kim...@gm...> - 2008-09-10 06:12:52
|
Hi Allister, On Wed, Sep 10, 2008 at 00:05, Allister Levi Sanchez <all...@gm...> wrote: > > The new server is in server/http_test_server.py and it is now called from > server/http_test_server.cpp. In server/cgi-bin/requestinfo.py, I also added > sample code on how to get these headers -- although it should be pretty > obvious ;-) Cool, thanks! I had trouble with custom headers, though, and it looks like they're still hard to pull through? I'm thinking about maybe base64-encoding all of self.headers and putting it into an env var as-is, and then decoding it on the CGI side. That should do the trick... - Kim |
|
From: Allister L. S. <all...@gm...> - 2008-09-10 07:07:03
|
Hi Kim,
On Wed, Sep 10, 2008 at 8:12 AM, Kim Gräsman <kim...@gm...> wrote:
> Cool, thanks!
You're welcome :-)
> I had trouble with custom headers, though, and it looks like they're
> still hard to pull through?
>
> I'm thinking about maybe base64-encoding all of self.headers and
> putting it into an env var as-is, and then decoding it on the CGI
> side. That should do the trick...
If we look deep at the code (see $PYTHON_HOME/Lib/rfc822.py), self.headers
is simply a list. One can simply concatenate the elements of a list (with a
really unique separator) and add it into os.environ. Something like this in
http_test_server.py:
all_headers = ',--aUniqueSeparator--,'.join(self.headers)
os.environ['HTTP_ALL_HEADERS'] = all_headers
and in the CGI script, we can easily recover it:
list_of_all_headers =
os.environ.get('HTTP_ALL_HEADERS','').split(',--aUniqueSeparator--,')
There you have it. I'll commit it after work -- or you could commit it
yourself ;-)
Cheers,
Allister |
|
From: K. G. <kim...@gm...> - 2008-09-10 07:50:13
|
Hi Allister,
On Wed, Sep 10, 2008 at 09:07, Allister Levi Sanchez
<all...@gm...> wrote:
>
> If we look deep at the code (see $PYTHON_HOME/Lib/rfc822.py), self.headers
> is simply a list. One can simply concatenate the elements of a list (with a
> really unique separator) and add it into os.environ. Something like this in
> http_test_server.py:
> all_headers = ',--aUniqueSeparator--,'.join(self.headers)
> os.environ['HTTP_ALL_HEADERS'] = all_headers
> and in the CGI script, we can easily recover it:
> list_of_all_headers =
> os.environ.get('HTTP_ALL_HEADERS','').split(',--aUniqueSeparator--,')
> There you have it. I'll commit it after work -- or you could commit it
> yourself ;-)
Sweet!
- Kim
|
|
From: K. G. <kim...@gm...> - 2008-09-10 14:43:02
|
Hi Allister,
On Wed, Sep 10, 2008 at 09:07, Allister Levi Sanchez
<all...@gm...> wrote:
> Something like this in
> http_test_server.py:
> all_headers = ',--aUniqueSeparator--,'.join(self.headers)
> os.environ['HTTP_ALL_HEADERS'] = all_headers
> and in the CGI script, we can easily recover it:
> list_of_all_headers =
> os.environ.get('HTTP_ALL_HEADERS','').split(',--aUniqueSeparator--,')
> There you have it. I'll commit it after work -- or you could commit it
> yourself ;-)
It wasn't quite as easy, but almost. I've committed a version that
forwards all headers, newline-separated, as a string in
HTTP_ALL_HEADERS.
Not sure if it was a good idea, but I replaced all your named header
handling with this simple, raw, forwarding. Maybe we should have both?
Not sure if it makes sense, though.
All tests seem to run anyway, at least on Windows.
Thanks,
- Kim
|
|
From: Allister L. S. <all...@gm...> - 2008-09-10 15:38:59
|
Hi Kim, I think it'd be better to parse the standard headers first in the handler class (HttpTestHandler), to make it easier for people to write scripts for various tests, and not have to duplicate code for the standard HTTP headers. Besides, we only use HTTP_ALL_HEADERS for custom headers. We can then write different CGI scripts for tests using different custom headers. So having both could be beneficial. Also, I think it's better that we expose the command handles do_GET, do_POST, do_DELETE, etc. (as was the case in revision 90 of http_test_server.py) at the level of the HttpTestHandler class as we are most likely to overload them in the future to fully comply with RFC 2616. That said though, I would love to hear the thoughts of the others on this. Cheers, Allister On Wed, Sep 10, 2008 at 4:42 PM, Kim Gräsman <kim...@gm...> wrote: > Hi Allister, > > It wasn't quite as easy, but almost. I've committed a version that > forwards all headers, newline-separated, as a string in > HTTP_ALL_HEADERS. > > Not sure if it was a good idea, but I replaced all your named header > handling with this simple, raw, forwarding. Maybe we should have both? > Not sure if it makes sense, though. > > All tests seem to run anyway, at least on Windows. > > Thanks, > - Kim > |
|
From: K. G. <kim...@gm...> - 2008-09-11 07:49:17
|
Hi Allister, On Wed, Sep 10, 2008 at 17:34, Allister Levi Sanchez <all...@gm...> wrote: > > I think it'd be better to parse the standard headers first in the handler > class (HttpTestHandler), to make it easier for people to write scripts for > various tests, and not have to duplicate code for the standard HTTP headers. > Besides, we only use HTTP_ALL_HEADERS for custom headers. We can then > write different CGI scripts for tests using different custom headers. So > having both could be beneficial. I'm thinking that it should be simple to build a little header dictionary that can parse the HTTP_ALL_HEADERS string, and use it from all CGI scripts. That way, the headers are available similarly to cgi.FieldStorage().headers, but in raw form. That said, my Python-Fu is not so strong, either, so I'm not sure what's involved in making a module that's importable from other scripts...? > Also, I think it's better that we expose the command handles do_GET, > do_POST, do_DELETE, etc. (as was the case in revision 90 of > http_test_server.py) at the level of the HttpTestHandler class as we are > most likely to overload them in the future to fully comply with RFC 2616. Ah, I just saw that they all delegated to the base, and so I figured they were redundant. To me it seems better to just override when necessary, but I don't have a strong opinion either way. - Kim |
|
From: Allister L. S. <all...@gm...> - 2008-09-11 08:07:53
|
Hi Kim, I'm thinking that it should be simple to build a little header > dictionary that can parse the HTTP_ALL_HEADERS string, and use it from > all CGI scripts. That way, the headers are available similarly to > cgi.FieldStorage().headers, but in raw form. Hmmm, I think you're on to something interesting here... Maybe that's better or more efficient or more elegant than having so many environment variables... Looking forward to what you can cook up then :-) > That said, my Python-Fu is not so strong, either, so I'm not sure > what's involved in making a module that's importable from other > scripts...? As you may know, a module is simply a *.py file. To make it importable, simply put it in the same folder as the importing code or add the path to its containing folder to $PYTHONPATH. > > Also, I think it's better that we expose the command handles do_GET, > > do_POST, do_DELETE, etc. (as was the case in revision 90 of > > http_test_server.py) at the level of the HttpTestHandler class as we are > > most likely to overload them in the future to fully comply with RFC 2616. > > Ah, I just saw that they all delegated to the base, and so I figured > they were redundant. To me it seems better to just override when > necessary, but I don't have a strong opinion either way. > Okay, we'll cross the bridge when we get there :-) We'll add those functions when needed -- although I just thought someone out there might be interested (tempted) in writing the necessary code once he/she sees those poor unimplemented functions :-) Cheers, Allister |
|
From: K. G. <kim...@gm...> - 2008-09-12 16:50:05
|
Hi people, On Thu, Sep 11, 2008 at 10:07, Allister Levi Sanchez <all...@gm...> wrote: > >> I'm thinking that it should be simple to build a little header >> dictionary that can parse the HTTP_ALL_HEADERS string, and use it from >> all CGI scripts. That way, the headers are available similarly to >> cgi.FieldStorage().headers, but in raw form. > > Hmmm, I think you're on to something interesting here... Maybe that's > better or more efficient or more elegant than having so many environment > variables... Looking forward to what you can cook up then :-) I've committed rev 93 to http_integration, which uses this technique to transport all headers from server to cgi script. I added cgisupport.http_headers, which wraps this relative weirdness and exposes the headers as a bona-fide dictionary. Seems to be working... > As you may know, a module is simply a *.py file. To make it importable, > simply put it in the same folder as the importing code or add the path to > its containing folder to $PYTHONPATH. Thanks! Currently cgisupport is only used from the cgi-bin dir, so I placed it there. I think that's good enough, Let me know if there are problems on non-Windows platforms. Cheers, - Kim |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X