cpp-netlib-devel

[cpp-netlib-devel] TODO's in http_integration

[Dean M. B. <mik...@gm...>]

Hi Guys,

As I've outlined in my latest commit (revision 60:
http://cpp-netlib.svn.sourceforge.net/viewvc/cpp-netlib?view=rev&revision=60)
I have a few items there that may need to be addressed somehow.

TODO:
- multi-line headers processing
- multi-part message posting
- change from multi-map to vector to preserve request header ordering?

With multi-line headers processing, I'm not sure if this is ever going
to be a problem with HTTP -- I'm not sure of the HTTP spec requires
that headers be a single line -- but I am a little worried about our
lack of HTTP headers checking. Does anybody have experience with
special headers that may span multiple lines? Or am I imagining things
here? :D

About multi-part message posting, I have not seen any documentation
regarding the semantics for a multi-part POST message. If it has
something to do with MIME, then that library has yet to be written
external to the HTTP implementation. AFAICT, each HTTP POST Message
has a single body, and a single Content-Length and Content-Type
headers -- If the actual body is a multi-part message, then I think
the body should have to be formed accordingly, and the Content-Type
correctly as well to something like 'multi-part/mixed;
boundary="some_boundary"'. The HTTP Client will then be sufficient in
its current form to be able to do:

  client_.put(request, "multi-part/mixed; boundary=\"some_boundary\"",
my_multipart_mime_body);

Another thing that may be required is to change the multi-map
underlying implementation for HTTP to use a list of std::pair<>'s to
preserve ordering. I've read something about the order of occurrence
and possible merging of repeating HTTP Headers.

One last thing which needs some scrutiny: maybe we should define a
User-Agent: header to identify cpp-netlib? :D

Thanks for the time (if you get this far) and let me know what you think! :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] TODO's in http_integration

[Divye K. <div...@gm...>]

Hi,

On Tue, Aug 12, 2008 at 2:24 PM, Dean Michael Berris <mik...@gm...
> wrote:

>
> About multi-part message posting, I have not seen any documentation
> regarding the semantics for a multi-part POST message. If it has
> something to do with MIME, then that library has yet to be written
> external to the HTTP implementation. AFAICT, each HTTP POST Message
> has a single body, and a single Content-Length and Content-Type
> headers -- If the actual body is a multi-part message, then I think
> the body should have to be formed accordingly, and the Content-Type
> correctly as well to something like 'multi-part/mixed;
> boundary="some_boundary"'. The HTTP Client will then be sufficient in
> its current form to be able to do:
>

The documentation for sending multiple files/form-fields/data via POST
(multipart data) is in RFC 2388 (copy at
http://www.faqs.org/rfcs/rfc2388.html ). A quick example of how the body is
structured for a multipart POST request is located at
http://www.codeproject.com/KB/cs/multipart_request_C_.aspx

I think supporting this feature should be our responsibility. The user
should not need to deal with the intricacies of structuring the data
according to the rules of a multipart form. It will be a source of errors in
the client application. The user should simply provide a content type and
corresponding stream and we should draft a proper request out of it.



> Another thing that may be required is to change the multi-map
> underlying implementation for HTTP to use a list of std::pair<>'s to
> preserve ordering. I've read something about the order of occurrence
> and possible merging of repeating HTTP Headers.
>

Some information about merging headers is located at
http://httpd.apache.org/docs/2.2/mod/mod_headers.html

With multi-line headers processing, I'm not sure if this is ever going
to be a problem with HTTP -- I'm not sure of the HTTP spec requires
that headers be a single line -- but I am a little worried about our
lack of HTTP headers checking. Does anybody have experience with
special headers that may span multiple lines? Or am I imagining things
here? :D

See section 4.2 - Message Headers of RFC 2616 (
http://www.faqs.org/rfcs/rfc2388.html ).
Headers can span multiple lines. This would require a change in the
processing of the headers.



> One last thing which needs some scrutiny: maybe we should define a
> User-Agent: header to identify cpp-netlib? :D
>

Maybe a default User-Agent header can be provided. Any other ideas?

Divye

-- 
An idealist is one who, on noticing that a rose smells better than a
cabbage, concludes that it will also make better soup.
   H. L. Mencken (1880 - 1956)
My official web site: http://people.iitr.ernet.in/shp/061305/
Webmaster: http://www.drkapoorsclinic.com
Blog: http://divyekapoor.blogspot.com

Re: [cpp-netlib-devel] TODO's in http_integration

[Divye K. <div...@gm...>]

>
>
> See section 4.2 - Message Headers of RFC 2616 (
> http://www.faqs.org/rfcs/rfc2388.html ).
> Headers can span multiple lines. This would require a change in the
> processing of the headers.
>

I meant http://www.faqs.org/rfcs/rfc2616.html

Also, right now, we don't support the rather infrequently used TRACE and
CONNECT methods. Maybe we should for the sake of completeness?

Another thing that struck me was the fact that using the just checked in
example, if you try to access http://www.google.com - you get a redirect
response (3xx status code). However, we don't automatically redirect. Should
the choice to redirect be left to the user or should we give an option to
enable automatic redirection? Personally, I am in favour of providing an
option to enable or disable automatic redirection for requests. What do you
think?

Divye

-- 
An idealist is one who, on noticing that a rose smells better than a
cabbage, concludes that it will also make better soup.
   H. L. Mencken (1880 - 1956)
My official web site: http://people.iitr.ernet.in/shp/061305/
Webmaster: http://www.drkapoorsclinic.com
Blog: http://divyekapoor.blogspot.com

Re: [cpp-netlib-devel] TODO's in http_integration

[Dean M. B. <mik...@gm...>]

On Wed, Aug 13, 2008 at 3:45 AM, Divye Kapoor <div...@gm...> wrote:
>>
>> See section 4.2 - Message Headers of RFC 2616 (
>> http://www.faqs.org/rfcs/rfc2388.html ).
>> Headers can span multiple lines. This would require a change in the
>> processing of the headers.
>
> I meant http://www.faqs.org/rfcs/rfc2616.html
>

Okay, thanks. I'll look into it.

Perhaps we should have a test for that as well to make sure there's a
bug in the client in the first place. Can you make this happen with
the Python CGI script?

> Also, right now, we don't support the rather infrequently used TRACE and
> CONNECT methods. Maybe we should for the sake of completeness?
>

Right. This should be trivial given how the client code is structured right now.

> Another thing that struck me was the fact that using the just checked in
> example, if you try to access http://www.google.com - you get a redirect
> response (3xx status code). However, we don't automatically redirect. Should
> the choice to redirect be left to the user or should we give an option to
> enable automatic redirection? Personally, I am in favour of providing an
> option to enable or disable automatic redirection for requests. What do you
> think?
>

I was thinking about that. Actually, even if you just go to
http://boost.org/  you will get a redirect to http://www.boost.org/.

Right now the client does not have a constructor. We can make use of
the Boost.Parameter lib to make some of the options easily defined in
the client constructor -- like redirect, authentication information,
proxy, etc. I'm also thinking for having an overload to each method
that allows you to define these options in each call.

If we can have the Python CGI test return a redirect, perhaps we can
expose the bug and test it as well.

How's the POST handler going? :D

Thanks for the links and responses. They definitely help a lot.

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] TODO's in http_integration

[Dean M. B. <mik...@gm...>]

On Wed, Aug 13, 2008 at 3:39 AM, Divye Kapoor <div...@gm...> wrote:
>
> On Tue, Aug 12, 2008 at 2:24 PM, Dean Michael Berris
> <mik...@gm...> wrote:
>>
>> About multi-part message posting, I have not seen any documentation
>> regarding the semantics for a multi-part POST message. If it has
>> something to do with MIME, then that library has yet to be written
>> external to the HTTP implementation. AFAICT, each HTTP POST Message
>> has a single body, and a single Content-Length and Content-Type
>> headers -- If the actual body is a multi-part message, then I think
>> the body should have to be formed accordingly, and the Content-Type
>> correctly as well to something like 'multi-part/mixed;
>> boundary="some_boundary"'. The HTTP Client will then be sufficient in
>> its current form to be able to do:
>
> The documentation for sending multiple files/form-fields/data via POST
> (multipart data) is in RFC 2388 (copy at
> http://www.faqs.org/rfcs/rfc2388.html ). A quick example of how the body is
> structured for a multipart POST request is located at
> http://www.codeproject.com/KB/cs/multipart_request_C_.aspx
>

Right, thanks.

> I think supporting this feature should be our responsibility. The user
> should not need to deal with the intricacies of structuring the data
> according to the rules of a multipart form. It will be a source of errors in
> the client application. The user should simply provide a content type and
> corresponding stream and we should draft a proper request out of it.
>

Like I implied, if it has something to do with MIME, we need to write
that library outside of HTTP. An imaginary MIME library would expose a
stream interface, and perhaps provide the following:

<code>

using namespace boost::network::mime;
omimestream multi_part_builder;
multi_part_builder
  << content(
    disposition(
      "form-data",
      make_tuple(
        make_pair("name", "param1")
        )
      ),
    data("param1Value")
    )
  << content(
    disposition(
      "file",
      make_tuple(
        make_pair("name", "file"),
        make_pair("filename", "filename1.txt")
        )
      ),
    type(
      "text/plain"
      ),
    data(file_stream)
  ;

...
client_.post(request, multi_part_builder.type(), multi_part_builder.body());

</code>

This allows us to use the MIME library outside of HTTP and be a usable
library in itself. In this case, the decoupling of the MIME handling
from the HTTP implementation will be a good thing.

I'm not saying it shouldn't be our responsibility -- I agree that we
should make it easier for users -- but how we're going to allow that
to happen has to fit within the goals of the project, which is
building a set of extensible components for building network-aware
applications in C++.

I can write that MIME builder interface with Boost.Proto or from
scratch, but that would require me to context-switch from the HTTP
implementation.

If anybody's interested in picking up that piece of the puzzle, or
would like to be part of the discussion about it, now would be a good
time to sound off. :D

>
>>
>> Another thing that may be required is to change the multi-map
>> underlying implementation for HTTP to use a list of std::pair<>'s to
>> preserve ordering. I've read something about the order of occurrence
>> and possible merging of repeating HTTP Headers.
>
> Some information about merging headers is located at
> http://httpd.apache.org/docs/2.2/mod/mod_headers.html
>

Thanks for the link. This is done on the server-side rather than the
client-side?

I think one thing we can do is to "un-merge" the merged headers
received from the server to make it consistent. In the case of merging
client-formed request headers, we can make it optional -- and should
be easy to do.

>> With multi-line headers processing, I'm not sure if this is ever going
>> to be a problem with HTTP -- I'm not sure of the HTTP spec requires
>> that headers be a single line -- but I am a little worried about our
>> lack of HTTP headers checking. Does anybody have experience with
>> special headers that may span multiple lines? Or am I imagining things
>> here? :D
>
> See section 4.2 - Message Headers of RFC 2616 (
> http://www.faqs.org/rfcs/rfc2388.html ).
> Headers can span multiple lines. This would require a change in the
> processing of the headers.
>

Oh bugger. Thanks for the link. Hmmm... Maybe a test would be good to
expose the bug? :D

>
>>
>> One last thing which needs some scrutiny: maybe we should define a
>> User-Agent: header to identify cpp-netlib? :D
>
> Maybe a default User-Agent header can be provided. Any other ideas?
>

A default should be provided unless it's overridden from the request,
I agree. Test, test, test... ;)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] TODO's in http_integration

[Divye K. <div...@gm...>]

Hi,


> See section 4.2 - Message Headers of RFC 2616 (
> >> http://www.faqs.org/rfcs/rfc2388.html ).
> >> Headers can span multiple lines. This would require a change in the
> >> processing of the headers.
> >
> > I meant http://www.faqs.org/rfcs/rfc2616.html
> >

Perhaps we should have a test for that as well to make sure there's a
> bug in the client in the first place. Can you make this happen with
> the Python CGI script?
>

Yes. It can. I'll try to do it tonight. Otherwise it will have to wait for
the weekend after next as I have a wedding in the family to take care of
over the next few days.


>
> Right now the client does not have a constructor. We can make use of
> the Boost.Parameter lib to make some of the options easily defined in
> the client constructor -- like redirect, authentication information,
> proxy, etc. I'm also thinking for having an overload to each method
> that allows you to define these options in each call.
>

Good idea.


>
> If we can have the Python CGI test return a redirect, perhaps we can
> expose the bug and test it as well.
>

The python CGI cannot return a redirect response due to the way the response
headers are created in the python library code. It pre-empts a 3xx response
by sending a 200 response before calling the CGI script. As of now, we need
to find another way to do it: maybe a simple asio app that redirects a
request to a url on the python server?


>
> How's the POST handler going? :D
>

The wedding's going to push things back for me for a while, so it will be on
the hold list. :(

Divye

-- 
An idealist is one who, on noticing that a rose smells better than a
cabbage, concludes that it will also make better soup.
H. L. Mencken (1880 - 1956)
My official web site: http://people.iitr.ernet.in/shp/061305/
Webmaster: http://www.drkapoorsclinic.com
Blog: http://divyekapoor.blogspot.com

Re: [cpp-netlib-devel] TODO's in http_integration

[Dean M. B. <mik...@gm...>]

Hi Divye,

On Wed, Aug 13, 2008 at 2:03 PM, Divye Kapoor <div...@gm...> wrote:
> Hi,
>
>>
>> See section 4.2 - Message Headers of RFC 2616 (
>> >> http://www.faqs.org/rfcs/rfc2388.html ).
>> >> Headers can span multiple lines. This would require a change in the
>> >> processing of the headers.
>> >
>> > I meant http://www.faqs.org/rfcs/rfc2616.html
>> >
>>
>> Perhaps we should have a test for that as well to make sure there's a
>> bug in the client in the first place. Can you make this happen with
>> the Python CGI script?
>
> Yes. It can. I'll try to do it tonight. Otherwise it will have to wait for
> the weekend after next as I have a wedding in the family to take care of
> over the next few days.
>

No rush, I can try hacking it, but I'm not confident enough touching
Python CGI code right now.

>>
>> Right now the client does not have a constructor. We can make use of
>> the Boost.Parameter lib to make some of the options easily defined in
>> the client constructor -- like redirect, authentication information,
>> proxy, etc. I'm also thinking for having an overload to each method
>> that allows you to define these options in each call.
>
> Good idea.
>

I'll get on it, but it will need some C++-side unit tests. I'll write
those but I think it's going to take some time to implement the
redirect handling to prevent infinite-looping redirects. There's also
the referrer thing that needs to be defined, but I will need to read
the HTTP docs about it as well.

>>
>> If we can have the Python CGI test return a redirect, perhaps we can
>> expose the bug and test it as well.
>
> The python CGI cannot return a redirect response due to the way the response
> headers are created in the python library code. It pre-empts a 3xx response
> by sending a 200 response before calling the CGI script. As of now, we need
> to find another way to do it: maybe a simple asio app that redirects a
> request to a url on the python server?
>

Hmmm... Can't the CGI Server be set up to recognize a specific URI,
then be told to redirect?

>>
>> How's the POST handler going? :D
>
> The wedding's going to push things back for me for a while, so it will be on
> the hold list. :(
>

No rush, enjoy the wedding. :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.

Re: [cpp-netlib-devel] TODO's in http_integration

[Allister L. S. <all...@gm...>]

Hi,

Sorry I've been silent due to a rather busy work week.  But this Friday's a
holiday in France (yehey!) so I'm thinking of working on the MIME part over
the 3-day weekend.  I've no clear idea how to do it yet, but it should be a
good learning experience.

Till this weekend, cheers!
Allister

On Wed, Aug 13, 2008 at 6:45 AM, Dean Michael Berris <mik...@gm...
> wrote:

>
> Like I implied, if it has something to do with MIME, we need to write
> that library outside of HTTP. An imaginary MIME library would expose a
> stream interface, and perhaps provide the following:
>
> <code>
>
> using namespace boost::network::mime;
> omimestream multi_part_builder;
> multi_part_builder
>  << content(
>    disposition(
>      "form-data",
>      make_tuple(
>        make_pair("name", "param1")
>        )
>      ),
>    data("param1Value")
>    )
>  << content(
>    disposition(
>      "file",
>      make_tuple(
>        make_pair("name", "file"),
>        make_pair("filename", "filename1.txt")
>        )
>      ),
>    type(
>      "text/plain"
>      ),
>    data(file_stream)
>  ;
>
> ...
> client_.post(request, multi_part_builder.type(),
> multi_part_builder.body());
>
> </code>
>
> This allows us to use the MIME library outside of HTTP and be a usable
> library in itself. In this case, the decoupling of the MIME handling
> from the HTTP implementation will be a good thing.
>
> I'm not saying it shouldn't be our responsibility -- I agree that we
> should make it easier for users -- but how we're going to allow that
> to happen has to fit within the goals of the project, which is
> building a set of extensible components for building network-aware
> applications in C++.
>
> I can write that MIME builder interface with Boost.Proto or from
> scratch, but that would require me to context-switch from the HTTP
> implementation.
>
> If anybody's interested in picking up that piece of the puzzle, or
> would like to be part of the discussion about it, now would be a good
> time to sound off. :D
>
>
>

Re: [cpp-netlib-devel] TODO's in http_integration

[Dean M. B. <mik...@gm...>]

On Wed, Aug 13, 2008 at 2:24 PM, Allister Levi Sanchez
<all...@gm...> wrote:
> Hi,
>
> Sorry I've been silent due to a rather busy work week.

No worries. We all have that. ;)

> But this Friday's a
> holiday in France (yehey!) so I'm thinking of working on the MIME part over
> the 3-day weekend.  I've no clear idea how to do it yet, but it should be a
> good learning experience.

Just let me know if you need any help on specific things. I may be
able to share some thoughts either through here or through IM. :)

>
> Till this weekend, cheers!

Have a good one! :)

-- 
Dean Michael C. Berris
Software Engineer, Friendster, Inc.