cpp-netlib-devel

[cpp-netlib-devel] HTTP 1.1 and Chunked Transfer Encoding

[Dean M. B. <mik...@gm...>]

Hi Everyone,

So 0.5-devel now has the beginnings of an implementation that will
enable HTTPS connections; the downside to this is that the library now
depends on OpenSSL -- I think we can easily make this a compile-time
switchable dependency by doing preprocessor checks and whatnot, but
I've yet to put those compile-time switches in.

One of the other crucial things I'm finding right now is the
requirement to handle 'Transfer-Encoding: chunked' on the HTTP/1.1
client side. This is high up on my TODO list right now and I'm looking
for some ideas and pointers on how others think this should be done.

Because we don't have a MIME library yet implemented, I'm thinking of
returning the whole body as a single string thunk in the meantime.

Here are some "random" thoughts I'm surfacing to get feedback from everyone:

  * I'm looking at exposing a stream interface for
basic_response<tags::http_keepalive_8bit_udp_resolve> which holds a
shared_ptr<> to the connection object associated with the call to
get/post, and allows a synchronous buffered pull
  * The asynchronous HTTP/1.1 client will support an asynchronous
function callback which would handle individual chunks or a buffered
chunk in a streaming fashion -- I still haven't settled on the
interface to that.
  * The future-aware version of the client will return a
future<basic_response<Tag> > which will be built asynchronously on an
active http client.
  * Because of chunked encoding, there should be a plugin system that
allows for handling of different types of encoded content; gzipped
data, base64 encoded, utf-8, etc.

Right now the simple implementation would be to just get the chunks as
they come, build a thunk of string, and then return that. Hopefully
after 0.5 we should be able to implement most of the other important
things that an HTTP/1.1 client should be able to do.

Another thing that has to be there is support for proxies and cookies,
but those can be auxiliary to the HTTP client. A cookie registry can
be implemented outside of the HTTP client and will be able to add the
appropriate cookies to HTTP requests for a certain domain, hopefully
through the directives interface we already expose and support.

In the next few days I'll finish up on some implementation details and
the HTTPS support over HTTP/1.1 and HTTP/1.0. Testing help and
thoughts would be very much appreciated.

BTW, I've written up the Wiki page for the tags that I intend to
implement: http://wiki.github.com/mikhailberis/cpp-netlib/tags . Help
and feedback would be very much appreciated.

-- 
Dean Michael Berris
blog.cplusplus-soup.com | twitter.com/mikhailberis
linkedin.com/in/mikhailberis | facebook.com/dean.berris | deanberris.com

Re: [cpp-netlib-devel] HTTP 1.1 and Chunked Transfer Encoding

[Jeroen H. <vex...@gm...>]

Hi Dean,

On Fri, Dec 18, 2009 at 08:07, Dean Michael Berris
<mik...@gm...> wrote:
> Hi Everyone,
>
> So 0.5-devel now has the beginnings of an implementation that will
> enable HTTPS connections; the downside to this is that the library now
> depends on OpenSSL -- I think we can easily make this a compile-time
> switchable dependency by doing preprocessor checks and whatnot, but
> I've yet to put those compile-time switches in.
>
> One of the other crucial things I'm finding right now is the
> requirement to handle 'Transfer-Encoding: chunked' on the HTTP/1.1
> client side. This is high up on my TODO list right now and I'm looking
> for some ideas and pointers on how others think this should be done.
>
> Because we don't have a MIME library yet implemented, I'm thinking of
> returning the whole body as a single string thunk in the meantime.
>
> Here are some "random" thoughts I'm surfacing to get feedback from everyone:
>
>  * I'm looking at exposing a stream interface for
> basic_response<tags::http_keepalive_8bit_udp_resolve> which holds a
> shared_ptr<> to the connection object associated with the call to
> get/post, and allows a synchronous buffered pull
>  * The asynchronous HTTP/1.1 client will support an asynchronous
> function callback which would handle individual chunks or a buffered
> chunk in a streaming fashion -- I still haven't settled on the
> interface to that.
>  * The future-aware version of the client will return a
> future<basic_response<Tag> > which will be built asynchronously on an
> active http client.
>  * Because of chunked encoding, there should be a plugin system that
> allows for handling of different types of encoded content; gzipped
> data, base64 encoded, utf-8, etc.
>
> Right now the simple implementation would be to just get the chunks as
> they come, build a thunk of string, and then return that. Hopefully
> after 0.5 we should be able to implement most of the other important
> things that an HTTP/1.1 client should be able to do.
>
> Another thing that has to be there is support for proxies and cookies,
> but those can be auxiliary to the HTTP client. A cookie registry can
> be implemented outside of the HTTP client and will be able to add the
> appropriate cookies to HTTP requests for a certain domain, hopefully
> through the directives interface we already expose and support.
>
> In the next few days I'll finish up on some implementation details and
> the HTTPS support over HTTP/1.1 and HTTP/1.0. Testing help and
> thoughts would be very much appreciated.
>
> BTW, I've written up the Wiki page for the tags that I intend to
> implement: http://wiki.github.com/mikhailberis/cpp-netlib/tags . Help
> and feedback would be very much appreciated.
>
> --
> Dean Michael Berris
> blog.cplusplus-soup.com | twitter.com/mikhailberis
> linkedin.com/in/mikhailberis | facebook.com/dean.berris | deanberris.com
>

This is some very nice progress, there is one thing that comes to mind
though considering chunked transfers. These can get very large, and
storing them in a string could lead to a lot of reallocations of that
string I believe. Maybe a rope,
http://en.wikipedia.org/wiki/Rope_(computer_science), is an option as
a data structure to store the chunks.

I'll have a proper look at the interface you propose tonight if I have time.

Jeroen Habraken

Re: [cpp-netlib-devel] HTTP 1.1 and Chunked Transfer Encoding

[Dean M. B. <mik...@gm...>]

Hi Jeroen!

On Sat, Dec 19, 2009 at 12:33 AM, Jeroen Habraken <vex...@gm...> wrote:
>
> On Fri, Dec 18, 2009 at 08:07, Dean Michael Berris
> <mik...@gm...> wrote:
>> Hi Everyone,
>>
[snip]
>
> This is some very nice progress,

Thanks!

> there is one thing that comes to mind
> though considering chunked transfers. These can get very large, and
> storing them in a string could lead to a lot of reallocations of that
> string I believe.

Yeah, unless you use an ostringstream which manages memory better than
the normal string.

> Maybe a rope,
> http://en.wikipedia.org/wiki/Rope_(computer_science), is an option as
> a data structure to store the chunks.
>

Yup, but unfortunately they aren't part of the standard C++ library IIRC.

> I'll have a proper look at the interface you propose tonight if I have time.

Thanks, feedback is very much appreciated!

-- 
Dean Michael Berris
blog.cplusplus-soup.com | twitter.com/mikhailberis
linkedin.com/in/mikhailberis | facebook.com/dean.berris | deanberris.com