Re: [cpp-netlib-devel] HTTP 1.1 and Chunked Transfer Encoding

[Jeroen H. <vex...@gm...>]

Hi Dean,

On Fri, Dec 18, 2009 at 08:07, Dean Michael Berris
<mik...@gm...> wrote:
> Hi Everyone,
>
> So 0.5-devel now has the beginnings of an implementation that will
> enable HTTPS connections; the downside to this is that the library now
> depends on OpenSSL -- I think we can easily make this a compile-time
> switchable dependency by doing preprocessor checks and whatnot, but
> I've yet to put those compile-time switches in.
>
> One of the other crucial things I'm finding right now is the
> requirement to handle 'Transfer-Encoding: chunked' on the HTTP/1.1
> client side. This is high up on my TODO list right now and I'm looking
> for some ideas and pointers on how others think this should be done.
>
> Because we don't have a MIME library yet implemented, I'm thinking of
> returning the whole body as a single string thunk in the meantime.
>
> Here are some "random" thoughts I'm surfacing to get feedback from everyone:
>
>  * I'm looking at exposing a stream interface for
> basic_response<tags::http_keepalive_8bit_udp_resolve> which holds a
> shared_ptr<> to the connection object associated with the call to
> get/post, and allows a synchronous buffered pull
>  * The asynchronous HTTP/1.1 client will support an asynchronous
> function callback which would handle individual chunks or a buffered
> chunk in a streaming fashion -- I still haven't settled on the
> interface to that.
>  * The future-aware version of the client will return a
> future<basic_response<Tag> > which will be built asynchronously on an
> active http client.
>  * Because of chunked encoding, there should be a plugin system that
> allows for handling of different types of encoded content; gzipped
> data, base64 encoded, utf-8, etc.
>
> Right now the simple implementation would be to just get the chunks as
> they come, build a thunk of string, and then return that. Hopefully
> after 0.5 we should be able to implement most of the other important
> things that an HTTP/1.1 client should be able to do.
>
> Another thing that has to be there is support for proxies and cookies,
> but those can be auxiliary to the HTTP client. A cookie registry can
> be implemented outside of the HTTP client and will be able to add the
> appropriate cookies to HTTP requests for a certain domain, hopefully
> through the directives interface we already expose and support.
>
> In the next few days I'll finish up on some implementation details and
> the HTTPS support over HTTP/1.1 and HTTP/1.0. Testing help and
> thoughts would be very much appreciated.
>
> BTW, I've written up the Wiki page for the tags that I intend to
> implement: http://wiki.github.com/mikhailberis/cpp-netlib/tags . Help
> and feedback would be very much appreciated.
>
> --
> Dean Michael Berris
> blog.cplusplus-soup.com | twitter.com/mikhailberis
> linkedin.com/in/mikhailberis | facebook.com/dean.berris | deanberris.com
>

This is some very nice progress, there is one thing that comes to mind
though considering chunked transfers. These can get very large, and
storing them in a string could lead to a lot of reallocations of that
string I believe. Maybe a rope,
http://en.wikipedia.org/wiki/Rope_(computer_science), is an option as
a data structure to store the chunks.

I'll have a proper look at the interface you propose tonight if I have time.

Jeroen Habraken