cpp-netlib-devel

[cpp-netlib-devel] Finished Integrating HTTP Incremental Parser

[Dean M. B. <mik...@gm...>]

Hi Guys!

I just wanted to give a heads up to everyone on the list, and give
everyone an update on what I've been up to so far.

First though, there's been a sudden change in my availability to work
on cpp-netlib. As of yesterday Philippine time I no longer have
anything else to do but to work on cpp-netlib (and find a job on the
side). My consulting engagement had been cut short due to some
unfortunate circumstances. So now until I get a new gig, I'm going to
be dedicating my days working on cpp-netlib (yay! :D).

So here's what's been happening to the library:

  * The tags system now uses heavy metaprogramming with Boost.MPL.
  * The incremental parser now has a stable generic interface and I
intend to lift a concept out of it as soon as I get around to
implementing other incremental parsers for different protocols.
  * There are a number of enhancements in the way the directive system
works, using the Boost.Variant trick that I emailed a couple weeks
ago. That's in 0.7-devel and should make directive making a little
more pleasant with definite extension points to look through.
  * The tags "http_async_8bit_(udp|tcp)_resolve" are now supported and
implemented; there should be no changes to client code that wants to
use the asynchronous (I should say "active") HTTP client
implementation.

There are a number of things that have yet to be done for packaging
and releasing 0.7-devel:

  * Make the HTTP 1.0 and HTTP 1.1 tests pass. Right now only the HTTP
1.0 test includes the new asynchronous tags in the build. The HTTP 1.1
chunked-encoding transfer is what I intend to work on next, which will
also require a new implementation of the incremental parser to support
just the chunked-encoding body.
  * Update and merge the documentation.
  * Implement a callback-based HTTP Client API. The rationale for this
is so that instead of returning a response object that has
Boost.Thread futures embedded in it (and neatly abstracted away), the
client can then just call the appropriate callback(s) for processing
headers and the body stream as they become available.
  * Implement a streaming HTTP Client API. This means returning a
response object that has a stream for a body instead of a pre-built
string, as well as taking a request object that has a stream for a
body. This should help with file-uploads through PUT/POST to make it
not require all the contents of the file be in memory, a shared_ptr<>
to a stream should suffice.

As of now I don't have access to a MSVC compiler yet -- I'll see if I
can get my hands on a Windows machine or VM soon, so that I can
continue development on Windows -- but I have been building and
testing with Clang and GCC in Linux, against Boost 1.44.0. I need some
help in this regard as well as with the identification of issues in
case there are any that you come across in the implementation.

Thanks for reading up to here and let me know if there's anything
immediate that I need to address before I hunker down and finish off
the HTTP client implementation for 0.7.

Have a good one everyone and I hope to hear from you soon!

-- 
Dean Michael Berris
deanberris.com

Re: [cpp-netlib-devel] Finished Integrating HTTP Incremental Parser

[Dean M. B. <mik...@gm...>]

Hey Guys!

Just an update. I'm still looking for a job, so if you've got any C++
jobs lying around, referrals would be greatly appreciated. I'm already
talking with a firm now though, but still keeping my options open.
Hopefully by November I'll go decide on what job to take.

Anyway, more information in-lined below.

On Wed, Sep 29, 2010 at 5:46 PM, Dean Michael Berris
<mik...@gm...> wrote:
>
>  * The tags system now uses heavy metaprogramming with Boost.MPL.

This is still true, and I highly recommend using clang [0] for your
development on Linux/Mac now.

>  * The tags "http_async_8bit_(udp|tcp)_resolve" are now supported and
> implemented; there should be no changes to client code that wants to
> use the asynchronous (I should say "active") HTTP client
> implementation.
>

It should be noted that the http client and the response objects are a
little heavier now in the asynchronous version. That's because the
client encapsulates a lifetime thread, and the response object
encapsulates futures internally; this uses Boost.Thread
implementations of the thread and the futures.

> There are a number of things that have yet to be done for packaging
> and releasing 0.7-devel:
>
>  * Make the HTTP 1.0 and HTTP 1.1 tests pass. Right now only the HTTP
> 1.0 test includes the new asynchronous tags in the build. The HTTP 1.1
> chunked-encoding transfer is what I intend to work on next, which will
> also require a new implementation of the incremental parser to support
> just the chunked-encoding body.

I still don't have the chunked transfer encoding done, and I would
really like to know if people do rely on chunked transfer encoding
before I go on working on this implementation. In the meantime the raw
body is being stored by the client regardless of the transfer
encoding.

>  * Update and merge the documentation.

I'll work with Glyn on this one.

>  * Implement a callback-based HTTP Client API. The rationale for this
> is so that instead of returning a response object that has
> Boost.Thread futures embedded in it (and neatly abstracted away), the
> client can then just call the appropriate callback(s) for processing
> headers and the body stream as they become available.

I realize that this requires a worker thread pool implementation which
has a "life of its own" and lives on different threads that don't deal
with I/O. This may have to wait as I have to implement this thread
pool implementation as well.

>  * Implement a streaming HTTP Client API. This means returning a
> response object that has a stream for a body instead of a pre-built
> string, as well as taking a request object that has a stream for a
> body. This should help with file-uploads through PUT/POST to make it
> not require all the contents of the file be in memory, a shared_ptr<>
> to a stream should suffice.
>

Also, this means that the response type may have to change for the
tags that enable this interface. What has to happen is the response
objects will contain a shared iostream that's bound to the socket of
the HTTP request to the server, that starts with the body of the HTTP
Response Message. This is in the case that the callback would like to
deal with the response body as a stream. I have an implementation
somewhere that doesn't expose a stream and holds the contents of the
response in memory as a string, which should do for most cases that
the responses aren't huge.

> As of now I don't have access to a MSVC compiler yet -- I'll see if I
> can get my hands on a Windows machine or VM soon, so that I can
> continue development on Windows -- but I have been building and
> testing with Clang and GCC in Linux, against Boost 1.44.0. I need some
> help in this regard as well as with the identification of issues in
> case there are any that you come across in the implementation.
>

Now that the tests build and pass on Linux with GCC and Clang, can
someone help out with the Windows testing?

Thanks guys and I hope this helps.

PS. Please review the code and let me know if you find any glaring
bugs or really ugly things that need explanation. That would be
greatly appreciated. [1]

[0] clang -- http://clang.llvm.org/
[1] 0.7-devel -- http://github.com/mikhailberis/cpp-netlib/tree/0.7-devel

-- 
Dean Michael Berris
deanberris.com

Re: [cpp-netlib-devel] Finished Integrating HTTP Incremental Parser

[Glyn M. <gly...@gm...>]

Hi Dean,

On 5 October 2010 19:15, Dean Michael Berris <mik...@gm...> wrote:

> Hey Guys!
>
> Just an update. I'm still looking for a job, so if you've got any C++
> jobs lying around, referrals would be greatly appreciated. I'm already
> talking with a firm now though, but still keeping my options open.
> Hopefully by November I'll go decide on what job to take.
>

I hope your job search is going well.


>
> Anyway, more information in-lined below.
>
> On Wed, Sep 29, 2010 at 5:46 PM, Dean Michael Berris
> <mik...@gm...> wrote:
> >
> >  * The tags system now uses heavy metaprogramming with Boost.MPL.
>
>
Following on from another thread, all these changes from 0.6-0.7 need to be
added to the docs, I think it means that what we have now is out-of-date.


> >  * Make the HTTP 1.0 and HTTP 1.1 tests pass. Right now only the HTTP
> > 1.0 test includes the new asynchronous tags in the build. The HTTP 1.1
> > chunked-encoding transfer is what I intend to work on next, which will
> > also require a new implementation of the incremental parser to support
> > just the chunked-encoding body.
>
> I still don't have the chunked transfer encoding done, and I would
> really like to know if people do rely on chunked transfer encoding
> before I go on working on this implementation. In the meantime the raw
> body is being stored by the client regardless of the transfer
> encoding.
>
> >  * Update and merge the documentation.
>
> I'll work with Glyn on this one.
>
> >  * Implement a callback-based HTTP Client API. The rationale for this
> > is so that instead of returning a response object that has
> > Boost.Thread futures embedded in it (and neatly abstracted away), the
> > client can then just call the appropriate callback(s) for processing
> > headers and the body stream as they become available.
>
> I realize that this requires a worker thread pool implementation which
> has a "life of its own" and lives on different threads that don't deal
> with I/O. This may have to wait as I have to implement this thread
> pool implementation as well.
>
> >  * Implement a streaming HTTP Client API. This means returning a
> > response object that has a stream for a body instead of a pre-built
> > string, as well as taking a request object that has a stream for a
> > body. This should help with file-uploads through PUT/POST to make it
> > not require all the contents of the file be in memory, a shared_ptr<>
> > to a stream should suffice.
> >
>
> Also, this means that the response type may have to change for the
> tags that enable this interface. What has to happen is the response
> objects will contain a shared iostream that's bound to the socket of
> the HTTP request to the server, that starts with the body of the HTTP
> Response Message. This is in the case that the callback would like to
> deal with the response body as a stream. I have an implementation
> somewhere that doesn't expose a stream and holds the contents of the
> response in memory as a string, which should do for most cases that
> the responses aren't huge.
>
> > As of now I don't have access to a MSVC compiler yet -- I'll see if I
> > can get my hands on a Windows machine or VM soon, so that I can
> > continue development on Windows -- but I have been building and
> > testing with Clang and GCC in Linux, against Boost 1.44.0. I need some
> > help in this regard as well as with the identification of issues in
> > case there are any that you come across in the implementation.
> >
>
> Now that the tests build and pass on Linux with GCC and Clang, can
> someone help out with the Windows testing?
>
> Thanks guys and I hope this helps.
>
> PS. Please review the code and let me know if you find any glaring
> bugs or really ugly things that need explanation. That would be
> greatly appreciated. [1]
>

I will make the time to do this.

Thanks,
Glyn

Re: [cpp-netlib-devel] Finished Integrating HTTP Incremental Parser

[Dean M. B. <mik...@gm...>]

On Wed, Oct 6, 2010 at 4:33 AM, Glyn Matthews <gly...@gm...> wrote:
> Hi Dean,
>
> On 5 October 2010 19:15, Dean Michael Berris <mik...@gm...> wrote:
>>
>> Hey Guys!
>>
>> Just an update. I'm still looking for a job, so if you've got any C++
>> jobs lying around, referrals would be greatly appreciated. I'm already
>> talking with a firm now though, but still keeping my options open.
>> Hopefully by November I'll go decide on what job to take.
>
> I hope your job search is going well.
>

Yes, going well enough I suppose. Thanks Glyn. :)

>>
>> Anyway, more information in-lined below.
>>
>> On Wed, Sep 29, 2010 at 5:46 PM, Dean Michael Berris
>> <mik...@gm...> wrote:
>> >
>> >  * The tags system now uses heavy metaprogramming with Boost.MPL.
>>
>
> Following on from another thread, all these changes from 0.6-0.7 need to be
> added to the docs, I think it means that what we have now is out-of-date.
>

Yes, definitely. I'm going to spend my day here bringing the docs up to date.

>>
>> PS. Please review the code and let me know if you find any glaring
>> bugs or really ugly things that need explanation. That would be
>> greatly appreciated. [1]
>
> I will make the time to do this.

Thanks Glyn, this would be greatly appreciated! :)

-- 
Dean Michael Berris
deanberris.com