Re: [cpp-netlib-devel] Can cpp_netlib perform asynchronous concurrent requests?

[qjmann <sou...@gm...>]

>
> On Sat, Nov 13, 2010 at 3:32 AM, qjmann<sou...@gm...  <mailto:sou...@gm...>>  wrote:
> >  Documentation contains example HTTP client, but I see it works in blocking
> >  mode:
> >
> >  http_client::response response = client.get(request)
> >
> >  - this call will stop program execution while waiting for server's response.
>
> Not if http_client is a typedef like:
>
>    typedef boost::network::http::basic_client<
>      boost::network::http::tags::http_async_8bit_udp_resolve
>      , 1, 0>  http_client;
>
> >  Is it possible to perform multiple concurrent requests with cpp_netlib
> >  without spawning multiple threads, as it can be done with ASIO asynchronous
> >  sockets?
>
> Yes, use the asynchronous tags for the http_client. This will use
> futures underneath.
>
> In a later release there will be a change in the interface for an
> asynchronous HTTP client that will take in a function to handle
> incoming data from a request -- most probably will follow the same
> interface that the async HTTP server in 0.8-devel will, I still
> haven't decided yet.
>
> >
> >  1) Create single io_service object
> >  2) Create and initialize client objects set (all attached to io_service
> >  object just created)
> >  3) Call io_service.poll()
> >  4) Check client objects for responses received
> >  5) If some responses has not been received yet, then goto step 3
> >
>
> No need for polling really, a callback mechanism or the current
> futures-based implementation should suffice for most cases.
>
> HTH
>
> -- 
> Dean Michael Berris
> deanberris.com
>    

Thank's a lot, I used asynchronous tags, and now "client.get(request)" 
call is not blocking progpam's execution.
In my application there will be a huge queue of URLs (from different 
hosts, it is just a crawler) and very limited set of requester objects 
(app must handle ~100 concurrent connections). So I want to implement a 
loop:

     while(/* queue has urls */) {
         for(/* loop through client objects set */)
             if(/* client[i] received the whole response */) {
                 /* get response body */
                 /* get new URL from queue */
                 /* set new URL for client[i] request */
                 /* client[i].get(request) */
             }
     }

So, next trouble is how to check if client[i] has received the whole 
response and disconnected from server ("Connection: close" request 
header was used) or not. How to check this? And additionally, how to 
pass callback handlers into client (or response maybe?) object?
I have found nothing about this in documentation and in examples also.