From: qjmann <sou...@gm...> - 2010-11-13 08:42:26
|
> > On Sat, Nov 13, 2010 at 3:32 AM, qjmann<sou...@gm... <mailto:sou...@gm...>> wrote: > > Documentation contains example HTTP client, but I see it works in blocking > > mode: > > > > http_client::response response = client.get(request) > > > > - this call will stop program execution while waiting for server's response. > > Not if http_client is a typedef like: > > typedef boost::network::http::basic_client< > boost::network::http::tags::http_async_8bit_udp_resolve > , 1, 0> http_client; > > > Is it possible to perform multiple concurrent requests with cpp_netlib > > without spawning multiple threads, as it can be done with ASIO asynchronous > > sockets? > > Yes, use the asynchronous tags for the http_client. This will use > futures underneath. > > In a later release there will be a change in the interface for an > asynchronous HTTP client that will take in a function to handle > incoming data from a request -- most probably will follow the same > interface that the async HTTP server in 0.8-devel will, I still > haven't decided yet. > > > > > 1) Create single io_service object > > 2) Create and initialize client objects set (all attached to io_service > > object just created) > > 3) Call io_service.poll() > > 4) Check client objects for responses received > > 5) If some responses has not been received yet, then goto step 3 > > > > No need for polling really, a callback mechanism or the current > futures-based implementation should suffice for most cases. > > HTH > > -- > Dean Michael Berris > deanberris.com > Thank's a lot, I used asynchronous tags, and now "client.get(request)" call is not blocking progpam's execution. In my application there will be a huge queue of URLs (from different hosts, it is just a crawler) and very limited set of requester objects (app must handle ~100 concurrent connections). So I want to implement a loop: while(/* queue has urls */) { for(/* loop through client objects set */) if(/* client[i] received the whole response */) { /* get response body */ /* get new URL from queue */ /* set new URL for client[i] request */ /* client[i].get(request) */ } } So, next trouble is how to check if client[i] has received the whole response and disconnected from server ("Connection: close" request header was used) or not. How to check this? And additionally, how to pass callback handlers into client (or response maybe?) object? I have found nothing about this in documentation and in examples also. |
From: Dean M. B. <mik...@gm...> - 2010-11-14 06:51:13
|
On Sat, Nov 13, 2010 at 4:42 PM, qjmann <sou...@gm...> wrote: > > Thank's a lot, I used asynchronous tags, and now "client.get(request)" > call is not blocking progpam's execution. > In my application there will be a huge queue of URLs (from different > hosts, it is just a crawler) and very limited set of requester objects > (app must handle ~100 concurrent connections). 100 concurrent connections is easy to handle. But not with the current implementation of the HTTP client. > > So, next trouble is how to check if client[i] has received the whole > response and disconnected from server ("Connection: close" request > header was used) or not. How to check this? You cannot do that yet now. I might implement a wrapper or a simple check on the response objects to see whether the body part is done. It should be trivial to do but I'm concentrating on the server side. Maybe others on the list might want to implement this for the asynchronous basic_response<Tag> overload where Tag is asynchronous? *hint*. :) > And additionally, how to > pass callback handlers into client (or response maybe?) object? > I have found nothing about this in documentation and in examples also. > These are not possible yet. As I've said earlier, this will be made possible in a later release, potentially 0.9, which will come after 0.8 (scheduled for release next week). HTH -- Dean Michael Berris deanberris.com |
From: Dean M. B. <mik...@gm...> - 2010-11-16 08:19:12
|
On Sun, Nov 14, 2010 at 2:50 PM, Dean Michael Berris <mik...@gm...> wrote: > On Sat, Nov 13, 2010 at 4:42 PM, qjmann <sou...@gm...> wrote: >> >> So, next trouble is how to check if client[i] has received the whole >> response and disconnected from server ("Connection: close" request >> header was used) or not. How to check this? > > You cannot do that yet now. > And with 0.8-beta, you can do this now. :) I just implemented a wrapper/checker for checking whether a response object is ready for processing. You can then loop through each response object with the 'ready' wrapper: if (ready(response)) { /* the response is ready here */ } else { /* the response is not ready yet */ } > > These are not possible yet. As I've said earlier, this will be made > possible in a later release, potentially 0.9, which will come after > 0.8 (scheduled for release next week). > I thought it wasn't easy to do, and found myself wanting something like this too, so I just did it. 0.8 should have it now, as much as 0.8-beta already has it now. -- Dean Michael Berris deanberris.com |