Re: [cpp-netlib-devel] FW: body copies on http::client::post (was: RE: Poll: Next Features to Implement)

[Nelson, E. - 2 <eri...@ba...>]

>Dean Michael Berris wrote on Tuesday, October 19, 2010 11:26 AM

>>Erik Nelson wrote:
>> The copying is pretty key... if I'm sending a 50GB message, I really
>> can't afford to have any copies besides the one that is the source.
>>
 
> You can do without copies if you create a message with the data in the
> body already. You also don't need to use the post interface that
> doesn't include the body and content-type.
> 
>> The design itself ought not impose unnecessary runtime
>> overhead in either the time or memory usage domains.  If I
>> understand it correctly, the asio lib doesn't require the message
>> to be marshaled in memory as a contiguous string, and neither
>> should netlib, in my opinion.
>>
 
> Actually, Asio does. It requires that you give it a set of buffers --
> and std::string objects qualify as good buffers -- to write out to the
> sockets.

The asio buffers can't be strings, right?  As I understand it, asio
buffers are simply pointers into contiguous memory regions, and
std::string is only one of many types that hold contiguous memory
regions (and is in no way preferred).
http://bit.ly/dubjW5


std::strings are actually terrible buffers if you're manipulating
binary objects, at least in our project.  If we have a big memory
region that's full of POD and want to send it across the wire,
making a std::string out of it means *yet another* copy.  That's why
the asio buffer interface is in no way std::string-centric.

> Also, if you have an std::string implementation that implements
> copy-on-write semantics, then you don't pay for a 50GB copy because a
> copy is pretty much just a single pointer copy anyway. Most, if not
> all implementations of std::string do the COW optimization precisely
> because of the overhead of copying strings around even if you don't
> modify them.
> 

I'm unconvinced that COW strings can be depended on as a central
library feature- a quick Google seems to indicate otherwise.

http://bit.ly/hiqqU
http://bit.ly/cqpKVN

I'm not sure I fully buy into the 'the-optimizer-will-cover-it-up'
argument, either.  The optimizer covers up very little in debug
mode and a program that can't be well exercised in debug mode is
hard to... debug.  The speed will be slower, but having the
memory footprint explode just due to the networking library is
a bitter pill to swallow.

> At any rate, the next version will have a message type that supports
> ranges for bodies. This allows for a means of making the body of a
> request a Boost.Range compatible range -- so an input_iterator range
> would work fine in that version. In that case data will be pulled from
> input iterator ranges, and cpp-netlib can conserve the buffers as they
> come.
> 

That sounds great!  When you say that cpp-netlib will 'conserve' the
ranges/buffers, what does that mean?

Erik

----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. 
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. 
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. 

References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: 
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.