Re: [cpp-netlib-devel] FW: body copies on http::client::post

[Nelson, E. - 2 <eri...@ba...>]

>Dean Michael Berris wrote on Wednesday, October 20, 2010 8:40 AM

>> Erik Nelson wrote:
>> std::strings are actually terrible buffers if you're manipulating
>> binary objects, at least in our project.
> 
> Hmmm... why? Binary buffers are just thunks of memory, and you can
> always get the .data() of std::string objects.
> 

Think about what you'd have to do to store and manipulate an array
of doubles in a string, referencing it through the .data() member.
vector<double> seems like a much more natural fit

>> If we have a big memory
>> region that's full of POD and want to send it across the wire,
>> making a std::string out of it means *yet another* copy.  That's why
>> the asio buffer interface is in no way std::string-centric.
> 
> Sure, but Asio requires that everything you want to send *is* in
> memory. Whether it's a string or just a range, well as far as Asio is
> concerned it'd have to be in memory to be sent. cpp-netlib is built to
> make simple things easy really. ;)

It's fine to have it in memory... just not copies of it.
Simple things *should* be easy.  I'm not sure that means there needs
to be multiple copies made of the payload.

> 
> If you really need zero-copy messages, that would require that you
> pass in pointers to the data, and make sure that they're "live" when
> cpp-netlib starts requiring the data.

That's how asio works, right?  The buffers are pointers, and the app
needs to make sure they have not been invalidated;

> 
> You have two choices here:
> 
> 1) Implement it yourself, use the concepts and make a "better" message
> type, and send a pull request. :)
> 
> 2) Wait until I (or someone else) gets around to fixing that part of
> the implementation. :)
> 

Understood, just wanted to throw the issue into the design discussion.

>>
>> That sounds great!  When you say that cpp-netlib will 'conserve' the
>> ranges/buffers, what does that mean?
>>
 
> That means, only when data is going to be sent will data be
> "linearized" to a fixed-size buffer that gets re-used every time a
> write is going to be attempted. Asio has the async_read and
> async_write functions that take non-mutable buffers, and at this time
> ranges don't count as buffers.
> 
> This is going to be helpful in the case when files are going to be
> served, and the possibility of having mmap'ed files/buffers can be
> expressed as a range of iterators. The data can even be non-linear, it
> can be a range of joined iterators, it can be a range of function
> input iterators, but the data will be linearized still to a fixed-size
> buffer for Asio's consumption.
> 

Does asio require linearized data?

> Until Asio has a way of dealing with Boost.Ranges on its own, the
> linearization will have to be done by cpp-netlib. At the worst case,
> there will be a fixed overhead which will be caused by a buffer for
> each connection.
>

I think we might just disagree on this point, but, in my view, this
linearization is a core problem since it will cause a complete
memory copy of anything that is sent (unless I'm missing something).
That's fine if you're serving up 1 MB of something, but a problem
if you're serving up 50 GB of something.  This design will preclude
a typical Windows 32-bit program from serving files larger than
1 or 2 GB, right?

Is Boost.Range on the roadmap for asio?

Erik



----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. 
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. 
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. 

References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: 
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.