Re: [Soaplite-devel] huge memory consumption on transferring big files...

[Martin K. <mar...@fe...>]

Hi Thorsten,

SOAP::Lite indeed copies the request data several times - in the
envelope creation (the one you suspected), the serializer (it keeps both
the original data and the XML form in memory), the deserializer, and, as
far as I know, twice in the transport layer (one for converting the XML
string to bytes, and one for passing the data to LWP::UserAgent by
value). This makes up for at least 4 copies for sending a request.

Using base64 also increases memory usage by 1.5, so the factor of > 20
you're reporting seems quite likely.

This behavior, however, is unlikely to change in the future: Passing
values by reference (instead of a pass-by-value, which means copying)
would require API changes and possibly break existing applications.

Eliminating all (really) unnecessary copies would probably reduce memory
usage by a factor of between 2 and 10. Unfortunately, even this is
unlikely to happen: SOAP::Lite's test suite is far from being perfect,
and memory optimizations (and testing they don't break anything) would
require quite some effort - and it's not even sure memory usage would
drop below a level acceptable for you.

The best approach would be to avoid the memory usage problem altogether
and stream the request to the server.

Unfortunately, SOAP::Lite doesn't have any means for streaming requests.
To my knowledge, there's also no other SOAP library in perl which has
direct support for streaming request, so you may need to roll your own.

Martin

Am Donnerstag, den 15.07.2010, 13:06 +0200 schrieb Thorsten Schöning:
> Hello,
> 
> we use SOAP::Lite to transfer binary files as Base64 encoded strings
> as a parameter like others within a normal Soap message. We are not
> using Soap with attachments or else and yet don't plan to use it for
> compatibility concerns. The Base64-encoding is handled in front of
> SOAP::Lite for historical reasons, too, therefore SOAP::Lite really
> does only see a very large, multi lined string.
> 
> The problem is that this approach scales pretty bad. Transferring a 40
> MB binary file exceeds the amount of addressable RAM of the perl
> process on 32 Bit systems. The HTTP-communication itself is not a
> problem, but serialization and deserialization by SOAP::Lite seems to
> be. I don't think that using more than 2 GB of memory for handling
> maybe 100 MB of data is really necessary and tried to find my way
> through the serializer and deserializer to look if I can find which
> obviously blow up memory consumption. "envelope" in the serializer is
> one method after which the used memory increased to several hundreds
> of MB, the real big thing happens afterwards, though.
> 
> The main thing envelope seems to do is creating SOAP::Data-instances
> for existing data. Looking at SOAP::Data::set_value it seems to me
> it's copying data, but I'm not sure if it's always copying the data it
> gets how it gets it or always dereferencing it and really producing
> maybe unneeded copies.
> 
> My soap call to SOAP::lite looks the following:
> 
>   eval{$t = $soap->elrev_EndUpload( SOAP::Data->new('name'  => 'sessionid',
>                                                     'type'  => 'string',
>                                                     'value' => $sessionid),
>                                     SOAP::Data->new('name'  => 'mandant',
>                                                     'type'  => 'string',
>                                                     'value' => $mandant),
>                                     SOAP::Data->new('name'  => 'satzid',
>                                                     'type'  => 'string',
>                                                     'value' => $dsnummer),
>                                     SOAP::Data->new('name'  => 'dateiinhalt',
>                                                     'type'  => 'string',
>                                                     'value' => $$datei_ref),
>                                     SOAP::Data->new('name'  => 'einreichart',
>                                                     'type'  => 'string',
>                                                     'value' => $art),
>                                     SOAP::Data->new('name'  => 'mailadresse',
>                                                     'type'  => 'string',
>                                                     'value' => $email));};
> 
> $$datei_ref holds the Base64 encoded file data.
> 
> Calling the method this way I thought that only references to objects
> are handled in envelope and copying them doesn't really matter. But
> if SOAP::Data::set_value would copy the contents of the references the
> huge memory consumption could easily be explained because it seems
> SOAP::Data-objects are created very often.
> 
> Am I right and set_value copies the content in the existing
> SOAP::Data-objects? Do you see any other apparently reasons why the
> memory consumption could be this high?
> 
> Thanks to all.
> 
> Mit freundlichen Grüßen,
> 
> Thorsten Schöning
>