[Soaplite-devel] huge memory consumption on transferring big files...

[Thorsten S. <tsc...@am...>]

Hello,

we use SOAP::Lite to transfer binary files as Base64 encoded strings
as a parameter like others within a normal Soap message. We are not
using Soap with attachments or else and yet don't plan to use it for
compatibility concerns. The Base64-encoding is handled in front of
SOAP::Lite for historical reasons, too, therefore SOAP::Lite really
does only see a very large, multi lined string.

The problem is that this approach scales pretty bad. Transferring a 40
MB binary file exceeds the amount of addressable RAM of the perl
process on 32 Bit systems. The HTTP-communication itself is not a
problem, but serialization and deserialization by SOAP::Lite seems to
be. I don't think that using more than 2 GB of memory for handling
maybe 100 MB of data is really necessary and tried to find my way
through the serializer and deserializer to look if I can find which
obviously blow up memory consumption. "envelope" in the serializer is
one method after which the used memory increased to several hundreds
of MB, the real big thing happens afterwards, though.

The main thing envelope seems to do is creating SOAP::Data-instances
for existing data. Looking at SOAP::Data::set_value it seems to me
it's copying data, but I'm not sure if it's always copying the data it
gets how it gets it or always dereferencing it and really producing
maybe unneeded copies.

My soap call to SOAP::lite looks the following:

  eval{$t = $soap->elrev_EndUpload( SOAP::Data->new('name'  => 'sessionid',
                                                    'type'  => 'string',
                                                    'value' => $sessionid),
                                    SOAP::Data->new('name'  => 'mandant',
                                                    'type'  => 'string',
                                                    'value' => $mandant),
                                    SOAP::Data->new('name'  => 'satzid',
                                                    'type'  => 'string',
                                                    'value' => $dsnummer),
                                    SOAP::Data->new('name'  => 'dateiinhalt',
                                                    'type'  => 'string',
                                                    'value' => $$datei_ref),
                                    SOAP::Data->new('name'  => 'einreichart',
                                                    'type'  => 'string',
                                                    'value' => $art),
                                    SOAP::Data->new('name'  => 'mailadresse',
                                                    'type'  => 'string',
                                                    'value' => $email));};

$$datei_ref holds the Base64 encoded file data.

Calling the method this way I thought that only references to objects
are handled in envelope and copying them doesn't really matter. But
if SOAP::Data::set_value would copy the contents of the references the
huge memory consumption could easily be explained because it seems
SOAP::Data-objects are created very often.

Am I right and set_value copies the content in the existing
SOAP::Data-objects? Do you see any other apparently reasons why the
memory consumption could be this high?

Thanks to all.

Mit freundlichen Grüßen,

Thorsten Schöning

-- 
Thorsten Schöning
AM-SoFT IT-Systeme - Hameln | Potsdam | Leipzig
 
Telefon: Potsdam: 0331-743881-0
E-Mail:  tsc...@am...
Web:     http://www.am-soft.de

AM-SoFT GmbH IT-Systeme, Konsumhof 1-5, 14482 Potsdam
Amtsgericht Potsdam HRB 21278 P, Geschäftsführer: Andreas Muchow