what size data can be passed thrg web service

Help
Rahul
2010-09-13
2013-06-06
  • Rahul

    Rahul - 2010-09-13

    Hello Everyone,
    I am using the NuSOAP for the first time, I have created one web service using NuSOAP (PHP), the server sends large data to client, it is working fine for data upto 20MB, beyond that I am unable to send more data, I want to know what is the size of data that can be send through the web service, is it possible to send 40MB data?, I have relatively increased the size of  post_max_size php.ini directive and connection time out value, any help on this will be really helpful for me

    Thanks,
    -Rahul

     
  • Pavel Bazanov

    Pavel Bazanov - 2010-09-13

    Hello, I don't know the exact restrictions, but I believe that's not a good design decision to send such big chunks of data at once. Try to send data in smaller chunks. I send hundreds of megabytes with chunks a few hundred kilobytes each.

     
  • Aaron Kreider

    Aaron Kreider - 2010-09-13

    I think php's memory limit is setting your maximum.  It can take 2-3 times more memory than the size of what you are sending (you might be hitting the 64 mb limit which is often used by php).  So you can increase the php memory limit in php.ini. 

    There might be a nusoap debug setting that you can turn off to improve memory usage. I remember doing this once (but don't seem to be doing it currently).

    You can comment out nusoap classes that you aren't using to save small amounts of memory.  I commented out the xmlschema and wsdl classes to reduce memory usage from 2mb to 1.2mb.

     
  • Aaron Kreider

    Aaron Kreider - 2010-09-13

    For sending large amounts of data, you might want to compress it.  I serialize the data, gzcompress, and convert to base64_encode.  And then base64_decode, gzuncompress, and unserialize it.  This will make NuSoap's job easier and might save some memory.

     
  • Rahul

    Rahul - 2010-09-14

    Thank You guyz  for your replies,
    @pashkatlt  : Can you please elaborate how you divided data into chunks and manage to send it to client side? I also tried this, basically my data is few records from database,  I created chunk of those data by using limit on queries,  here our requirement is that the client should receive all data at one go or at least client should make single request only , not  multiple requests for each chunk.

    @aaronlk  I have already set the memory limit to 512 in my script, also have set the debug level to 0 to avoid debug info being getting sent to client to save the memory and  have removed the WSDL classes for caching which I am not using in my script.
    will try to  serialize the data, gzcompress, and convert to base64_encode to check out if this help

    Thanks
    -Rahul

     
  • Aaron Kreider

    Aaron Kreider - 2010-09-14

    What is the script's php memory usage?  You could test that at various points in the script (write it out to a text file or use another debug method).

     
  • Pavel Bazanov

    Pavel Bazanov - 2010-09-14

    rahulpalake,
    I have to send thousands of data rows from the database to synchronize tables in the remote and local databases, so I read and send, for example, 1000 rows each time.

    First of all I request max timestamp value from the remote server, then I read max timestamp value from the local database and then I simple execute the following query in a cycle sending 1000 rows through webservice on each iteration of the cycle.

        SELECT * FROM pricelist
        WHERE TS > MaxTsOnRemoteServer AND DateAdded <= MaxLocalTs AND ID > LastReadID
        ORDER BY ID
        LIMIT 1000;

    This query allows "resuming" if connection to the webservice fails.

     
  • Rahul

    Rahul - 2010-09-15

    Thanks for your replies,
    can you guyz please tell me  how to gzcompress server response  and convert it to base64_encode before sending it to client,  I dont know exactly where to apply these,
    -rahulpalake

     
  • Aaron Kreider

    Aaron Kreider - 2010-09-15

    Normally I return an array of data.
    So before I return the array I do

    function actn_SerializeCompressBase64Encode ($aArray)
    {
    $sSerialize=serialize ($aArray);
    $sCompress=gzcompress ($sSerialize);
    $sBase64=base64_encode ($sCompress); 
    return $sBase64;
    }

     
  • Rahul

    Rahul - 2010-09-15

    Hi aaronlk,
    I am returning  complextype struct comprising arrays of the format

    $server->wsdl->addComplexType(                                                                          
            'Product',                                                                                           
            'complexType',                                                                                      
            'struct',                                                                                           
            'all',                                                                                              
            '',                                                                                                 
            array(                                                                                              
                'product_id'             =>  array('name'=>'product_id','type'=>'xsd:string'),                                  
                'product_name'           =>  array('name'=>'product_name','type'=>'xsd:string'),    
    'short_desc'           =>  array('name'=>'short_desc','type'=>'xsd:string'),                                  
    'long_desc'         =>  array('name'=>'long_desc','type'=>'xsd:string'),
    'category_desc'          =>  array('name'=>'category_desc','type'=>'xsd:string'),                                  
    'merchant_name'          =>  array('name'=>'merchant_name','type'=>'xsd:string'),                                  
    'brand_name'         =>  array('name'=>'brand_name','type'=>'xsd:string'),                                  
    'model'         =>  array('name'=>'model','type'=>'xsd:string'),   
    'prod_img_url'           =>  array('name'=>'prod_img_url','type'=>'xsd:string'),                                  
    'prod_url'           =>  array('name'=>'prod_url','type'=>'xsd:string'),                                  
    'buy_url'                =>  array('name'=>'buy_url','type'=>'xsd:double'),                                  
    'price'          =>  array('name'=>'price','type'=>'xsd:double'),                                  
    'final_price'          =>  array('name'=>'final_price','type'=>'xsd:double'),                                  
    'dlu'          =>  array('name'=>'dlu','type'=>'xsd:string'), 
                                   
                                  
            )

    I guess while registering the method we also specifying it's return type (in my case the above complexType) so if I serialize, compress and encode the return value,  will it  match the return type?, cause I got nothing in response when I did this,

    pasting the resposne:
    HTTP/1.1 200 OK

    Date: Wed, 15 Sep 2010 19:27:29 GMT

    Server: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.12

    X-Powered-By: PHP/5.2.12

    X-SOAP-Server: NuSOAP/0.9.5 (1.123)

    Content-Length: 605

    Connection: close

    Content-Type: text/xml; charset=ISO-8859-1

    <?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:tns="urn:feedmanagementwebservice"><SOAP-ENV:Body><ns1:GetProductsResponse xmlns:ns1="feedmanagementwebservice"><return xsi:type="SOAP-ENC:Array" SOAP-ENC:arrayType="tns:Product"></return></ns1:GetProductsResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

    am I missing something here?

    -Thanks
    rahulpalake

     
  • Aaron Kreider

    Aaron Kreider - 2010-09-15

    What I'm suggesting might not work for you.  I return a single array.  You could change your code to return a single array if other people are not relying upon your WSDL.

    You simply turn your existing data structure into an array before sending it (from the server), and then do the reverse operation on the client side.

    You should also look into the php memory footprint as I suggested earlier:
    http://php.net/manual/en/function.memory-get-usage.php

     
  • Rahul

    Rahul - 2010-09-16

    Thank You guyz for your help

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks