#562 Maximum .ics file size for import.php

open
nobody
None
1
2014-07-06
2014-07-03
Anonymous
No

Seems that the import.php does not function for an .ics file > 1.4MB...or approximately 32,000 lines. Is there a way to increase this? My .ICS file is about 7MB and about 120,000 lines.

Thanks,
Nick

Related

Support Requests: #562

Discussion

  • How does it fail? It's possible that it's failing because you've hit 'max_execution_time' (normally defaults to 30 seconds) or else 'upload_max_filesize' (normally 2M). You can change these in php.ini.

    [I'm thinking of rewriting import.php so that it uses streams, which would avoid both these problems.]

     
  • John Beranek
    John Beranek
    2014-07-03

    Also 'post_max_size' as the PHP manual[1] states:

    post_max_size integer
    Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. If memory limit is enabled by your configure script, memory_limit also affects file uploading. Generally speaking, memory_limit should be larger than post_max_size. When an integer is used, the value is measured in bytes. Shorthand notation, as described in this FAQ, may also be used. If the size of post data is greater than post_max_size, the $_POST and $_FILES superglobals are empty. This can be tracked in various ways, e.g. by passing the $_GET variable to the script processing the data, i.e.

    , and then checking if $_GET['processed'] is set.

    [1] - http://www.php.net/manual/en/ini.core.php#ini.post-max-size

     
  • John Beranek
    John Beranek
    2014-07-03

    I guess potentially also 'memory_limit' ...

     

  • Anonymous
    2014-07-03

    I have tried the following inside php.ini

    upload_max_filesize = 10M
    post_max_size = 10M
    max_execution_time = 300
    memory_limit = 10M

    I have also tried to add the following lines to import.php:

    / Sets some INI Variables /
    ini_set("post_max_size", "10M");
    ini_set("upload_max_filesize", "10M");
    ini_set("memory_limit", "10M" );
    set_time_limit(300);

    As far as errors go, I don't see any at all... its just a blank output screen that just shows the header of the MBRS page, but no data is shown after that.

    Nick

     

  • Anonymous
    2014-07-03

    Turns out that my host provider at siteground will does not allow any changes to global settings... the one that I think is causing me the issue is max_execution_time as it is set to 45 seconds and I cannot change it.

    So I guess I am going to have to find a way to write my .ics files for import to no more than 1500 entries at a time... or wait until your "streams" version of import.php is built.

    Nick

     
  • Yes, in the short term I think your best bet is to split the file up into smaller chunks and import them one at a time.

    I'm looking at two ways of improving import.php. One is to import a file that has already been uploaded via FTP to the web server. The other is to allow a compressed file to be uploaded from the client and then get import.php to de-compress it. It's probably going to be a few weeks before I get a chance to do either of these. However the first one may be relatively easy for you to do by hacking import.php and hard-coding a filename.

    Campbell

     

    • Anonymous
      2014-07-04

      Anything that still performs the processing in the web server could still
      hit 'max_execution_time' though. :(

      If you have shell access to the web server it would be possible to make
      import.php run on the CLI I guess.

      John

      On 4 July 2014 10:08, Campbell Morrison cimorrison@users.sf.net wrote:

      Yes, in the short term I think your best bet is to split the file up into
      smaller chunks and import them one at a time.

      I'm looking at two ways of improving import.php. One is to import a file
      that has already been uploaded via FTP to the web server. The other is to
      allow a compressed file to be uploaded from the client and then get
      import.php to de-compress it. It's probably going to be a few weeks before
      I get a chance to do either of these. However the first one may be
      relatively easy for you to do by hacking import.php and hard-coding a
      filename.

      Campbell

      Status: open
      Group:
      Created: Thu Jul 03, 2014 05:49 AM UTC by Anonymous
      Last Updated: Thu Jul 03, 2014 02:47 PM UTC
      Owner: nobody

      Seems that the import.php does not function for an .ics file > 1.4MB...or
      approximately 32,000 lines. Is there a way to increase this? My .ICS file
      is about 7MB and about 120,000 lines.

      Thanks,
      Nick


      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/mrbs/support-requests/562/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

      --
      John Beranek - MRBS Developer http://mrbs.sourceforge.net/

      To generalise is to be an idiot.
      -- William Blake

       

      Related

      Support Requests: #562

  • John Beranek
    John Beranek
    2014-07-04

    Anything that still performs the processing in the web server could still hit 'max_execution_time' though. :(

    If you have shell access to the web server it would be possible to make import.php run on the CLI I guess.

     
  • Good point. Maybe I'll also need to do some performance analysis and optimisation of import.php. The PHP manual says that

    "Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real."

    Off hand, I can't think that there's much processing outside of stream operations and database queries to be done, so maybe import.php is amenable to some optimisation. Nick if your file is not confidential would you be able to mail it to me off-list so that I can use it as a test file? If it is confidential, don't worry, I can easily construct some test files.

     

  • Anonymous
    2014-07-04

    Its not confidential... now I just gotta figure out what email address I need to send the file to....

    :-)

    Nick

     
  • [email address snipped]

     
    Last edit: Campbell Morrison 2014-07-06

  • Anonymous
    2014-07-05

    I am still waiting for the sourceforge folks to re-enable my account so I can PM you the file.

    Nick

     

  • Anonymous
    2014-07-05

    I am still waiting for the sourceforge folks to re-enable my account so I can PM you the file.

    Nick

     

  • Anonymous
    2014-07-06

    send to you email from mine.... [email address snipped]

     
    Last edit: Campbell Morrison 2014-07-06
  • Thanks for the file. I tried it on my system and I think what's happening is that it is running out of memory and that's why you just get the blank screen. I've rewritten import.php so that it doesn't use so much memory and checked this into the trunk in Rev 2849. I then re-ran it on my system and successfully managed to import 7158 events. The other 1577 events were conflicts and couldn't be imported - and having checked the .ics file I can see that they are genuine conflicts. The script took ~130 seconds on my system using your test file already uploaded to the server using FTP.

    To get it to run I had to increase the size of the room_name field in the SQL table from 25 characters (the MRBS default) to 255 as you have some quite long room names. Together with this I had to set

    $maxlength['room.room_name']       = 255;
    

    and also set_time_limit() to something larger than 130.

    I'll have a look when I get more time at allowing compressed files to be uploaded as this should get round many cases where the file size is greater than the max upload size. I notice your 7MB file compressed to 482KB, so if that's a typical compression ratio for .ics files a 2MB file upload limit would allow files of up to about 25-30MB to be imported.

    Also when I get more time I'll have a look at adding some better indications of progress.

    Campbell

     


Anonymous


Cancel   Add attachments