Re: [chiPHPug-discuss] Very Large Uploads Problem

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

I didn't catch Jason's ini settings but we upload CAD files all the time up to 800MB w/ Apache/Linux.
Did you increase form processing time? 
max_input_time 3600 3600 
I would also try and just copy a 1GB file to tmp as a saity check.

________________________________
From: derby <der...@gm...>
To: Discussions of PHP-related topics among members of the Chicago PHP User's Group. <chi...@li...>
Sent: Thursday, November 20, 2008 12:30:40 AM
Subject: Re: [chiPHPug-discuss] Very Large Uploads Problem

I am also in need of uploading large files via PHP.  I tried a few  
settings in php.ini to increase time and upload size, but the most I  
could upload was about 70 MB worth a 200MB file.

I saw some example scripts that integrate a ftp client into a PHP  
script, but I haven't tested it and I am not sure if this would even  
work since I would still be running the PHP on top of the ftp portion.

derby

ps thanks Jason for all the details

On Nov 19, 2008, at 13:03, Peter Snyder wrote:

> Hi Wilfried,
>
> I agree that HTTP probably (ok, absolutely) isn't the most technically
> sound way of doing this, but the client needs a dead-simple way for
> his clients to submit large audio files for editing.  So, sadly, HTTP
> is the best tool for the job.
>
> I've tried two flash applets (SWFuploader and FancyUpload), and both
> work well except they similarly stop sending data after 40 or 50mb.  I
> think this won't get around the problem since they're still submitting
> data via HTTP POST anyway.  Can you, or anyone else for that matter,
> recommend a flash or java solution that you've had good luck with?
> Anyone have any idea what sites like MegaUpload or YouTube or
> othersites that allow users to upload mega amounts of data use?
>
> Thanks for your input Wilfred.  Maybe you're right and its time to go
> back to the client and say "are you sure this the way you want to
> go..." :-)
>
> Pete
>
>
> On Nov 19, 2008, at 12:50 PM, Wilfried Schobeiri wrote:
>
>> I sent this to the list but it didnt let me:
>>
>> You know, part of me wants to say if you're trying to move gigs of
>> data around via web form, youre doing something wrong.
>>
>> Think:
>> * tmp or /var space could run out, because its getting written to /
>> var/
>> tmp then rebuilt to /tmp
>> * PHP could run out of memory rebuilding the 1MB /var/tmp chunks (or
>> so I read)
>> * You're tying up an httpd instance for who knows how long
>> * No Fault Tolerance... process manager might kill the process or the
>> client might time out.
>>
>> Plus, HTTP wasn't designed for large file transfers.
>>
>> Then again, if its hosted locally, some of this may not apply.
>>
>> Try a Java or Flash applet instead, perhaps? Yes, i know... *groan*.
>> On Nov 19, 2008, at 12:47 , Peter Snyder wrote:
>>
>>> Thank you Jason and Arlo for your suggestions.
>>>
>>> I checked to make sure that the php settings were reflected in
>>> phpinfo() and they were, so I don't believe that could be the
>>> problem.
>>>
>>> I also tried rewriting the sizes in .htaccess as 1000000000 instead
>>> of
>>> 1G, and tried adding to session.gc_maxlifetime to 14400.  The only
>>> difference I've from doing this is that Firefox gives me a
>>> "Connection
>>> Interrupted' error page instead of a "timed out error message".
>>>
>>> Both servers I've tested on are linux / apache ones, so thankfully I
>>> don't need to worry about IIS issues.  And there is no MAX_FILE_SIZE
>>> form field, nor
>>>
>>> The only remaining issue I can think of might be with the
>>> LimitRequestBody setting.  I've emailed my hosting company to ask
>>> them
>>> if they have a limit enforced.  In the meantime though, is there
>>> someway I can test this on my own?
>>>
>>> Again, thank you both for your comments.  They are greatly
>>> appreciated!
>>>
>>> Pete
>>>
>>>
>>> On Nov 19, 2008, at 8:12 AM, Jason Rexilius wrote:
>>>
>>>> A couple of feedback items from the google (off the top of my  
>>>> head I
>>>> cant see anything obvious):
>>>>
>>>> 1) are you running newer PHP5? from manual:
>>>> "G (for Gigabytes; available since PHP 5.1.0)"
>>>>
>>>>
>>>> 2) IIS has som limits if you are on Windows (why, oh why):
>>>> IIS7 has a upload limit of 30000000 (about 30mb)
>>>> You can change this with the command (for 250mb):
>>>> c:\windows\system32\inetsrv\appcmd set config -
>>>> section:requestFiltering
>>>> -requestLimits.maxAllowedContentLength:262144000
>>>> Or manual define it in:
>>>> %windir%\system32\inetsrv\config\applicationhost.config
>>>> Add this rule before the </requestFiltering> tag:
>>>> <requestLimits maxAllowedContentLength ="262144000" />
>>>>
>>>>
>>>> 3) There has been reference to session.gc_maxlifetime default value
>>>> 1440
>>>> (24min) causing GC to wack session duriung upload and cause
>>>> problems.
>>>> Not sure on that one..
>>>>
>>>>
>>>> 4) Also reported issues of MAX_FILE_SIZE hidden form field, and
>>>> zlib.output_compression = On in Apache..
>>>>
>>>>
>>>> 5) Check Apache LimitRequestBody (max size in bytes)
>>>>
>>>>
>>>> 6) Obscure but possible:
>>>> "Because some systems send the packets with the headers last (or  
>>>> the
>>>> header packet may be received after some of the data packets),
>>>> firewalls
>>>> can't filter this traffic based on destination port and address.
>>>> Many
>>>> firewalls (including iptables) have to be configured to allow
>>>> fragments
>>>> separately from standard traffic. Unfortunately, it's an all-or-
>>>> nothing
>>>> thing in these cases, and exploits based on packet fragmentation
>>>> have
>>>> been a problem in the past (teardrop, boink, etc.). Note that ICMP
>>>> may
>>>> be used to notify the host (your server) of oncoming fragmentation,
>>>> so
>>>> you may need to allow ICMP traffic as well."
>>>>
>>>> The iptables rules for this are as follows:
>>>> # allow all fragments
>>>> -A INPUT -f -j ACCEPT
>>>> # allow icmp traffic
>>>> -A INPUT -p icmp -j ACCEPT
>>>>
>>>>
>>>> Most of the above thanks to manual page:
>>>>
>>>> http://us3.php.net/features.file-upload
>>>>
>>>>
>>>>
>>>> Peter Snyder wrote:
>>>>> Hello Everyone,
>>>>>
>>>>> I'm having a problem creating an app for a client who will need to
>>>>> upload very large files, around 1GB.  I'm running into problems
>>>>> though
>>>>> with the upload.  In an effort to rule out problems, I've boosted
>>>>> all
>>>>> the normal suspects to absurd levels (see below), but am still
>>>>> getting
>>>>> problems.  If I just watch the uploads in the temp dir I can see
>>>>> that
>>>>> the server is receiving the uploads correctly, but for some reason
>>>>> the
>>>>> upload stops at around 40 - 50 MB each time.  The browser will
>>>>> putter
>>>>> on for a while and then say that the server has timed out.
>>>>>
>>>>> Does anyone have any other ideas for possible problems?  I've  
>>>>> tried
>>>>> this on two different servers (clients and my staging one),  
>>>>> with no
>>>>> luck.  Any hints anyone could provide would be greatly  
>>>>> appreciated.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Sincerely,
>>>>> Pete Snyder
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----
>>>>> .htaccess file
>>>>> php_value memory_limit 2G
>>>>> php_value post_max_size 2G
>>>>> php_value upload_max_filesize 2G
>>>>> php_value max_input_time 144000
>>>>> php_value max_execution_time 144000
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------ 
>>>>> -------
>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>>> challenge
>>>>> Build the coolest Linux based applications with Moblin SDK & win
>>>>> great prizes
>>>>> Grand prize is a trip for two to an Open Source event anywhere in
>>>>> the world
>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>> _______________________________________________
>>>>> chiPHPug-discuss mailing list
>>>>> chi...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss
>>>>
>>>> ------------------------------------------------------------------- 
>>>> ------
>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>>> challenge
>>>> Build the coolest Linux based applications with Moblin SDK & win
>>>> great prizes
>>>> Grand prize is a trip for two to an Open Source event anywhere in
>>>> the world
>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>> _______________________________________________
>>>> chiPHPug-discuss mailing list
>>>> chi...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss
>>>
>>>
>>> -------------------------------------------------------------------- 
>>> -----
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>>> challenge
>>> Build the coolest Linux based applications with Moblin SDK & win
>>> great prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in
>>> the world
>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>> _______________________________________________
>>> chiPHPug-discuss mailing list
>>> chi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss
>>
>>
>> --------------------------------------------------------------------- 
>> ----
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
>> great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in
>> the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> chiPHPug-discuss mailing list
>> chi...@li...
>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss
>
>
> ---------------------------------------------------------------------- 
> ---
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> chiPHPug-discuss mailing list
> chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
chiPHPug-discuss mailing list
chi...@li...
https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss