Menu
▾
▴
Re: [chiPHPug-discuss] Very Large Uploads Problem
|
From: Peter S. <sn...@gm...> - 2008-11-19 19:03:41
|
Hi Wilfried, I agree that HTTP probably (ok, absolutely) isn't the most technically sound way of doing this, but the client needs a dead-simple way for his clients to submit large audio files for editing. So, sadly, HTTP is the best tool for the job. I've tried two flash applets (SWFuploader and FancyUpload), and both work well except they similarly stop sending data after 40 or 50mb. I think this won't get around the problem since they're still submitting data via HTTP POST anyway. Can you, or anyone else for that matter, recommend a flash or java solution that you've had good luck with? Anyone have any idea what sites like MegaUpload or YouTube or othersites that allow users to upload mega amounts of data use? Thanks for your input Wilfred. Maybe you're right and its time to go back to the client and say "are you sure this the way you want to go..." :-) Pete On Nov 19, 2008, at 12:50 PM, Wilfried Schobeiri wrote: > I sent this to the list but it didnt let me: > > You know, part of me wants to say if you're trying to move gigs of > data around via web form, youre doing something wrong. > > Think: > * tmp or /var space could run out, because its getting written to / > var/ > tmp then rebuilt to /tmp > * PHP could run out of memory rebuilding the 1MB /var/tmp chunks (or > so I read) > * You're tying up an httpd instance for who knows how long > * No Fault Tolerance... process manager might kill the process or the > client might time out. > > Plus, HTTP wasn't designed for large file transfers. > > Then again, if its hosted locally, some of this may not apply. > > Try a Java or Flash applet instead, perhaps? Yes, i know... *groan*. > On Nov 19, 2008, at 12:47 , Peter Snyder wrote: > >> Thank you Jason and Arlo for your suggestions. >> >> I checked to make sure that the php settings were reflected in >> phpinfo() and they were, so I don't believe that could be the >> problem. >> >> I also tried rewriting the sizes in .htaccess as 1000000000 instead >> of >> 1G, and tried adding to session.gc_maxlifetime to 14400. The only >> difference I've from doing this is that Firefox gives me a >> "Connection >> Interrupted' error page instead of a "timed out error message". >> >> Both servers I've tested on are linux / apache ones, so thankfully I >> don't need to worry about IIS issues. And there is no MAX_FILE_SIZE >> form field, nor >> >> The only remaining issue I can think of might be with the >> LimitRequestBody setting. I've emailed my hosting company to ask >> them >> if they have a limit enforced. In the meantime though, is there >> someway I can test this on my own? >> >> Again, thank you both for your comments. They are greatly >> appreciated! >> >> Pete >> >> >> On Nov 19, 2008, at 8:12 AM, Jason Rexilius wrote: >> >>> A couple of feedback items from the google (off the top of my head I >>> cant see anything obvious): >>> >>> 1) are you running newer PHP5? from manual: >>> "G (for Gigabytes; available since PHP 5.1.0)" >>> >>> >>> 2) IIS has som limits if you are on Windows (why, oh why): >>> IIS7 has a upload limit of 30000000 (about 30mb) >>> You can change this with the command (for 250mb): >>> c:\windows\system32\inetsrv\appcmd set config - >>> section:requestFiltering >>> -requestLimits.maxAllowedContentLength:262144000 >>> Or manual define it in: >>> %windir%\system32\inetsrv\config\applicationhost.config >>> Add this rule before the </requestFiltering> tag: >>> <requestLimits maxAllowedContentLength ="262144000" /> >>> >>> >>> 3) There has been reference to session.gc_maxlifetime default value >>> 1440 >>> (24min) causing GC to wack session duriung upload and cause >>> problems. >>> Not sure on that one.. >>> >>> >>> 4) Also reported issues of MAX_FILE_SIZE hidden form field, and >>> zlib.output_compression = On in Apache.. >>> >>> >>> 5) Check Apache LimitRequestBody (max size in bytes) >>> >>> >>> 6) Obscure but possible: >>> "Because some systems send the packets with the headers last (or the >>> header packet may be received after some of the data packets), >>> firewalls >>> can't filter this traffic based on destination port and address. >>> Many >>> firewalls (including iptables) have to be configured to allow >>> fragments >>> separately from standard traffic. Unfortunately, it's an all-or- >>> nothing >>> thing in these cases, and exploits based on packet fragmentation >>> have >>> been a problem in the past (teardrop, boink, etc.). Note that ICMP >>> may >>> be used to notify the host (your server) of oncoming fragmentation, >>> so >>> you may need to allow ICMP traffic as well." >>> >>> The iptables rules for this are as follows: >>> # allow all fragments >>> -A INPUT -f -j ACCEPT >>> # allow icmp traffic >>> -A INPUT -p icmp -j ACCEPT >>> >>> >>> Most of the above thanks to manual page: >>> >>> http://us3.php.net/features.file-upload >>> >>> >>> >>> Peter Snyder wrote: >>>> Hello Everyone, >>>> >>>> I'm having a problem creating an app for a client who will need to >>>> upload very large files, around 1GB. I'm running into problems >>>> though >>>> with the upload. In an effort to rule out problems, I've boosted >>>> all >>>> the normal suspects to absurd levels (see below), but am still >>>> getting >>>> problems. If I just watch the uploads in the temp dir I can see >>>> that >>>> the server is receiving the uploads correctly, but for some reason >>>> the >>>> upload stops at around 40 - 50 MB each time. The browser will >>>> putter >>>> on for a while and then say that the server has timed out. >>>> >>>> Does anyone have any other ideas for possible problems? I've tried >>>> this on two different servers (clients and my staging one), with no >>>> luck. Any hints anyone could provide would be greatly appreciated. >>>> >>>> Thanks! >>>> >>>> Sincerely, >>>> Pete Snyder >>>> >>>> >>>> >>>> >>>> ----- >>>> .htaccess file >>>> php_value memory_limit 2G >>>> php_value post_max_size 2G >>>> php_value upload_max_filesize 2G >>>> php_value max_input_time 144000 >>>> php_value max_execution_time 144000 >>>> >>>> >>>> >>>> ------------------------------------------------------------------------- >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>>> challenge >>>> Build the coolest Linux based applications with Moblin SDK & win >>>> great prizes >>>> Grand prize is a trip for two to an Open Source event anywhere in >>>> the world >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>> _______________________________________________ >>>> chiPHPug-discuss mailing list >>>> chi...@li... >>>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss >>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>> challenge >>> Build the coolest Linux based applications with Moblin SDK & win >>> great prizes >>> Grand prize is a trip for two to an Open Source event anywhere in >>> the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> chiPHPug-discuss mailing list >>> chi...@li... >>> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss >> >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> Build the coolest Linux based applications with Moblin SDK & win >> great prizes >> Grand prize is a trip for two to an Open Source event anywhere in >> the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> chiPHPug-discuss mailing list >> chi...@li... >> https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win > great prizes > Grand prize is a trip for two to an Open Source event anywhere in > the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > chiPHPug-discuss mailing list > chi...@li... > https://lists.sourceforge.net/lists/listinfo/chiphpug-discuss |
