From: Dave I. <dav...@en...> - 2006-11-28 16:02:44
|
I must be missing something. I've rerun my tests and they show the entire file being uploaded before my module is being called. This is in the case when forkcgis=0. When I set forkcgis=1, Webmin stops working completely. I just get a file permissions error showing up the in browser. I am using Webmin version 1.290 on RedHat EL3. Thanks Dave I -----Original Message----- From: web...@li... [mailto:web...@li...] On Behalf Of Jamie Cameron Sent: Monday, November 27, 2006 5:51 PM To: Webmin users list Subject: Re: [webmin-l] Restricting file upload size Hi Dave, Yes, for both forked and internally-executed Perl scripts it no longer reads the entire input into memory, at least not in miniserv.pl. - Jamie On 27/Nov/2006 12:58 Dave Isaacs wrote .. Thanks! Did you also fix the forkcgis path? We now have the requirement to upload 500MB files to the server, so instead of limiting the file size we need to ensure that the file is not uploaded into memory first. If I remember correctly, the forkcgis path in miniserv.pl would do just that, but was not working correctly? Forgive me if I am remembering this wrong. I am still working on dredging up all the details from my head. Thanks Dave I -----Original Message----- From: web...@li... [mailto:web...@li...] On Behalf Of Jamie Cameron Sent: Monday, November 27, 2006 3:52 PM To: Webmin users list Subject: Re: [webmin-l] Restricting file upload size Hi Dave, Yes, this has been fixed for a few versions now .. - Jamie On 27/Nov/2006 12:42 Dave Isaacs wrote .. Jamie, Almost a year ago I posted this question about restricting file upload size. At the time you admitted to some miniserv.pl limitations and said that you would take care of this. Have you? If yes, great! If not, any estimates on when this can get in? Thanks Dave I From: Jamie Cameron <jca...@we...> To: web...@li... Reply-To: web...@li... Date: Feb 10 2006 - 6:38pm On 11/Feb/2006 03:19 Dave Isaacs wrote .. > My experience shows that this does not work. > > I put a 1000000 limit in my call to ReadParseMime then attempted to upload > a > 1GB file. Using top, I watched the miniserv.pl process climb to about > 600MB > before crashing. ReadParseMime was never called because my module was > never > invoked. > > If I look at miniserv.pl, at around line 1740, I see > > $clen = $header{"content-length"}; > if ($method eq "POST" && $clen_read < $clen) { > # Still some more POST data to read > while(length($postinput) < $clen) { > $buf = &read_data($clen - length($postinput)); > if (!length($buf)) { > &http_error(500, "Failed to read ". > "POST request"); > } > $postinput .= $buf; > } > } > > This looks an awful lot like reading in the entire file upload. As a test, > I wrote the length($postinput) value to a log file (right before the call > to > read_data) and found that miniserv.pl was looping in an attempt to read > the > entire file upload. Hi Dave, You are absolutely correct .. Webmin really does the whole posted input into memory! Sorry, I totally forgot about that.. > Then I stumbled upon the forkcgis configuration setting, which appears > to > switch on a alternative method of invoking the webmin modules. This method > has miniserv.pl forwarding the file upload to the forked process as it > is > received. Unfortunately, this does not work either. Now when I upload > a > large file, something goes wrong and there is never a response. The log > messages I put in the miniserv loop shows that about 7500 bytes are read > in, > and then everything stops. Although this is better than crashing the > server, it is still not correct. I looked into this too, and found that Webmin is currently terminating the browser connection if the uploaded data is more than the set limit. Unfortunately, no browsers take kindly to this, and display an error message about the connection being closed. In the next release of Webmin, it will handle this better by reading all the data submitted by the browser, but not actually storing it in memory if the limit is exceeded. That is not quite ideal, but still better than the current situation. - Jamie |