From: Jamie C. <jca...@we...> - 2006-11-27 22:51:37
|
Hi Dave,<br />Yes, for both forked and internally-executed Perl scripts it no longer reads the entire input into memory, at least not in miniserv.pl.<br /><br />=A0- Jamie<br /><br />On 27/Nov/2006 12:58 Dave Isaacs wrote .. <blockquote type=3D"cite"> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Thanks!</font></span></div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Did you also fix the forkcgis path?=A0 We now have the requirement to upload 500MB files to the server, so instead of limiting the file size we need to ensure that the file is not uploaded into memory first.=A0 If I remember correctly, the forkcgis path in miniserv.pl would do just that, but was not working correctly?</font></span></div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Forgive me if I am remembering this wrong.=A0 I am still working on dredging up all the details from my head.</font></span></div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Thanks</font></span></div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Dave I</font></span></div> <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div> <blockquote style=3D"margin-right: 0px;"> <div></div> <div lang=3D"en-us" dir=3D"ltr" align=3D"left" class=3D"OutlookMessageHeader"><font size=3D"2" face=3D"Tahoma">-----Original Message-----<br /><b>From:</b> web...@li... [mailto:web...@li...] <b>On Behalf Of </b>Jamie Cameron<br /><b>Sent:</b> Monday, November 27, 2006 3:52 PM<br /><b>To:</b> Webmin users list<br /><b>Subject:</b> Re: [webmin-l] Restricting file upload size<br /><br /></font></div>Hi Dave,<br />Yes, this has been fixed for a few versions now ..<br /><br />=A0- Jamie<br /><br />On 27/Nov/2006 12:42 Dave Isaacs wrote .. <blockquote type=3D"cite"> <p><font size=3D"2">Jamie,</font> </p> <p><font size=3D"2">Almost a year ago I posted this question about restricting file upload size.=A0 At the time you admitted to some miniserv.pl limitations and said that you would take care of this.=A0 </font></p> <p><font size=3D"2">Have you?=A0 If yes, great!=A0 If not, any estimates on when this can get in?</font> </p> <p><font size=3D"2">Thanks</font> </p> <p><font size=3D"2">Dave I</font> </p> <p><font size=3D"2">From: Jamie Cameron <jca...@we...></font> <br /><font size=3D"2">To: web...@li...</font> <br /><font size=3D"2">Reply-To: web...@li...</font> <br /><font size=3D"2">Date: Feb 10 2006 - 6:38pm</font> </p> <p><font size=3D"2">On 11/Feb/2006 03:19 Dave Isaacs wrote ..</font> <br /><font size=3D"2">> My experience shows that this does not work.</font> <br /><font size=3D"2">></font> <br /><font size=3D"2">> I put a 1000000 limit in my call to ReadParseMime then attempted to upload</font> <br /><font size=3D"2">> a</font> <br /><font size=3D"2">> 1GB file. Using top, I watched the miniserv.pl process climb to about</font> <br /><font size=3D"2">> 600MB</font> <br /><font size=3D"2">> before crashing. ReadParseMime was never called because my module was</font> <br /><font size=3D"2">> never</font> <br /><font size=3D"2">> invoked.</font> <br /><font size=3D"2">></font> <br /><font size=3D"2">> If I look at miniserv.pl, at around line 1740, I see</font> <br /><font size=3D"2">></font> <br /><font size=3D"2">> $clen =3D $header{"content-length"};</font> <br /><font size=3D"2">> if ($method eq "POST" && $clen_read < $clen) {</font> <br /><font size=3D"2">> # Still some more POST data to read</font> <br /><font size=3D"2">> while(length($postinput) < $clen) {</font> <br /><font size=3D"2">> $buf =3D &read_data($clen - length($postinput));</font> <br /><font size=3D"2">> if (!length($buf)) {</font> <br /><font size=3D"2">> &http_error(500, "Failed to read ".</font> <br /><font size=3D"2">> "POST request");</font> <br /><font size=3D"2">> }</font> <br /><font size=3D"2">> $postinput .=3D $buf;</font> <br /><font size=3D"2">> }</font> <br /><font size=3D"2">> }</font> <br /><font size=3D"2">></font> <br /><font size=3D"2">> This looks an awful lot like reading in the entire file upload. As a test,</font> <br /><font size=3D"2">> I wrote the length($postinput) value to a log file (right before the call</font> <br /><font size=3D"2">> to</font> <br /><font size=3D"2">> read_data) and found that miniserv.pl was looping in an attempt to read</font> <br /><font size=3D"2">> the</font> <br /><font size=3D"2">> entire file upload.</font> </p> <p><font size=3D"2">Hi Dave,</font> <br /><font size=3D"2">You are absolutely correct .. Webmin really does the whole posted input into</font> <br /><font size=3D"2">memory! Sorry, I totally forgot about that..</font> </p> <p><font size=3D"2">> Then I stumbled upon the forkcgis configuration setting, which appears</font> <br /><font size=3D"2">> to</font> <br /><font size=3D"2">> switch on a alternative method of invoking the webmin modules. This method</font> <br /><font size=3D"2">> has miniserv.pl forwarding the file upload to the forked process as it</font> <br /><font size=3D"2">> is</font> <br /><font size=3D"2">> received. Unfortunately, this does not work either. Now when I upload</font> <br /><font size=3D"2">> a</font> <br /><font size=3D"2">> large file, something goes wrong and there is never a response. The log</font> <br /><font size=3D"2">> messages I put in the miniserv loop shows that about 7500 bytes are read</font> <br /><font size=3D"2">> in,</font> <br /><font size=3D"2">> and then everything stops. Although this is better than crashing the</font> <br /><font size=3D"2">> server, it is still not correct.</font> </p> <p><font size=3D"2">I looked into this too, and found that Webmin is currently terminating the</font> <br /><font size=3D"2">browser connection if the uploaded data is more than the set limit. Unfortunately,</font> <br /><font size=3D"2">no browsers take kindly to this, and display an error message about the connection</font> <br /><font size=3D"2">being closed.</font> </p> <p><font size=3D"2">In the next release of Webmin, it will handle this better by reading all the data</font> <br /><font size=3D"2">submitted by the browser, but not actually storing it in memory if the limit is exceeded.</font> <br /><font size=3D"2">That is not quite ideal, but still better than the current situation.</font> </p> <p><font size=3D"2">- Jamie</font> </p></blockquote><br /></blockquote></blockquote><br /> |