Re: [webmin-l] Restricting file upload size

[Jamie C. <jca...@we...>]

Hi Dave,<br />I checked my code again, and found that only with forkcgis=3D1 does it not read the uploaded data into memory, sorry.<br />However, in that mode you need to make sure that all your .cgi scripts are executable (with chmod +x *.cgi).<br /><br />=A0- Jamie<br /><br />On 28/Nov/2006 08:02 Dave Isaacs wrote ..
<blockquote type=3D"cite">

<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">I must 
be missing something.=A0 I've rerun my tests and they show the entire file 
being uploaded before my module is being called.=A0 This is in the case when 
forkcgis=3D0.</font></span></div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">When I 
set forkcgis=3D1, Webmin stops working completely.=A0 I just get a file 
permissions error showing up the in browser.</font></span></div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">I am 
using Webmin version 1.290 on RedHat EL3.</font></span></div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Thanks</font></span></div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
<div><span class=3D"146175715-28112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Dave 
I</font></span></div>
<blockquote style=3D"margin-right: 0px;">
  <div></div>
  <div lang=3D"en-us" align=3D"left" dir=3D"ltr" class=3D"OutlookMessageHeader"><font size=3D"2" face=3D"Tahoma">-----Original Message-----<br /><b>From:</b> 
  web...@li... 
  [mailto:web...@li...] <b>On Behalf Of </b>Jamie 
  Cameron<br /><b>Sent:</b> Monday, November 27, 2006 5:51 PM<br /><b>To:</b> Webmin 
  users list<br /><b>Subject:</b> Re: [webmin-l] Restricting file upload 
  size<br /><br /></font></div>Hi Dave,<br />Yes, for both forked and 
  internally-executed Perl scripts it no longer reads the entire input into 
  memory, at least not in miniserv.pl.<br /><br />=A0- Jamie<br /><br />On 
  27/Nov/2006 12:58 Dave Isaacs wrote .. 
  <blockquote type=3D"cite">
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Thanks!</font></span></div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Did you also fix the forkcgis path?=A0 We now have the requirement 
    to upload 500MB files to the server, so instead of limiting the file size we 
    need to ensure that the file is not uploaded into memory first.=A0 If I 
    remember correctly, the forkcgis path in miniserv.pl would do just that, but 
    was not working correctly?</font></span></div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Forgive me if I am remembering this wrong.=A0 I am still working 
    on dredging up all the details from my head.</font></span></div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Thanks</font></span></div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff">Dave I</font></span></div>
    <div><span class=3D"810325420-27112006"><font size=3D"2" face=3D"Arial" color=3D"#0000ff"></font></span>=A0</div>
    <blockquote style=3D"margin-right: 0px;">
      <div></div>
      <div lang=3D"en-us" align=3D"left" dir=3D"ltr" class=3D"OutlookMessageHeader"><font size=3D"2" face=3D"Tahoma">-----Original Message-----<br /><b>From:</b> 
      web...@li... 
      [mailto:web...@li...] <b>On Behalf Of 
      </b>Jamie Cameron<br /><b>Sent:</b> Monday, November 27, 2006 3:52 
      PM<br /><b>To:</b> Webmin users list<br /><b>Subject:</b> Re: [webmin-l] 
      Restricting file upload size<br /><br /></font></div>Hi Dave,<br />Yes, this has 
      been fixed for a few versions now ..<br /><br />=A0- Jamie<br /><br />On 
      27/Nov/2006 12:42 Dave Isaacs wrote .. 
      <blockquote type=3D"cite">
        <p><font size=3D"2">Jamie,</font> </p>
        <p><font size=3D"2">Almost a year ago I posted this question about 
        restricting file upload size.=A0 At the time you admitted to some 
        miniserv.pl limitations and said that you would take care of this.=A0 
        </font></p>
        <p><font size=3D"2">Have you?=A0 If yes, great!=A0 If not, any 
        estimates on when this can get in?</font> </p>
        <p><font size=3D"2">Thanks</font> </p>
        <p><font size=3D"2">Dave I</font> </p>
        <p><font size=3D"2">From: Jamie Cameron &lt;jca...@we...&gt;</font> 
        <br /><font size=3D"2">To: web...@li...</font> 
        <br /><font size=3D"2">Reply-To: web...@li...</font> 
        <br /><font size=3D"2">Date: Feb 10 2006 - 6:38pm</font> </p>
        <p><font size=3D"2">On 11/Feb/2006 03:19 Dave Isaacs wrote ..</font> 
        <br /><font size=3D"2">&gt; My experience shows that this does not 
        work.</font> <br /><font size=3D"2">&gt;</font> <br /><font size=3D"2">&gt; I put a 
        1000000 limit in my call to ReadParseMime then attempted to 
        upload</font> <br /><font size=3D"2">&gt; a</font> <br /><font size=3D"2">&gt; 1GB 
        file. Using top, I watched the miniserv.pl process climb to about</font> 
        <br /><font size=3D"2">&gt; 600MB</font> <br /><font size=3D"2">&gt; before 
        crashing. ReadParseMime was never called because my module was</font> 
        <br /><font size=3D"2">&gt; never</font> <br /><font size=3D"2">&gt; invoked.</font> 
        <br /><font size=3D"2">&gt;</font> <br /><font size=3D"2">&gt; If I look at 
        miniserv.pl, at around line 1740, I see</font> <br /><font size=3D"2">&gt;</font> <br /><font size=3D"2">&gt; $clen =3D 
        $header{&quot;content-length&quot;};</font> <br /><font size=3D"2">&gt; if ($method eq 
        &quot;POST&quot; &amp;&amp; $clen_read &lt; $clen) {</font> <br /><font size=3D"2">&gt; 
        # Still some more POST data to read</font> <br /><font size=3D"2">&gt; 
        while(length($postinput) &lt; $clen) {</font> <br /><font size=3D"2">&gt; $buf 
        =3D &amp;read_data($clen - length($postinput));</font> <br /><font size=3D"2">&gt; if (!length($buf)) {</font> <br /><font size=3D"2">&gt; 
        &amp;http_error(500, &quot;Failed to read &quot;.</font> <br /><font size=3D"2">&gt; 
        &quot;POST request&quot;);</font> <br /><font size=3D"2">&gt; }</font> <br /><font size=3D"2">&gt; $postinput .=3D $buf;</font> <br /><font size=3D"2">&gt; }</font> 
        <br /><font size=3D"2">&gt; }</font> <br /><font size=3D"2">&gt;</font> <br /><font size=3D"2">&gt; This looks an awful lot like reading in the entire file 
        upload. As a test,</font> <br /><font size=3D"2">&gt; I wrote the 
        length($postinput) value to a log file (right before the call</font> 
        <br /><font size=3D"2">&gt; to</font> <br /><font size=3D"2">&gt; read_data) and 
        found that miniserv.pl was looping in an attempt to read</font> 
        <br /><font size=3D"2">&gt; the</font> <br /><font size=3D"2">&gt; entire file 
        upload.</font> </p>
        <p><font size=3D"2">Hi Dave,</font> <br /><font size=3D"2">You are absolutely 
        correct .. Webmin really does the whole posted input into</font> 
        <br /><font size=3D"2">memory! Sorry, I totally forgot about that..</font> 
</p>
        <p><font size=3D"2">&gt; Then I stumbled upon the forkcgis configuration 
        setting, which appears</font> <br /><font size=3D"2">&gt; to</font> <br /><font size=3D"2">&gt; switch on a alternative method of invoking the webmin 
        modules. This method</font> <br /><font size=3D"2">&gt; has miniserv.pl 
        forwarding the file upload to the forked process as it</font> <br /><font size=3D"2">&gt; is</font> <br /><font size=3D"2">&gt; received. Unfortunately, 
        this does not work either. Now when I upload</font> <br /><font size=3D"2">&gt; a</font> <br /><font size=3D"2">&gt; large file, something goes 
        wrong and there is never a response. The log</font> <br /><font size=3D"2">&gt; messages I put in the miniserv loop shows that about 7500 
        bytes are read</font> <br /><font size=3D"2">&gt; in,</font> <br /><font size=3D"2">&gt; and then everything stops. Although this is better than 
        crashing the</font> <br /><font size=3D"2">&gt; server, it is still not 
        correct.</font> </p>
        <p><font size=3D"2">I looked into this too, and found that Webmin is 
        currently terminating the</font> <br /><font size=3D"2">browser connection if 
        the uploaded data is more than the set limit. Unfortunately,</font> 
        <br /><font size=3D"2">no browsers take kindly to this, and display an error 
        message about the connection</font> <br /><font size=3D"2">being 
        closed.</font> </p>
        <p><font size=3D"2">In the next release of Webmin, it will handle this 
        better by reading all the data</font> <br /><font size=3D"2">submitted by the 
        browser, but not actually storing it in memory if the limit is 
        exceeded.</font> <br /><font size=3D"2">That is not quite ideal, but still 
        better than the current situation.</font> </p>
        <p><font size=3D"2">- Jamie</font> 
  </p></blockquote><br /></blockquote></blockquote><br /></blockquote></blockquote><br />