Re: [webmin-devel] Webmin 1.320 file upload improvements question

[Jamie C. <jca...@we...>]

On 31/Jan/2007 14:33 Chris Mckenzie wrote ..
<blockquote type=3D"cite">


<p><font size=3D"2">Hi all.</font>
</p>

<p><font size=3D"2">Long time webmin tinkerer, short time webamin-devel list troller.</font>
</p>

<p><font size=3D"2">I had a question regarding the the improved file upload module and progress bar window.</font>
</p>

<p><font size=3D"2">* Improved handling of large file uploads so that they are no longer read into memory by Webmin webserver. Also added a progress bar window for tracking uploads. </font></p>

<p><font size=3D"2">Firstly, my tests have shown that when attempting a file upload to miniserv, the main miniserv process maynot gobble up memory for the download but the spawned CGI that's being posted to does.<br /></font></p></blockquote>Yes, that is still unfortunately true - when using the default ReadParseMime function for reading uploaded files, the entire content is read into memory.<br /><blockquote type=3D"cite">

<p><font size=3D"2">I wrote a quick test to see if the fix would be suitable for a module I'm looking at doing. (large file upload) It starts with a simple form:</font></p>

<p><font size=3D"2">&lt;form method=3D&quot;POST&quot; action=3D&quot;/test/handleUpload.cgi&quot; enctype=3D&quot;multipart/form-data&quot;&gt;</font>
<br /><font size=3D"2">=A0 &lt;input type=3D&quot;file&quot; name=3D&quot;file&quot; id=3D&quot;file&quot; size=3D&quot;20&quot;&gt;</font>
<br /><font size=3D"2">&lt;/form&gt;</font>
</p>

<p><font size=3D"2">My handleUpload.cgi is pretty straightforward too:</font>
</p>

<p><font size=3D"2">&amp;ReadParseMime();</font>
<br /><font size=3D"2">...</font>
<br /><font size=3D"2">=A0=A0=A0 my $fh =3D new IO::File &quot;&gt;$write_file&quot;;</font>
<br /><font size=3D"2">=A0=A0=A0 if(defined($fh)) {</font>
<br /><font size=3D"2">=A0=A0=A0=A0=A0 binmode $fh;</font>
<br /><font size=3D"2">=A0=A0=A0=A0=A0 print $fh $in{'file'};</font>
<br /><font size=3D"2">=A0=A0=A0 }</font>
<br /><font size=3D"2">=A0=A0=A0 $fh-&gt;close();</font>
<br /><font size=3D"2">...</font></p></blockquote><p><font size=3D"2">The only way to avoid comsuming memory like this would be to write your own replacement for ReadParseMime to write data directly to a file as it is read by the CGI. I haven't done this yet, but it would be at least theoretically possible now that the miniserv.pl process doesn't read the whole upload into memory too :-)</font></p><p><font size=3D"2">=A0- Jamie</font></p><br />