Re: [webmin-devel] Webmin 1.320 file upload improvements question

[Chris M. <Chr...@en...>]

Well I slapped something together to prove to myself that some solution
could be worked out. Nothing I'd show anyone as is. :-)
 
I think the only way to deal with this situation is to write the file out
within the function, or possibly pass the file handle back to the
application and create a utility function to read the file handle for us.
The boundary checking could be in the read funct. The later might be to
complex for webmin lib, it would require secondary usage explaination and I
don't think it fits into the single function call spirit of the existing lib
functs.
 
Also, I want to try and utilize sysread to control STDIN reading if
possible. The only catch is boundary testing, but since the CONTENT_TYPE
boundary is defined early, we know the length, so something could be worked
out.
 
Thanks.
 
- Chris

-----Original Message-----
From: web...@li...
[mailto:web...@li...] On Behalf Of Jamie
Cameron
Sent: Wednesday, January 31, 2007 8:20 PM
To: Webmin development list
Subject: Re: [webmin-devel] Webmin 1.320 file upload improvements question


On 31/Jan/2007 16:26 Chris Mckenzie wrote .. 

Thanks for the pointer Jamie. 

So I funked around with my own version of ReadParseMime() and found that it
was quite easy to get my own version going. The attached file read/write
needed to be handled in the ReadParseMime() funct.

In web-lib-funcs.pl, around line 392, there's a $in{$name} concat of each
line read from the posted file. Is it because of the CONTENT_TYPE bounrdy
checking? This could probably be worked around. Anyways, it seems to be the
culprit. Once the concatination wasn't factored in, memory usage was
hovering around 5.8%.


Yes, it needs to check for the boundary on each line. 
I'd be intersted to see your code change that reduced memory use .. I didn't
see anything attached to your posting.


I could speculate that the <STDIN> read used is reading in browser sent
blocks of data, and isn't controlled from the function. If a sysread from
<STDIN> was used, we could probably control the size of data read. I don't
necessarily like that 5.8% number, I'm not sure where the size block size is
determined. (based on something in the file/data that the Perl read decided
was a read line, or something the browser blocked itself)


A read from stdin will just get a single line at a time, which shouldn't be
too large.


The only way I can see to really reduce memory use is for the caller of
ReadParseMime to specify a file that should be written to for a particular
named input. That way a large upload could be saved directly to disk on the
server..


 - Jamie


Please let me know what you think, though I'll probably write my own version
of ReadParseMime(). You're more than welcome to the result, but I have a
less Perl style of coding, so you'd probably end up re-writing yourself
anyways.

I really like the way you've used callback functions, it's going to factor
into my upload progress. 

Thanks! 

- Chris 
-----Original Message----- 
From: web...@li...
[mailto:web...@li...] On Behalf Of Jamie
Cameron

Sent: Wednesday, January 31, 2007 6:30 PM 
To: Webmin development list 
Subject: Re: [webmin-devel] Webmin 1.320 file upload improvements question 


On 31/Jan/2007 14:33 Chris Mckenzie wrote .. 
Hi all. 
Long time webmin tinkerer, short time webamin-devel list troller. 
I had a question regarding the the improved file upload module and progress
bar window. 
* Improved handling of large file uploads so that they are no longer read
into memory by Webmin webserver. Also added a progress bar window for
tracking uploads. 

Firstly, my tests have shown that when attempting a file upload to miniserv,
the main miniserv process maynot gobble up memory for the download but the
spawned CGI that's being posted to does.

Yes, that is still unfortunately true - when using the default ReadParseMime
function for reading uploaded files, the entire content is read into memory.

I wrote a quick test to see if the fix would be suitable for a module I'm
looking at doing. (large file upload) It starts with a simple form:

<form method="POST" action="/test/handleUpload.cgi"
enctype="multipart/form-data"> 
  <input type="file" name="file" id="file" size="20"> 
</form> 
My handleUpload.cgi is pretty straightforward too: 
&ReadParseMime(); 
... 
    my $fh = new IO::File ">$write_file"; 
    if(defined($fh)) { 
      binmode $fh; 
      print $fh $in{'file'}; 
    } 
    $fh->close(); 
... 
The only way to avoid comsuming memory like this would be to write your own
replacement for ReadParseMime to write data directly to a file as it is read
by the CGI. I haven't done this yet, but it would be at least theoretically
possible now that the miniserv.pl process doesn't read the whole upload into
memory too :-)

 - Jamie