Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 09:06 schrieb Stephen Deasey:
>
> However...  Vlad's question about ns_getform has me wondering about
> the common case of HTTP file upload.  Consider a 10MB file upload --
> just before the form parsing code does it's thing, the situation is
> that there is a file on disk containing the 10MB of data, plus the
> mime header, plus the content separator as a trailer.  What you want
> is a file with just the 10MB of data.  You can truncate a file to trim
> off the trailer, but you can't trim the mime header at the beginning.
> The 10MB of data will have to be copied to a new file, and the spool
> file will be deleted when the connection is closed.
>
> This is no worse than the situation we're currently in or that of the
> 4.1 code base. I'm not sure what 3.x did here.  But it does seem a bit
> unfortunate.
>
> How about this:  we introduce a new private interface something like
> NsConnPeek(Ns_Conn *conn, int *lenPtr) which returns a pointer into
> the read-ahead buffer just after the HTTP headers, *without* first
> spooling the entire request body to disk.  The form processing code
> would use this to hopefully parse the mime header, and probably some
> standard www-form-urlencoded data that was sent at the same time.  It
> would then reset the current offset into the buffer (which was
> previously positioned at the end of the HTTP headers) so that when the
> spooling code takes over it begins at the start of the file content.
> The spool file would then be truncated to remove the trailing content
> separator.

This assumes that form elements are packed before the actual data
in the multipart container. But what if not? Then you'd still need
to get the whole thing into the tempfile.
I beleive that's why you said "hopefully" above, right?

>
> For multiple simultaneous file uploads you'd have to revert to copy
> the data to new files.  But for the common case, we'd have something
> pretty efficient without too much complexity and retaining the nice
> interface that mmap allows.
>

I understand. I believe we can add this as one of the improvements
after the first (bigger) change is done.

What I also thought about is: a way of somehow register some Tcl
callbacks when accepting a large input. If a guy uploads 10MB file
over slow connection, the browser just does not give you any clue
how for it went with upl=F6ading. OTOH, on the server we know this.
We could register a Tcl callback to monitor the upload process,
store this data in shared memory so people could write a poll-page
which just examines this statistics-data and gives the user (over =20
another
connection) the idea how much has been received.
Actually, we implemented this in 3.x server and it worked fine
(since the whole upload thing was actually done in Tcl alone).
In 4.x it stopped working because of the fact that all receiving
and parsing is done internally.

Zoran