naviserver-devel — Developer discussion

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 12:30 schrieb Zoran Vasiljevic:

>>
>> I see that quite a lot of plumbing must be done
>> to get this working. We must keep the Sock for
>

Wrong. Not very many plumbings, as we have almost
all in place.

The chages would be mainly in the driver.c SockRead()
to declare SOCK_READY at the point of reading maxreadahead
even if there are still bytes left to consume from socket.

The Ns_ConnContent has to check this fact and go and
fetch the remaining (spooling all thats left in a
mmaped file) before returning the pointer to the
content.

Ns_ConnClose should respectively do unmapping
of files, if any mapped.  Conn structure should be expaned
to hold mmaped file maintenance fields of course.

The NsTclConnObjCmd should get someting like
ns_conn setmaxinput or similar (details can be cleared here).

All in all, it does not seems like too much plumbing
as I anticipated.

Now you core hackers, correct me if I'm wrong.

Zoran 

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 11:45 schrieb Zoran Vasiljevic:

>
> Am 17.06.2005 um 10:07 schrieb Stephen Deasey:
>
>
>>
>> Yeah, looks like a mult-stage project.
>>
>>
>
> I see that quite a lot of plumbing must be done
> to get this working. We must keep the Sock for
> the request in the Request structure for later
> use by the Ns_ConnContent. Also, the Conn
> structure must be expanded for tracking the mmaped
> file... This is just a first glance...
>
> I see I'd have to go thru the whole request processing
> machinery and understand all pieces in order to fit-in
> this change...


Ok, the Conn already tracks the sockPtr
so we must not modify the Request struct
(if i see this correctly).

Still digging (electronic archeology) the code :-)
I' was not that often in those areas of the code...

Zoran

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 11:08 schrieb Stephen Deasey:

>
> File a bug report with Mozilla, Safari, Opera etc. "No visible
> progress for large file uploads"...  :-)
>
> Actually, they're all pretty responsive these days, and when one of
> them gets a good idea, the others are often quick to copy.  The
> undying gratitude of the browsing public may not be beyond your
> reach...
>


Do you seriously think somebody would listen?
OTOH, this is such a generic thing that I absolutely do not
understand why nobody added this before?!?!

Anyways, we can live with the "uploading file..." animated
gif for the time being.

Zoran

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 10:07 schrieb Stephen Deasey:

>
> Yeah, looks like a mult-stage project.
>

I see that quite a lot of plumbing must be done
to get this working. We must keep the Sock for
the request in the Request structure for later
use by the Ns_ConnContent. Also, the Conn
structure must be expanded for tracking the mmaped
file... This is just a first glance...

I see I'd have to go thru the whole request processing
machinery and understand all pieces in order to fit-in
this change...

Zoran

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/17/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 17.06.2005 um 10:07 schrieb Stephen Deasey:
>=20
> > You can of course fake it:  as soon as the connection which accepts
> > the large file runs it begins a server push, the first page of which
> > contains an animated gif 'uploading...'.  Once the upload completes
> > the server pushes a new page which confirms the upload.
> >
> > You don't get actual upload progress, but it does reassure users that
> > something is happening and prevents them from pressing the upload
> > button again...
> >
>=20
> Exactly *this* is what we have reverted to with 4.x server!


File a bug report with Mozilla, Safari, Opera etc. "No visible
progress for large file uploads"...  :-)

Actually, they're all pretty responsive these days, and when one of
them gets a good idea, the others are often quick to copy.  The
undying gratitude of the browsing public may not be beyond your
reach...

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 10:07 schrieb Stephen Deasey:

> You can of course fake it:  as soon as the connection which accepts
> the large file runs it begins a server push, the first page of which
> contains an animated gif 'uploading...'.  Once the upload completes
> the server pushes a new page which confirms the upload.
>
> You don't get actual upload progress, but it does reassure users that
> something is happening and prevents them from pressing the upload
> button again...
>

Exactly *this* is what we have reverted to with 4.x server!
Zoran

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/17/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 17.06.2005 um 09:24 schrieb Stephen Deasey:
>=20
> >
> > I believe this is pretty common, but I'm not sure if this is what you
> > want.  An anonymous mapping is still going to account against the
> > processes memory budget, and I don't think it's going to be any more
> > likely to be swapped out than malloc'ed memory.
> >
> > In fact, on Linux (well, glibc) if you ask malloc for a chunk of
> > memory over a certain size then mmap is used internally.  I think the
> > advantage is reduced memory fragmentation, but there is some overhead
> > for small sizes and you have to allocate in multiples of the page size
> > (4k).
> >
> > tempfs would be a better bet.  Modern Linux systems have this mounted
> > at /dev/shm.  I think this is a Solaris thing.  This is a very low
> > overhead file system where the file data springs into existence when
> > you ask for it.  So, no files, no overhead.  Under memory pressure
> > it's sent to swap.  It was designed for sharing memory between
> > processes using mmap, but it's handy for /tmp and some other things.
> >
> > You could try setting the TMPDIR environment variable to some tempfs
> > file system if you wanted to experiment with this.
>=20
> Not to forget: Darwin, Windows... I think I will have to find a
> common acceptable solution for all OS'es. Fortunately, mapping
> a regular (temp) file will always work. The rest is just optimizations.


Right, but you don't need any code changes.  You should be able to set
the temp directory via environment variables on those platforms that
support a fancy tempfs. Something for the sysadmins...

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/17/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 17.06.2005 um 09:06 schrieb Stephen Deasey:
> >
> > However...  Vlad's question about ns_getform has me wondering about
> > the common case of HTTP file upload.  Consider a 10MB file upload --
> > just before the form parsing code does it's thing, the situation is
> > that there is a file on disk containing the 10MB of data, plus the
> > mime header, plus the content separator as a trailer.  What you want
> > is a file with just the 10MB of data.  You can truncate a file to trim
> > off the trailer, but you can't trim the mime header at the beginning.
> > The 10MB of data will have to be copied to a new file, and the spool
> > file will be deleted when the connection is closed.
> >
> > This is no worse than the situation we're currently in or that of the
> > 4.1 code base. I'm not sure what 3.x did here.  But it does seem a bit
> > unfortunate.
> >
> > How about this:  we introduce a new private interface something like
> > NsConnPeek(Ns_Conn *conn, int *lenPtr) which returns a pointer into
> > the read-ahead buffer just after the HTTP headers, *without* first
> > spooling the entire request body to disk.  The form processing code
> > would use this to hopefully parse the mime header, and probably some
> > standard www-form-urlencoded data that was sent at the same time.  It
> > would then reset the current offset into the buffer (which was
> > previously positioned at the end of the HTTP headers) so that when the
> > spooling code takes over it begins at the start of the file content.
> > The spool file would then be truncated to remove the trailing content
> > separator.
>=20
> This assumes that form elements are packed before the actual data
> in the multipart container. But what if not? Then you'd still need
> to get the whole thing into the tempfile.
> I beleive that's why you said "hopefully" above, right?


Hopefully in that all the data you need to pre-parse fits in the
read-ahead buffer.  It's not a problem if the extra form data comes
after the file upload data.  Files can be truncated, but they can't
have data trimmed from the beginning.  That's what this is all about:
we're setting the read offset so as to 'trim' the beginning of the
spool file contents.  It can later be truncated however you want.=20
Truncating is cheap, trimming is expensive because it means copying
data.


> >
> > For multiple simultaneous file uploads you'd have to revert to copy
> > the data to new files.  But for the common case, we'd have something
> > pretty efficient without too much complexity and retaining the nice
> > interface that mmap allows.
> >
>=20
> I understand. I believe we can add this as one of the improvements
> after the first (bigger) change is done.


Yeah, looks like a mult-stage project.


> What I also thought about is: a way of somehow register some Tcl
> callbacks when accepting a large input. If a guy uploads 10MB file
> over slow connection, the browser just does not give you any clue
> how for it went with upl=F6ading. OTOH, on the server we know this.
> We could register a Tcl callback to monitor the upload process,
> store this data in shared memory so people could write a poll-page
> which just examines this statistics-data and gives the user (over
> another
> connection) the idea how much has been received.
> Actually, we implemented this in 3.x server and it worked fine
> (since the whole upload thing was actually done in Tcl alone).
> In 4.x it stopped working because of the fact that all receiving
> and parsing is done internally.


Hmm, not sure what the best way to do that is.

You can of course fake it:  as soon as the connection which accepts
the large file runs it begins a server push, the first page of which
contains an animated gif 'uploading...'.  Once the upload completes
the server pushes a new page which confirms the upload.

You don't get actual upload progress, but it does reassure users that
something is happening and prevents them from pressing the upload
button again...

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 09:24 schrieb Stephen Deasey:

>
> I believe this is pretty common, but I'm not sure if this is what you
> want.  An anonymous mapping is still going to account against the
> processes memory budget, and I don't think it's going to be any more
> likely to be swapped out than malloc'ed memory.
>
> In fact, on Linux (well, glibc) if you ask malloc for a chunk of
> memory over a certain size then mmap is used internally.  I think the
> advantage is reduced memory fragmentation, but there is some overhead
> for small sizes and you have to allocate in multiples of the page size
> (4k).
>
> tempfs would be a better bet.  Modern Linux systems have this mounted
> at /dev/shm.  I think this is a Solaris thing.  This is a very low
> overhead file system where the file data springs into existence when
> you ask for it.  So, no files, no overhead.  Under memory pressure
> it's sent to swap.  It was designed for sharing memory between
> processes using mmap, but it's handy for /tmp and some other things.
>
> You could try setting the TMPDIR environment variable to some tempfs
> file system if you wanted to experiment with this.

Not to forget: Darwin, Windows... I think I will have to find a
common acceptable solution for all OS'es. Fortunately, mapping
a regular (temp) file will always work. The rest is just optimizations.

Cheers
Zoran

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 09:06 schrieb Stephen Deasey:
>
> However...  Vlad's question about ns_getform has me wondering about
> the common case of HTTP file upload.  Consider a 10MB file upload --
> just before the form parsing code does it's thing, the situation is
> that there is a file on disk containing the 10MB of data, plus the
> mime header, plus the content separator as a trailer.  What you want
> is a file with just the 10MB of data.  You can truncate a file to trim
> off the trailer, but you can't trim the mime header at the beginning.
> The 10MB of data will have to be copied to a new file, and the spool
> file will be deleted when the connection is closed.
>
> This is no worse than the situation we're currently in or that of the
> 4.1 code base. I'm not sure what 3.x did here.  But it does seem a bit
> unfortunate.
>
> How about this:  we introduce a new private interface something like
> NsConnPeek(Ns_Conn *conn, int *lenPtr) which returns a pointer into
> the read-ahead buffer just after the HTTP headers, *without* first
> spooling the entire request body to disk.  The form processing code
> would use this to hopefully parse the mime header, and probably some
> standard www-form-urlencoded data that was sent at the same time.  It
> would then reset the current offset into the buffer (which was
> previously positioned at the end of the HTTP headers) so that when the
> spooling code takes over it begins at the start of the file content.
> The spool file would then be truncated to remove the trailing content
> separator.

This assumes that form elements are packed before the actual data
in the multipart container. But what if not? Then you'd still need
to get the whole thing into the tempfile.
I beleive that's why you said "hopefully" above, right?

>
> For multiple simultaneous file uploads you'd have to revert to copy
> the data to new files.  But for the common case, we'd have something
> pretty efficient without too much complexity and retaining the nice
> interface that mmap allows.
>

I understand. I believe we can add this as one of the improvements
after the first (bigger) change is done.

What I also thought about is: a way of somehow register some Tcl
callbacks when accepting a large input. If a guy uploads 10MB file
over slow connection, the browser just does not give you any clue
how for it went with upl=F6ading. OTOH, on the server we know this.
We could register a Tcl callback to monitor the upload process,
store this data in shared memory so people could write a poll-page
which just examines this statistics-data and gives the user (over =20
another
connection) the idea how much has been received.
Actually, we implemented this in 3.x server and it worked fine
(since the whole upload thing was actually done in Tcl alone).
In 4.x it stopped working because of the fact that all receiving
and parsing is done internally.

Zoran

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 16.06.2005 um 18:18 schrieb Zoran Vasiljevic:
>=20
> >
> > One can even explore the memmap machinery and see if we can
> > entirely drop the temp-file and use the system paging for that:
> > so if the input exceeds the maxinput, we just mmap /dev/zero file
> > I believe something like that should be possible but will have to
> > double-check if this is true.
> >
>=20
> I knew it ("man mmap" on solaris):
>=20
>=20
>       When MAP_ANON is set in flags, and fd is set to  -1,  mmap()
>       provides  a  direct  path  to  return anonymous pages to the
>       caller.  This operation is equivalent to passing  mmap()  an
>       open  file descriptor on /dev/zero with MAP_ANON elided from
>       the flags argument.
>=20
> I'd have to see if this is really working on Linux, Mac and Win.


I believe this is pretty common, but I'm not sure if this is what you
want.  An anonymous mapping is still going to account against the
processes memory budget, and I don't think it's going to be any more
likely to be swapped out than malloc'ed memory.

In fact, on Linux (well, glibc) if you ask malloc for a chunk of
memory over a certain size then mmap is used internally.  I think the
advantage is reduced memory fragmentation, but there is some overhead
for small sizes and you have to allocate in multiples of the page size
(4k).

tempfs would be a better bet.  Modern Linux systems have this mounted
at /dev/shm.  I think this is a Solaris thing.  This is a very low
overhead file system where the file data springs into existence when
you ask for it.  So, no files, no overhead.  Under memory pressure
it's sent to swap.  It was designed for sharing memory between
processes using mmap, but it's handy for /tmp and some other things.

You could try setting the TMPDIR environment variable to some tempfs
file system if you wanted to experiment with this.

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 16.06.2005 um 22:13 schrieb Stephen Deasey:
>=20
> >
> > It was at one point implemented with mmap:
> >
> > http://cvs.sourceforge.net/viewcvs.py/aolserver/aolserver/nsd/
> > driver.c?rev=3D1.34&view=3Dmarkup
> >
>=20
> Aha! But it was taken out for some reason...
> It would be good to know why?
>=20
> >
> > Spooling large requests to disk is clearly necessary.  Almost always,
> > a large request needs to be stored in a file server-side anyway, so it
> > might as well go there from the start.
> >
> > The only problem I see is that the calls to open a file and write
> > content to disk are blocking, and they happen in the context of the
> > driver thread which is busy multiplexing many non-blocking operations
> > on sockets.  Should one of the calls used to spool to disk block,
> > everything comes to a stop.
>=20
> Oh yes. This is very right.
>=20
> >
> > The version 3.x model was to read-ahead up to the end of headers in
> > the driver thread, then pass the conn to a conn thread.  As far as I
> > remember, the conn thread would then spool all content to disk, but
> > this could have been on-demand and it might have been only file-upload
> > data. A programmer could then access the spooled data using
> > Ns_ConnRead(), Ns_ConnReadLine() etc., or they would be called for you
> > by ns_getform etc.  The thing to note however is that all the blocking
> > calls happened in separate conn threads.
>=20
> Yes. I remember this because the multipart data was parsed by a
> Tcl script. Not very speedy but it worked surprisingly stable.
>=20
> >
> > In early version 4.0 the model was changed so that the driver thread
> > would read-ahead all data (up to Content-Length bytes) before the conn
> > was passed to a conn thread.  In a point release a limit was
> > introduced to avoid the obvious DOS attack.  This allowed an easier
> > interface to the data for programmers: Ns_ConnContent() which returns
> > a pointer to an array of bytes.  Ns_ConnReadLine() etc. are no longer
> > used and are currently broken.
>=20
> But this brought the memory bloat as a side-effect
> (you can make all happy) :-)
>=20
> >
> > Version 4.1 work seems to be trying to tackle the problem of what
> > happens when large files are uploaded.  The version 4.0 model work
> > extremely well for HTTP POST data up to a few K in size, but as you've
> > noticed it really bloats memory when multi-megabyte files are
> > uploaded.  This version also introduces limits, which are a
> > URL-specific way of pre-declaring the maxupload size and some other
> > parameters.
>=20
> But still, just spools the data into temp-file at the very sensitive
> point (in the driver.c) as you already pointed-out above.
>=20
> >
> >
> > So anyway, here's my idea of how it should work:
>=20
> I'm all ears :-)
>=20
> >
> > There's a maxreadahead parameter which is <=3D maxinput.  When a reques=
t
> > arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are
> > read into a buffer by the driver thread before the conn is passed to a
> > conn thread.
> >
> > The conn thread runs the registered proc for that URL.  If that
> > procedure does not try to acces the content, then when the conn is
> > returned to the driver thread any content > maxreadahead is drained.
> >
> > If the registered proc does try to access the content via e.g.
> > Ns_ConnContent() (and I think this would be the main method, used
> > underneath by all others) and the content is <=3D maxreadahead, then a
> > pointer to the readahead buffer is returned.
> >
> > If the content is accessed and it is > maxreadahead, a temp file is
> > mmaped, the readahead buffer is dumped to it, and the remaining bytes
> > are read from the socket, possibly blocking many times, and dumped to
> > the mmaped file, again possibly blocking.  A call to Ns_ConnContent()
> > returns a pointer to the mmaped bytes of the open file.
>=20
> But this is now happening outside the driver and in the connection
> thread, ok.
>=20
> >
> > At any time before the registered proc asks for the content it can
> > check the content length header and decide whether it is too large to
> > accept.  You could imagine setting a low global maxinput, and a call
> > such as Ns_ConnSetMaxInput() which a registered proc could call to
> > increase the limit for that connection only.  The advantage over the
> > limits scheme in 4.1 is that the code which checks the size of the
> > content and processes it is kept together, rather than having to
> > pre-decalare maxinput sizes for arbitrary URLs in the config file.
>=20
> Hm... this is the only part I do not understand.
>=20
> >
> > This is similar to 4.1 except the task of overflowing to disk is moved
> > to the conn thread and is lazy.  The mental model is also slightly
> > different, the assumption is that content goes to disk, but theres a
> > cache for read-ahead which may be enough to hold everything.  In 4.1
> > it's everything is read into memory, unless it's large in which case
> > it overflows to disk.
> >
> >
> > How does that sound?
> >
>=20
> Summary (my understanding):
>=20
> This is a kind of marriage between the 3.0 and 4.0 strategies as I see.
> Up to maxreadahead, it is behaving like 4.0 (loads all in memory) and
> above maxreadahead it behaves like 3.0 (spools to disk).


We would be slightly better than 3.x in that we would use your new
mmap abstraction, which is nice because of the simple interface
(buffer of bytes) it allows us to give the application programmer.  In
3.x, I think your only options were the provided Ns_ConnReadLine()
etc. or a file descriptor.

As far as I remember, 3.x was a little bit more complicated than
described so far.  Once the headers were read by the driver thread and
the conn passed to a conn thread, the remaining request body would be
read lazily in chunks.  That is, if you called Ns_ConnReadLine(), it
would read a single buffer of bytes from the network and look for the
next newline character in that buffer.  If one was found, it would
hand you a line of data.  If not, it would block reading more data
from the network, and so on until the line is found or there's no more
data.

Just to be clear, that's *not* what I suggested we build above.  I was
suggesting that on the first attempt to read any of the request body,
*all* remaining data > maxreadahead would be spooled to disk.  This is
a much simpler model, and I think code simplification was a big reason
for the 3.x -> 4.x transition.

However...  Vlad's question about ns_getform has me wondering about
the common case of HTTP file upload.  Consider a 10MB file upload --
just before the form parsing code does it's thing, the situation is
that there is a file on disk containing the 10MB of data, plus the
mime header, plus the content separator as a trailer.  What you want
is a file with just the 10MB of data.  You can truncate a file to trim
off the trailer, but you can't trim the mime header at the beginning.=20
The 10MB of data will have to be copied to a new file, and the spool
file will be deleted when the connection is closed.

This is no worse than the situation we're currently in or that of the
4.1 code base. I'm not sure what 3.x did here.  But it does seem a bit
unfortunate.

How about this:  we introduce a new private interface something like
NsConnPeek(Ns_Conn *conn, int *lenPtr) which returns a pointer into
the read-ahead buffer just after the HTTP headers, *without* first
spooling the entire request body to disk.  The form processing code
would use this to hopefully parse the mime header, and probably some
standard www-form-urlencoded data that was sent at the same time.  It
would then reset the current offset into the buffer (which was
previously positioned at the end of the HTTP headers) so that when the
spooling code takes over it begins at the start of the file content.=20
The spool file would then be truncated to remove the trailing content
separator.

For multiple simultaneous file uploads you'd have to revert to copy
the data to new files.  But for the common case, we'd have something
pretty efficient without too much complexity and retaining the nice
interface that mmap allows.


How does this sound?



> The most significant fact is that potentially blocking calls while
> spooling
> to disk are taken out of the driver and done in the connection thread
> (where it does not matter that much) and that the process is lazy
> (done only when content is actually requested and not in advance).
>=20
> Yes, this sound fine, indeed. The only think I'd have yet to
> understand is the twiddling with the maxinput value on per-connection
> basis. I do not know *when* this would happen. Can you give a simple
> practical example?
>=20
>=20
> Zoran
>=20
>=20
>=20
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=3D7477&alloc_id=3D16492&op=3Dclic=
k
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 17.06.2005 um 07:29 schrieb Stephen Deasey:

>
> Hope I didn't confuse the issue here, but Vlad seemed worried about
> having connection threads tied up handling blocking IO.  We have some
> good options: adjust the readahead buffer, upto maxinput if needed,
> and use connection thread pools to partition resource intensive URLs.
>

This is acceptable for me. I would not go into the extent of adding
yet another layer of processing just to save conn threads from blocking.
I believe the thread-pool of conn threads dealing large uploads is
perfectly OK to start with. If this ever becomes a problem for somebody
we could add intermediate socket spooler threads.

And the implementation also does not seem to be a rocket science.
I'm all for this.

Zoran 

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 16.06.2005 um 22:13 schrieb Stephen Deasey:
>=20
> >
> > At any time before the registered proc asks for the content it can
> > check the content length header and decide whether it is too large to
> > accept.  You could imagine setting a low global maxinput, and a call
> > such as Ns_ConnSetMaxInput() which a registered proc could call to
> > increase the limit for that connection only.  The advantage over the
> > limits scheme in 4.1 is that the code which checks the size of the
> > content and processes it is kept together, rather than having to
> > pre-decalare maxinput sizes for arbitrary URLs in the config file.
> >
>=20
>=20
> Aha... do:
>=20
>    ns_conn setmaxinput
>=20
> before we do
>=20
>    ns_conn content
>=20
> to get the content of the request.
>=20
> Hm... so we'd keep the maxinput reasonably *small* for most
> of the requests and will re-set it to a larger value *before*
> requesting the content when we anticipate large file upload.
>=20
> I think I'm beginning to understand....
> Yes, this makes sense.


Yes, that's the idea.  I mentioned it because 4.1 introduces a new
'limits' mechanism to deal with this, and as we're looking at
different implementations to decide what to do, I thought I'd compare
the two strategies.

Nothing stops us from also implementing limits.  But with 4.1 where
the request body is read eagerly and spilled to disk via the driver
thread, the *only* way to set the maxinput on a per-URL as opposed to
per-server basis is to preregister this limits data.

With the scheme we're describing here, we can use a more natural,
linear programming style to set the limit before asking for the data.

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 16.06.2005 um 23:35 schrieb Vlad Seryakov:
>=20
> > I understand that but my question is:
> > How and when all content will be downloaded?
> > By the driver? By the conn thread on first call?
> >
>=20
> The driver will load up to maxreadahead.
>=20
> At the first call to Ns_ConnContent, the rest
> will be sinked from the socket into the buffer
> (until it fills) and the rest will be sinked
> into the mmaped temp file. This is now done
> in the connection thread.
>=20
> > Currently driver reads everything into memory.
> >
>=20
> Yes, which is a problem. So it will be split between
> the driver thread (up to maxreadahead) and connection
> thread (the rest, if available).


Yes, exactly right.  You could however set maxreadahead =3D=3D maxinput
for exactly the same behaviour as you get now, all content will be
read by the driver thread before a conn thread ever gets woken.

Obviously this is somewhat of a juggling act.  What's the largest
request body size you ever expect, what's the frequency of these large
requests compared to small requests, therefore how big should the
readahead buffer be to maximize performance but minimize memory bloat?

The other resource being used is the conn thread itself, which is
expensive because of the attached Tcl interp and all its procs in both
text and byte code form.  For every request body > maxreadahead an
expensive conn thread will have to block waiting for data over the
network, causing a context switch.

One option here would be a pool of lightweight threads whose only job
is to read data from the network and write it to disk.  Once the
readahead buffer is full, the connection would be passed to a
lightweight thread which would spool all the data to disk, and then
pass the connection to a conn thread.

I don't think it's worth the complexity, not at this stage.  We
already have a more general purpose partitioning scheme, connection
thread pools.  In this context, you assume that only few URLs need to
accept large request bodies and so you partition them into their own
thread pool.  Now, should a group of poor modem users start dribbling
20MB of data1k at a time, you may run out of photo upload threads, but
the rest of your service should be fine.


Hope I didn't confuse the issue here, but Vlad seemed worried about
having connection threads tied up handling blocking IO.  We have some
good options: adjust the readahead buffer, upto maxinput if needed,
and use connection thread pools to partition resource intensive URLs.

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 16.06.2005 um 23:35 schrieb Vlad Seryakov:

> I understand that but my question is:
> How and when all content will be downloaded?
> By the driver? By the conn thread on first call?
>

The driver will load up to maxreadahead.

At the first call to Ns_ConnContent, the rest
will be sinked from the socket into the buffer
(until it fills) and the rest will be sinked
into the mmaped temp file. This is now done
in the connection thread.

> Currently driver reads everything into memory.
>

Yes, which is a problem. So it will be split between
the driver thread (up to maxreadahead) and connection
thread (the rest, if available).

Zoran

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

>
> There's a maxreadahead parameter which is <= maxinput.  When a request
> arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are
> read into a buffer by the driver thread before the conn is passed to a
> conn thread.

Actually, there is already (kind of) maxreadahead param: bufsize.
It is used as the buffer size when starting the read from socket
and it defaults to 16K. We can use and/or rename this to maxreadahead
to better illustrate the fact what is being done.

Zoran

Re: [naviserver-devel] uploading large files

[Vlad S. <vl...@cr...>]

> 
> You can check ns_conn contentlength in advance and
> decide to get all the content or to discard it (because too large).
> You can use ns_conn setmaxinput to some large(r) value and then
> call ns_queryget.
> The ns_queryget will call Ns_ConnContent and this is now clever
> to either return the content from memory (if < maxreadahead)
> or do more complex stuff by mapping file and sinking all the
> content in a memmaped file. The returned byte array for futher
> parsing would be just from the mmaped file. The callers of the
> Ns_ConnContent will not see the difference.
> 

I understand that but my question is:
How and when all content will be downloaded?
By the driver? By the conn thread on first call?

Currently driver reads everything into memory.

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 16.06.2005 um 22:52 schrieb Vlad Seryakov:

>
> When you call ns_queryget/ns_getform, NS parses the input and split  
> content into temporaty file in case of attachements. When sending  
> multipart/data content and i called ns_queryget at the beginning of my
> connection script, parser needs all the data available, big  
> attachments can be the first one and other query parameters at the  
> end. How lazy
> parsing will happen? Will my ns_queryget wait while the content
> is being downloaded?

Lets see if I understand...

You can check ns_conn contentlength in advance and
decide to get all the content or to discard it (because too large).
You can use ns_conn setmaxinput to some large(r) value and then
call ns_queryget.
The ns_queryget will call Ns_ConnContent and this is now clever
to either return the content from memory (if < maxreadahead)
or do more complex stuff by mapping file and sinking all the
content in a memmaped file. The returned byte array for futher
parsing would be just from the mmaped file. The callers of the
Ns_ConnContent will not see the difference.

Stephen, did I got it right?

Zoran

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 16.06.2005 um 22:13 schrieb Stephen Deasey:

>
> At any time before the registered proc asks for the content it can
> check the content length header and decide whether it is too large to
> accept.  You could imagine setting a low global maxinput, and a call
> such as Ns_ConnSetMaxInput() which a registered proc could call to
> increase the limit for that connection only.  The advantage over the
> limits scheme in 4.1 is that the code which checks the size of the
> content and processes it is kept together, rather than having to
> pre-decalare maxinput sizes for arbitrary URLs in the config file.
>


Aha... do:

   ns_conn setmaxinput

before we do

   ns_conn content

to get the content of the request.

Hm... so we'd keep the maxinput reasonably *small* for most
of the requests and will re-set it to a larger value *before*
requesting the content when we anticipate large file upload.

I think I'm beginning to understand....
Yes, this makes sense.

Zoran

Re: [naviserver-devel] uploading large files

[Vlad S. <vl...@cr...>]

When you call ns_queryget/ns_getform, NS parses the input and split=20
content into temporaty file in case of attachements. When sending=20
multipart/data content and i called ns_queryget at the beginning of my
connection script, parser needs all the data available, big attachments=20
can be the first one and other query parameters at the end. How lazy
parsing will happen? Will my ns_queryget wait while the content
is being downloaded?

Stephen Deasey wrote:
> On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
>>Hi!
>>
>>I ran into several problems while debuggging a problem at the
>>customer site related to uploading files and size of those files.
>>
>>As it looks like, the NS/AS spools all the incoming content into
>>the memory (urks!). Depending on the content-type, the content
>>may be parsed, as in case of multipar/form-data.
>>
>>This is very convenient (you need not fiddle with the low-level
>>stuff) but it can be extremely inefficient and bloat the memory
>>footprint of the server if somebody likes to upload 10's or 100's
>>of MB of data.
>>
>>I have examined the recent changes in the AS done by Jim which
>>tackle this issue. He's basically using the "maxinput" config
>>option to decide wether to stuff everything into memory (as usual)
>>or, in the case input exceeds the maxinput level,  stuff the
>>input into a temporary file. But... the programmer is then left
>>alone and has to parse the incoming input by himself by using
>>ns_conn contentchannel command to read the content.
>>Even more, the form machinery which is basically parsing the
>>multipart data still operates on the connPtr->content which
>>does not reflect the entire content (the excessive is spooled
>>into a temp file!).
>>
>>Hm... this seems pretty unusable for this particular case to me.
>>
>>Therefore, and as usual, I have a idea/suggestion how to improve
>>this...
>>
>>I would take the same direction as Jim did, but will simply mmap the
>>temp-file to the connPtr->content! This will make all the rest
>>of the code work as usual. On the connection end, the file will
>>be unmmaped and there you go.
>>
>>One can even explore the memmap machinery and see if we can
>>entirely drop the temp-file and use the system paging for that:
>>so if the input exceeds the maxinput, we just mmap /dev/zero file
>>I believe something like that should be possible but will have to
>>double-check if this is true.
>=20
>=20
>=20
> It was at one point implemented with mmap:
>=20
> http://cvs.sourceforge.net/viewcvs.py/aolserver/aolserver/nsd/driver.c?=
rev=3D1.34&view=3Dmarkup
>=20
>=20
> Spooling large requests to disk is clearly necessary.  Almost always,
> a large request needs to be stored in a file server-side anyway, so it
> might as well go there from the start.
>=20
> The only problem I see is that the calls to open a file and write
> content to disk are blocking, and they happen in the context of the
> driver thread which is busy multiplexing many non-blocking operations
> on sockets.  Should one of the calls used to spool to disk block,
> everything comes to a stop.
>=20
> The version 3.x model was to read-ahead up to the end of headers in
> the driver thread, then pass the conn to a conn thread.  As far as I
> remember, the conn thread would then spool all content to disk, but
> this could have been on-demand and it might have been only file-upload
> data. A programmer could then access the spooled data using
> Ns_ConnRead(), Ns_ConnReadLine() etc., or they would be called for you
> by ns_getform etc.  The thing to note however is that all the blocking
> calls happened in separate conn threads.
>=20
> In early version 4.0 the model was changed so that the driver thread
> would read-ahead all data (up to Content-Length bytes) before the conn
> was passed to a conn thread.  In a point release a limit was
> introduced to avoid the obvious DOS attack.  This allowed an easier
> interface to the data for programmers: Ns_ConnContent() which returns
> a pointer to an array of bytes.  Ns_ConnReadLine() etc. are no longer
> used and are currently broken.
>=20
> Version 4.1 work seems to be trying to tackle the problem of what
> happens when large files are uploaded.  The version 4.0 model work
> extremely well for HTTP POST data up to a few K in size, but as you've
> noticed it really bloats memory when multi-megabyte files are
> uploaded.  This version also introduces limits, which are a
> URL-specific way of pre-declaring the maxupload size and some other
> parameters.
>=20
>=20
> So anyway, here's my idea of how it should work:
>=20
> There's a maxreadahead parameter which is <=3D maxinput.  When a reques=
t
> arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are
> read into a buffer by the driver thread before the conn is passed to a
> conn thread.
> =20
> The conn thread runs the registered proc for that URL.  If that
> procedure does not try to acces the content, then when the conn is
> returned to the driver thread any content > maxreadahead is drained.
>=20
> If the registered proc does try to access the content via e.g.
> Ns_ConnContent() (and I think this would be the main method, used
> underneath by all others) and the content is <=3D maxreadahead, then a
> pointer to the readahead buffer is returned.
>=20
> If the content is accessed and it is > maxreadahead, a temp file is
> mmaped, the readahead buffer is dumped to it, and the remaining bytes
> are read from the socket, possibly blocking many times, and dumped to
> the mmaped file, again possibly blocking.  A call to Ns_ConnContent()
> returns a pointer to the mmaped bytes of the open file.
>=20
> At any time before the registered proc asks for the content it can
> check the content length header and decide whether it is too large to
> accept.  You could imagine setting a low global maxinput, and a call
> such as Ns_ConnSetMaxInput() which a registered proc could call to
> increase the limit for that connection only.  The advantage over the
> limits scheme in 4.1 is that the code which checks the size of the
> content and processes it is kept together, rather than having to
> pre-decalare maxinput sizes for arbitrary URLs in the config file.
>=20
> This is similar to 4.1 except the task of overflowing to disk is moved
> to the conn thread and is lazy.  The mental model is also slightly
> different, the assumption is that content goes to disk, but theres a
> cache for read-ahead which may be enough to hold everything.  In 4.1
> it's everything is read into memory, unless it's large in which case
> it overflows to disk.
>=20
>=20
> How does that sound?
>=20
>=20
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id=16492&op=CCk
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

--=20
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] uploading large files

[Zoran V. <zv...@ar...>]

Am 16.06.2005 um 22:13 schrieb Stephen Deasey:

>
> It was at one point implemented with mmap:
>
> http://cvs.sourceforge.net/viewcvs.py/aolserver/aolserver/nsd/ 
> driver.c?rev=1.34&view=markup
>

Aha! But it was taken out for some reason...
It would be good to know why?

>
> Spooling large requests to disk is clearly necessary.  Almost always,
> a large request needs to be stored in a file server-side anyway, so it
> might as well go there from the start.
>
> The only problem I see is that the calls to open a file and write
> content to disk are blocking, and they happen in the context of the
> driver thread which is busy multiplexing many non-blocking operations
> on sockets.  Should one of the calls used to spool to disk block,
> everything comes to a stop.

Oh yes. This is very right.

>
> The version 3.x model was to read-ahead up to the end of headers in
> the driver thread, then pass the conn to a conn thread.  As far as I
> remember, the conn thread would then spool all content to disk, but
> this could have been on-demand and it might have been only file-upload
> data. A programmer could then access the spooled data using
> Ns_ConnRead(), Ns_ConnReadLine() etc., or they would be called for you
> by ns_getform etc.  The thing to note however is that all the blocking
> calls happened in separate conn threads.

Yes. I remember this because the multipart data was parsed by a
Tcl script. Not very speedy but it worked surprisingly stable.

>
> In early version 4.0 the model was changed so that the driver thread
> would read-ahead all data (up to Content-Length bytes) before the conn
> was passed to a conn thread.  In a point release a limit was
> introduced to avoid the obvious DOS attack.  This allowed an easier
> interface to the data for programmers: Ns_ConnContent() which returns
> a pointer to an array of bytes.  Ns_ConnReadLine() etc. are no longer
> used and are currently broken.

But this brought the memory bloat as a side-effect
(you can make all happy) :-)

>
> Version 4.1 work seems to be trying to tackle the problem of what
> happens when large files are uploaded.  The version 4.0 model work
> extremely well for HTTP POST data up to a few K in size, but as you've
> noticed it really bloats memory when multi-megabyte files are
> uploaded.  This version also introduces limits, which are a
> URL-specific way of pre-declaring the maxupload size and some other
> parameters.

But still, just spools the data into temp-file at the very sensitive
point (in the driver.c) as you already pointed-out above.

>
>
> So anyway, here's my idea of how it should work:

I'm all ears :-)

>
> There's a maxreadahead parameter which is <= maxinput.  When a request
> arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are
> read into a buffer by the driver thread before the conn is passed to a
> conn thread.
>
> The conn thread runs the registered proc for that URL.  If that
> procedure does not try to acces the content, then when the conn is
> returned to the driver thread any content > maxreadahead is drained.
>
> If the registered proc does try to access the content via e.g.
> Ns_ConnContent() (and I think this would be the main method, used
> underneath by all others) and the content is <= maxreadahead, then a
> pointer to the readahead buffer is returned.
>
> If the content is accessed and it is > maxreadahead, a temp file is
> mmaped, the readahead buffer is dumped to it, and the remaining bytes
> are read from the socket, possibly blocking many times, and dumped to
> the mmaped file, again possibly blocking.  A call to Ns_ConnContent()
> returns a pointer to the mmaped bytes of the open file.

But this is now happening outside the driver and in the connection
thread, ok.

>
> At any time before the registered proc asks for the content it can
> check the content length header and decide whether it is too large to
> accept.  You could imagine setting a low global maxinput, and a call
> such as Ns_ConnSetMaxInput() which a registered proc could call to
> increase the limit for that connection only.  The advantage over the
> limits scheme in 4.1 is that the code which checks the size of the
> content and processes it is kept together, rather than having to
> pre-decalare maxinput sizes for arbitrary URLs in the config file.

Hm... this is the only part I do not understand.

>
> This is similar to 4.1 except the task of overflowing to disk is moved
> to the conn thread and is lazy.  The mental model is also slightly
> different, the assumption is that content goes to disk, but theres a
> cache for read-ahead which may be enough to hold everything.  In 4.1
> it's everything is read into memory, unless it's large in which case
> it overflows to disk.
>
>
> How does that sound?
>

Summary (my understanding):

This is a kind of marriage between the 3.0 and 4.0 strategies as I see.
Up to maxreadahead, it is behaving like 4.0 (loads all in memory) and
above maxreadahead it behaves like 3.0 (spools to disk).
The most significant fact is that potentially blocking calls while  
spooling
to disk are taken out of the driver and done in the connection thread
(where it does not matter that much) and that the process is lazy
(done only when content is actually requested and not in advance).

Yes, this sound fine, indeed. The only think I'd have yet to
understand is the twiddling with the maxinput value on per-connection
basis. I do not know *when* this would happen. Can you give a simple
practical example?


Zoran

Re: [naviserver-devel] uploading large files

[Stephen D. <sd...@gm...>]

On 6/16/05, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi!
>=20
> I ran into several problems while debuggging a problem at the
> customer site related to uploading files and size of those files.
>=20
> As it looks like, the NS/AS spools all the incoming content into
> the memory (urks!). Depending on the content-type, the content
> may be parsed, as in case of multipar/form-data.
>=20
> This is very convenient (you need not fiddle with the low-level
> stuff) but it can be extremely inefficient and bloat the memory
> footprint of the server if somebody likes to upload 10's or 100's
> of MB of data.
>=20
> I have examined the recent changes in the AS done by Jim which
> tackle this issue. He's basically using the "maxinput" config
> option to decide wether to stuff everything into memory (as usual)
> or, in the case input exceeds the maxinput level,  stuff the
> input into a temporary file. But... the programmer is then left
> alone and has to parse the incoming input by himself by using
> ns_conn contentchannel command to read the content.
> Even more, the form machinery which is basically parsing the
> multipart data still operates on the connPtr->content which
> does not reflect the entire content (the excessive is spooled
> into a temp file!).
>=20
> Hm... this seems pretty unusable for this particular case to me.
>=20
> Therefore, and as usual, I have a idea/suggestion how to improve
> this...
>=20
> I would take the same direction as Jim did, but will simply mmap the
> temp-file to the connPtr->content! This will make all the rest
> of the code work as usual. On the connection end, the file will
> be unmmaped and there you go.
>
> One can even explore the memmap machinery and see if we can
> entirely drop the temp-file and use the system paging for that:
> so if the input exceeds the maxinput, we just mmap /dev/zero file
> I believe something like that should be possible but will have to
> double-check if this is true.


It was at one point implemented with mmap:

http://cvs.sourceforge.net/viewcvs.py/aolserver/aolserver/nsd/driver.c?rev=
=3D1.34&view=3Dmarkup


Spooling large requests to disk is clearly necessary.  Almost always,
a large request needs to be stored in a file server-side anyway, so it
might as well go there from the start.

The only problem I see is that the calls to open a file and write
content to disk are blocking, and they happen in the context of the
driver thread which is busy multiplexing many non-blocking operations
on sockets.  Should one of the calls used to spool to disk block,
everything comes to a stop.

The version 3.x model was to read-ahead up to the end of headers in
the driver thread, then pass the conn to a conn thread.  As far as I
remember, the conn thread would then spool all content to disk, but
this could have been on-demand and it might have been only file-upload
data. A programmer could then access the spooled data using
Ns_ConnRead(), Ns_ConnReadLine() etc., or they would be called for you
by ns_getform etc.  The thing to note however is that all the blocking
calls happened in separate conn threads.

In early version 4.0 the model was changed so that the driver thread
would read-ahead all data (up to Content-Length bytes) before the conn
was passed to a conn thread.  In a point release a limit was
introduced to avoid the obvious DOS attack.  This allowed an easier
interface to the data for programmers: Ns_ConnContent() which returns
a pointer to an array of bytes.  Ns_ConnReadLine() etc. are no longer
used and are currently broken.

Version 4.1 work seems to be trying to tackle the problem of what
happens when large files are uploaded.  The version 4.0 model work
extremely well for HTTP POST data up to a few K in size, but as you've
noticed it really bloats memory when multi-megabyte files are
uploaded.  This version also introduces limits, which are a
URL-specific way of pre-declaring the maxupload size and some other
parameters.


So anyway, here's my idea of how it should work:

There's a maxreadahead parameter which is <=3D maxinput.  When a request
arrives with Content-Length > 0 && < maxinput, maxreadahead bytes are
read into a buffer by the driver thread before the conn is passed to a
conn thread.
=20
The conn thread runs the registered proc for that URL.  If that
procedure does not try to acces the content, then when the conn is
returned to the driver thread any content > maxreadahead is drained.

If the registered proc does try to access the content via e.g.
Ns_ConnContent() (and I think this would be the main method, used
underneath by all others) and the content is <=3D maxreadahead, then a
pointer to the readahead buffer is returned.

If the content is accessed and it is > maxreadahead, a temp file is
mmaped, the readahead buffer is dumped to it, and the remaining bytes
are read from the socket, possibly blocking many times, and dumped to
the mmaped file, again possibly blocking.  A call to Ns_ConnContent()
returns a pointer to the mmaped bytes of the open file.

At any time before the registered proc asks for the content it can
check the content length header and decide whether it is too large to
accept.  You could imagine setting a low global maxinput, and a call
such as Ns_ConnSetMaxInput() which a registered proc could call to
increase the limit for that connection only.  The advantage over the
limits scheme in 4.1 is that the code which checks the size of the
content and processes it is kept together, rather than having to
pre-decalare maxinput sizes for arbitrary URLs in the config file.

This is similar to 4.1 except the task of overflowing to disk is moved
to the conn thread and is lazy.  The mental model is also slightly
different, the assumption is that content goes to disk, but theres a
cache for read-ahead which may be enough to hold everything.  In 4.1
it's everything is read into memory, unless it's large in which case
it overflows to disk.


How does that sound?

[naviserver-devel] uh, sorry...

[Zoran V. <zv...@ar...>]

Hi!

I just checked in the maxpost changes and all of the
identation changes I DID NOT WANT to chekin at that
time have crept in! Bah!

This makes difficult to read the diff and this is bad.
I'd normally first ident, checkin and then make functional
changes. But this time I just plain forgot to do that in
two distinct steps.

I will be more careful next time, promise.

Zoran

Re: [naviserver-devel] maxinput/maxpost: why both?

[Zoran V. <zv...@ar...>]

Am 16.06.2005 um 20:05 schrieb Stephen Deasey:

>
>
> Yeah, this broken test looks redundant.
>
>

OK. So we have a consensus. I will trash it out now.
Zoran