Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 09:21 schrieb Stephen Deasey:

>
> POST request with small amounts of data are really common.  Think of a
> user logging in via a web form.  Couple of bytes.
>
> The way you've coded it at the mo (and I realize it's just a first
> cut), all requests with more than 0 bytes of body content will cause
> the upload stats code to fire.  That's why I suggested we may want to
> have some lower threshold, or restrict it to certain URLs via
> urlspecific data.  We don't want to track every form submission.

No. That is of course not good.

>>
>> One problem i see with lazy uploads that if you have multiple clients
>> doing large POSTs, spawning multiple clients for a long time reading
>> that content will waste resources, each conn thread is heavy, with  
>> Tcl
>> interp.
>
>
> Not necessarily.  You can create another thread pool and then ensure
> that those threads never run Tcl code.  Remember, Tcl interps are
> allocated lazily on first use. We could also adjust stacksize per
> thread pool.

But then, how are you going to interpret the stats? You need to
run some Tcl code for that and this will load the Tcl interp?


>
>
>> Using driver thread reading small chunks from the connections
>> and putting it into file will keep everything smooth. But with small
>> uploads on fast network this may not be an issue, so it needs a
>> compromize solution here, may be configurable options. Currently,
>> spooling into file can be disabled/enabled, lazy spooling may be
>> implemented similar way. Actually, lazy file spooling can be easily
>> done, because even Ns_ConnContent calls SockRead which does  
>> spooling, we
>> just need to introduce an option that tells how much we should  
>> spool in
>> the main thread and then continue in the conn thread.
>
>
> We already have maxreadahead, which is the amount of data read by the
> driver thread into a memory buffer.  I think you're either happy
> letting the driver thread block writing to disk, or you're not.  Why
> would you set a threshold on this?

Right. We already have that knob: maxreadahead. It is the question
what happens when we reach that point. At the moment, extra data is
spooled into the file and file is mmaped at the end of reading
when all the content from the client is received. I'm not that happy
with the driver thread pumping the data into the file as this might
affect the overall performance, therefore I suggested kernel AIO.
This feature of OS is developed precisely for that purpose.

Zoran