Re: [naviserver-devel] Upload Progress

[Gustaf N. <ne...@wu...>]

Stephen Deasey wrote:
>> The driver thread gets the connection and reads all up to maxreadahead.
>> It then passes the connection to the conn thread.
>> The conn thread decides to run that connection entirely (POST of a
>> simple form
>> or GET, all content read-in) OR it decides to pass the connection to the
>> spooling thread(s).
>>     
>
>
> What happens to the conn thread after this? It can't wait for
> completion, that would defeat the purpose.
>
> Do traces (logging etc.) run now, in the conn thread, or later in a
> spool thread?  If logging runs now, but the upload fails, the log will
> be wrong.  If traces run in the spool threads they may block.
>
>   
In my setup, we have currently the spooling thread just for sending data.
The code sent yesterday is a replacement for ns_returnfile. The connection
thread finishes asynchronously, before all data is spooled. This means, 
that the
connection thread finishes after it delegates the file delivery to the 
spool-thread.
In order to get the correct sent-content-length in the
logfile, i had to set (somewhat optimistic) the sent-contentlength manually.
The traces are run immediately after the delegation.

So the request lifecycle (state flow) for get request for a large file is
currently

  accept(DT) -> preauth(CT) -> postauth(CT) -> procs(CT)
    -> trace(CT) -> spool(ST)

DT: driver thread, CT: connection thread, ST: spool thread

To make the implementation cleaner, it would preferable to
provide means to pass the connection + context back to a
connection thread that should run the final trace

  accept(DT) -> preauth(CT1) -> postauth(CT1) -> procs(CT1)
    -> spool(ST) -> trace(CT2)

i would think, that if we can solve this part cleanly, the upload
would work with the same mechanism. if the states are handled
explicitely, one could think about a tcl command
  ns_enqueue $state $client_data
that will enqueue a job for the connection thread pools,
containing the state, file where the connection should continue
together with a tcl stucture client_data that passes around
connection specific context information (e.g user_id, ...) and
the information from ns_conn (socket, head info, ..).

For the example above, there would be an (implicit)
  ns_enqueue preauth ""
issued from the driver thread, an
  ns_spool send $filename (or $fd) $context
to pass the control to the spooling-thread, an
  ns_enqueue trace $context
issued from the spooling trace to pass control back to
a connection thread.

> If further processing must be done after upload, does the spool thread
> pass control back to a conn thread?  Does this look like a new
> request? The state of any running Tcl script will be lost at this
> point (the conn thread will have cleaned up after hand-off to a spool
> thread, right?).
>   
upload could be done with the same machanisms_

1) driver thread: request-head processing
2) start connection thread with a new filter (e.g. request-head)
3) the request-head filter will call pass control to the spooling-thread
    (e.g. ns_spool receive $length $context)
4) when the spool file is fully received (e.g. fcopy -command-callback)
    it enqueues the request for the connection threads (e.g.
    ns_enqueue preauth $context).
5) the connection thread obtains the context  starts request processing
    as usual in the preauth state (start preauth, postauth, trace, ...
    unless control is passed back to the spool threads)

So the request lifecycle (state flow) for a file upload could be:

  accept(DT) -> request-head(CT1) -> spool(ST)
     -> preauth(CT2) -> postauth(CT2) -> procs(CT2) -> trace(CT2)

i would think that with the two commands ns_enqueue and ns_spool,
which switch the control flow between connection threads and
spooling threads and copy connection state information (from ns_conn)
and client_data, one would have a very flexible framework.

-gustaf

PS: this model would be still compatible with aolserver, but would require
that e.g. authentification code will be called from the request-head filter
callback as well as from the preauth callback (maybe avoided by data
in the client_data) Moving preauth and postauth
to CT1 would be logically cleaner, but might be incompatible with aolserver,
since these callback might want already to access the posted data.