naviserver-devel

[naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/30/05, Vlad Seryakov <vl...@cr...> wrote:
> On that note i have another idea i'd like to discuss before i even start
> coding a prototype. There is thread on aolserver mailing list about
> upload prgoress, so i thought would it be a good idea to have global
> url-specific cache of all uploads, let's say, all requestes with
> content-length > 0. It will have only 2 alues, current size and total
> length and will last only for the time of upload. It will belong to Sock
> structure of the Request, so on close it will be freed as well.
>
> Making it url-specific will give Web developer ability to generate
> uniaue urls for upload and then request statistics, requests with the
> same url will not override each other, server will update statistics for
>   the first created cache only, subsequent uploads with the same url
> will not show anything or show old values which is fine for security
> reasons.
>
> Overhead is minimal and it will add one new commmand like
> ns_upload_stats url. SockRead will handle it all, so no other places are
> affected.
>
> Is it worth trying?


I think we talked about this before, but I can't find it in the
mailing list archive.  Anyway, the problem with recording the upload
process is all the locking that's required.  You could minimize this,
e.g. by only recording uploads above a certain size, or to a certain
URL.

It reminds me of a similar problem we had.  Spooling large uploads to disk:

https://sourceforge.net/mailarchive/forum.php?thread_id=3D7524448&forum_id=
=3D43966

Vlad implemented the actual spooling, but moving that work into the
conn threads, reading lazily, is still to be done.

Lazy uploading is exactly the hook you need to track upload progress.=20
The client starts to upload a file.  Read-ahead occurs in the driver
thread, say 8k.  Control is passed to a conn thread, which then calls
Ns_ConnContent().  The remaining content is read from the client, in
the context of the conn thread and so not blocking the driver thread,
and perhaps the content is spooled to disk.

To implement upload tracking you would register a proc for /upload
which instead of calling Ns_ConnContent(), calls Ns_ConnRead()
multiple times, recording the number of bytes read in the upload
tracking cache, and saving the data to disk or wherever.

A lot more control of the upload process is needed, whether it be to
control size, access, to record stats, or something we haven't thought
of yet.  If we complete the work to get lazy reading from the client
working, an upload tracker will be an easy module to write.

Does this make sense?

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 11:58 schrieb Stephen Deasey:

>
> I think we talked about this before, but I can't find it in the
> mailing list archive.  Anyway, the problem with recording the upload
> process is all the locking that's required.  You could minimize this,
> e.g. by only recording uploads above a certain size, or to a certain
> URL.

Hm... not very useful.

>
> It reminds me of a similar problem we had.  Spooling large uploads  
> to disk:
>
> https://sourceforge.net/mailarchive/forum.php? 
> thread_id=7524448&forum_id=43966
>
> Vlad implemented the actual spooling, but moving that work into the
> conn threads, reading lazily, is still to be done.

Yes. This is what I wanted to do but never got to it because
of the internal work... huh.

>
> Lazy uploading is exactly the hook you need to track upload progress.
> The client starts to upload a file.  Read-ahead occurs in the driver
> thread, say 8k.  Control is passed to a conn thread, which then calls
> Ns_ConnContent().  The remaining content is read from the client, in
> the context of the conn thread and so not blocking the driver thread,
> and perhaps the content is spooled to disk.

But that would occupy the conn thread for ages, right?
I can imagine several slow-line large-file uploads could
consume many of those. Wasn't that the reason everything
was moved in the driver thread for the 4.0 ?

What I have/had in mind is aync writes from the driver thread.
Most of the OS'es have this feature (kaio) so we can employ it.
The question of locking, however still remains in such case.
So the decision has to be made on: what is cheaper? Locking
or spooling to disk out of the conn thread? I have no real-life
experience but I'm inclined to believe that spooling out of the
conn thread would be more costly.

What do you think?

Zoran

Re: [naviserver-devel] Upload Progress

[Gustaf N. <ne...@wu...>]

Zoran Vasiljevic wrote:
>
> But that would occupy the conn thread for ages, right?
> I can imagine several slow-line large-file uploads could
> consume many of those. Wasn't that the reason everything
> was moved in the driver thread for the 4.0 ?
>
> What I have/had in mind is aync writes from the driver thread.
> Most of the OS'es have this feature (kaio) so we can employ it.
> The question of locking, however still remains in such case.
> So the decision has to be made on: what is cheaper? Locking
> or spooling to disk out of the conn thread? I have no real-life
> experience but I'm inclined to believe that spooling out of the
> conn thread would be more costly.
>
> What do you think?
i have not much looked into the c-code, so please forgive, if my
mail here is obviously wrong or naive.

One thing, which i dislike about the aolserver for file uploads is that
there is no hook for early permission or quota checking; one has to
upload the whole file before aolserver is able to give some feedback.
i see this even more important than the progress bar.

What about the following model:

A) HTTP header (up to empty line) -> driver thread
 
B) Then passing control+socket to a separate receiving thread that
  1) issues a callback at the start (e.g. for permission checking) 
  2) does async i/o to receive the message body (e.g. PUT, POST request)
    here, one can issue as well callbacks per async block, and decide
    fom tcl, whether one would like to spool to memory or disc.

C) When end of message received, pass control to connection thread as usual.

The receiving thread is like a connection thread with tcl stuff; when the
receiving thread is implemented via libthtread, there won't be probably
much locking. Only one receiving thread will be necessary.
The overall model could be quite compatible with the aolserver, since the
connection tread is invoked when all data is received. It would be necessary
to pass context information from the receiving thread to the connection 
thread
to avoid double permission checking.

It could as well make sense, to pass control to the receiving thread only
for chunked content. Maybe, the receiving thread is not needed at all,
since the driver thread could handle everything as well.

-gustaf

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 15:27 schrieb Gustaf Neumann:

>  Maybe, the receiving thread is not needed at all,
> since the driver thread could handle everything as well.

This is what I ment. Driver thread is very hot as it
does not (should not?) find iself into any blocking at all.
At the moment it does, as it writes into a mmaped file
when somebody uploads a thing larger that would fit in
a predefined buffer.  But this should be avoided. Precisely
that's the reason why I'm contemplating the kaio as this would
allow the driver thread to run at highest possible speed.

The locking.

In order to get any meaningful statistics, there have to
be places to hook into that processing which will inevitably
lead to locks. But, OTOH, how frequent would that be?
We are talking about very short and relatively seldom lock
attempts. I can't say it out of the sleeve, but I somehow
have a feeling that locking would not be an issue here.
Or?

Other kind of processing (like quota enforcing etc).

This would/could be pretty difficult as I wouldn't like to run
any Tcl code from within the driver thread. So, if this need
be done, than we must thing of some bit toggling and signalling
the driver thread to abort/stop/whatever the upload process if needed.

Just my 2 cents.

Cheers
Zoran

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Gustaf Neumann <ne...@wu...> wrote:
> Zoran Vasiljevic wrote:
> >
> > But that would occupy the conn thread for ages, right?
> > I can imagine several slow-line large-file uploads could
> > consume many of those. Wasn't that the reason everything
> > was moved in the driver thread for the 4.0 ?
> >
> > What I have/had in mind is aync writes from the driver thread.
> > Most of the OS'es have this feature (kaio) so we can employ it.
> > The question of locking, however still remains in such case.
> > So the decision has to be made on: what is cheaper? Locking
> > or spooling to disk out of the conn thread? I have no real-life
> > experience but I'm inclined to believe that spooling out of the
> > conn thread would be more costly.
> >
> > What do you think?
> i have not much looked into the c-code, so please forgive, if my
> mail here is obviously wrong or naive.
>
> One thing, which i dislike about the aolserver for file uploads is that
> there is no hook for early permission or quota checking; one has to
> upload the whole file before aolserver is able to give some feedback.
> i see this even more important than the progress bar.


I agree. This is important.


> What about the following model:
>
> A) HTTP header (up to empty line) -> driver thread
>
> B) Then passing control+socket to a separate receiving thread that
>   1) issues a callback at the start (e.g. for permission checking)
>   2) does async i/o to receive the message body (e.g. PUT, POST request)
>     here, one can issue as well callbacks per async block, and decide
>     fom tcl, whether one would like to spool to memory or disc.
>
> C) When end of message received, pass control to connection thread as usu=
al.
>
> The receiving thread is like a connection thread with tcl stuff; when the
> receiving thread is implemented via libthtread, there won't be probably
> much locking. Only one receiving thread will be necessary.
> The overall model could be quite compatible with the aolserver, since the
> connection tread is invoked when all data is received. It would be necess=
ary
> to pass context information from the receiving thread to the connection
> thread
> to avoid double permission checking.
>
> It could as well make sense, to pass control to the receiving thread only
> for chunked content. Maybe, the receiving thread is not needed at all,
> since the driver thread could handle everything as well.


This is similar to the way the old Flash web server worked.  It had a
pool of IO threads which would make sure data on disk was in memory,
possibly blocking, before the main thread would try to write it out to
the client.  Multiple threads are used to get more throughput out of
the disk.  As one thread blocks, another can submit a request.

Your single thread helper mechanism above may not work the disk as
hard as multiple threads, but it does mean that the driver thread
doesn't have to block waiting for disk.

You've also moved quota checking etc. into the helper thread.  This
doesn't so much matter if there's just one, but with more than one you
have to remember that a Tcl interp will be allocated for each, and
they'll be just as heavyweight as normal conn threads.

We have a similarly problem with normal downloads, i.e. writing data
to the client, an mp3 file for example, ties up a heavyweight conn
thread.  A generic pool of IO threads might be useful in a number of
places.  Of course, this is just a simulation of AIO to disk, but see
my other email for why AIO is not so easy.


AOLserver HEAD has a new filter type, NS_FILTER_PREQUEUE, which gets
called just before the connection is queued for processing by a conn
thread.  The unique thing about this filter type is that the
registered code runs in the context of the driver thread.=20
Unfortunately that's after all the data has been read and spooled, but
we could implement it such that they ran just after all headers have
been received.

If you want to check quotas by querying a database then then this
doesn't help -- your db query will block the driver thread.  These
filters will have to be coded carefully.  We could also implement
non-blocking db calls, and that's certainly something I'm interested
in, but it doesn't help right now.

However, if the code you want to run is short and non blocking,
prequeue filters we be a good way to run quota checks and such without
having to involve a conn thread at all.


Prequeue filters don't help with upload tracking, nor with actually
spooling large uploads to disk.

Re: [naviserver-devel] Upload Progress

[Gustaf N. <ne...@wu...>]

Stephen Deasey wrote:
>
> Your single thread helper mechanism above may not work the disk as
> hard as multiple threads, but it does mean that the driver thread
> doesn't have to block waiting for disk.
>   
it does not necessary have to write to the disk.
> You've also moved quota checking etc. into the helper thread.  This
> doesn't so much matter if there's just one, but with more than one you
> have to remember that a Tcl interp will be allocated for each, and
> they'll be just as heavyweight as normal conn threads.
>
> We have a similarly problem with normal downloads, i.e. writing data
> to the client, an mp3 file for example, ties up a heavyweight conn
> thread.  A generic pool of IO threads might be useful in a number of
> places.  Of course, this is just a simulation of AIO to disk, but see
> my other email for why AIO is not so easy.
>   
we are using on our production site a background delivery for large 
files (up to 4 mio requests
per day, up to 30 GB/day, up to 1200 concurrent users, oacs) which does 
not block the
connection threads. i did this with some help from zoran and it works 
like a champ
(single spooling thread, using ns_conn channel + libthread with 
thread::transfer and thread:;send
code below). This bgdelivery implementation has already reduced the 
memory footprint of
our server, frees us from slow-download-DOS-"attacks" and yielded no 
measurable
performance degradation.
> AOLserver HEAD has a new filter type, NS_FILTER_PREQUEUE, which gets
> called just before the connection is queued for processing by a conn
> thread.  The unique thing about this filter type is that the
> registered code runs in the context of the driver thread. 
> Unfortunately that's after all the data has been read and spooled, but
> we could implement it such that they ran just after all headers have
> been received.
>   
interesting
> If you want to check quotas by querying a database then then this
> doesn't help -- your db query will block the driver thread.  These
> filters will have to be coded carefully.  We could also implement
> non-blocking db calls, and that's certainly something I'm interested
> in, but it doesn't help right now.
>   
true.
> However, if the code you want to run is short and non blocking,
> prequeue filters we be a good way to run quota checks and such without
> having to involve a conn thread at all.
>   
also the singed cookie checking of oacs might be already to expensive 
for a single thread
(no matter whether is happens in the driver or in the model suggested in 
my last mail).

The more i think about the problem, the general case (expensive 
permission + quota checking)
of uploads should be handled by connection threads. In order to avoid 
having these
sitting more or less idle around, one cal use a spooling thread (or a 
few of these).
What i have seen, all ajax file upload bars are associating an
Upload-ID with the file upload and query later via GET requests the 
status of these.
This means, that ordinary requests must be able to query the status... 
This is quite similar
to my request monitor part in oacs-head (you can see this in mannheim in 
action, where it is
publically accessible https://dotlrn.uni-mannheim.de/request-monitor/)

A revised model:
 1) request HEAD processing alway in the driver thread
 2) for incoming data processing delegation to a connection thread with 
a new filter type
     (can do db-queries etc)
 3) delegation to a spooling thread (where it can be monitored)
 4) delegation to a connection thread ala  aolserver.

when end-of-file/error occurs in the spooling thread, it would require a 
tcl-command
to enqueue the request as connection thread.

Actually, this could be done only when there is a POST/PUT request with a
Upload-ID as a query parameter, such that the programmer has full control
when this happen, there will be no penalty for small forms...

-gustaf

=====================================================
ad_library {
    Routines for background delivery of files

    @author Gustaf Neumann (ne...@wu...)
    @creation-date 19 Nov 2005
    @cvs-id $Id: background-delivery-procs.tcl,v 1.2 2005/12/30 00:07:23 
gustafn Exp $
}

::xotcl::THREAD create bgdelivery {
  set ::delivery_count 0

  proc deliver {ch filename context} {
    set fd [open $filename]
    fconfigure $fd -translation binary
    fconfigure $ch -translation binary
    fcopy $fd $ch -command [list end-delivery $filename $fd $ch]
    set ::running($ch,$filename) $context
    incr ::delivery_count
  }

  proc end-delivery {filename fd ch bytes args} {
    #ns_log notice "--- end of delivery of $filename, $bytes bytes 
written $args"
    if {[catch {close $ch} e]} {ns_log notice "bgdelivery, closing 
channel for $filename, error: $e"}
    if {[catch {close $fd} e]} {ns_log notice "bgdelivery, closing file 
$filename, error: $e"}
    unset ::running($ch,$filename)
  }
} -persistent 1

bgdelivery ad_forward running {
  Interface to the background delivery thread to query the currently 
running deliveries.
  @return list of key value pairs of all currently running background 
processes
} %self do array get running


bgdelivery ad_forward nr_running {
  Interface to the background delivery thread to query the number of 
currently running deliveries.
  @return number of currently running background deliveries
} %self do array size running

if {[ns_info name] eq "NaviServer"} {
  bgdelivery forward write_headers ns_headers
} else {
  bgdelivery forward write_headers ns_headers DUMMY
}

bgdelivery ad_proc returnfile {statuscode mime_type filename} {
  Deliver the given file to the requestor in the background. This proc 
uses the
  background delivery thread to send the file in an event-driven manner 
without
  blocking a request thread. This is especially important when large 
files are
  requested over slow (e.g. dial-ip) connections.
} {
  set size [file size $filename]
  if {[my write_headers $statuscode $mime_type $size]} {
    set ch [ns_conn channel]
    thread::transfer [my get_tid] $ch
    throttle get_context
    my do -async deliver $ch $filename \
    [list [throttle set requestor],[throttle set url] [ns_conn start]]
    ns_conn contentsentlength $size       ;# maybe overly optimistic
  }
}
 
 

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

> 
> What I have/had in mind is aync writes from the driver thread.
> Most of the OS'es have this feature (kaio) so we can employ it.
> The question of locking, however still remains in such case.
> So the decision has to be made on: what is cheaper? Locking
> or spooling to disk out of the conn thread? I have no real-life
> experience but I'm inclined to believe that spooling out of the
> conn thread would be more costly.

We d locking in nsd in a lot of places anyway, locking itself is not the 
issue if it does it for very short time, of course, everything is 
serialized.

The other solution could be special-formatted query arguments that will 
tell to do upload statistics, looks like a hack but why not, it is 
web/application server, it can introduce specific protocol extensions.

something like: /upload/url?arg1=1&arg2=2;ns_upload_stats=1

Another possible solution can be, pre-allocating maxconn upload structs 
and update them without locks, it is integer anyway, so no need to lock, 
4 byte write is never innterupted, usually it is 1 CPU instruction(true 
for Intel, maybe not for Sparc).

In the request for upload stat thread, we scan that array for matching 
url and return upload stats. So driver thread is writing only, other 
thread are reading only.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 18:20 schrieb Vlad Seryakov:

> Another possible solution can be, pre-allocating maxconn upload  
> structs and update them without locks, it is integer anyway, so no  
> need to lock, 4 byte write is never innterupted, usually it is 1  
> CPU instruction(true for Intel, maybe not for Sparc).

Nope. Need locking even for integers. I've seen crazy things...

OTOH, I'm not that concerned about the locking if it is short-lived
and not frequently done. I believe the "not frequently done" is the
hardest thing to judge.

Zoran

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

Could config option, like 1 second or 10Kbytes

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 18:20 schrieb Vlad Seryakov:
> 
>> Another possible solution can be, pre-allocating maxconn upload  
>> structs and update them without locks, it is integer anyway, so no  
>> need to lock, 4 byte write is never innterupted, usually it is 1  CPU 
>> instruction(true for Intel, maybe not for Sparc).
> 
> 
> Nope. Need locking even for integers. I've seen crazy things...
> 
> OTOH, I'm not that concerned about the locking if it is short-lived
> and not frequently done. I believe the "not frequently done" is the
> hardest thing to judge.
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 19:03 schrieb Vlad Seryakov:

> Could config option, like 1 second or 10Kbytes

Yup. For example. This could reduce locking attempts.
Seems fine to me.

Also, the blocking of the driver thread during spooling
into the file should be taken care. I wanted to look into
the kernel async IO (as it is available on Darwin/Sol/Linux).

Zoran 

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

> 
> Also, the blocking of the driver thread during spooling
> into the file should be taken care. I wanted to look into
> the kernel async IO (as it is available on Darwin/Sol/Linux).
> 

What about separate thread doing all spooling I/O, driver thread will 
send buffer to be written and immediately continue while spooling thread 
will do writes?


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 19:26 schrieb Vlad Seryakov:

> What about separate thread doing all spooling I/O, driver thread  
> will send buffer to be written and immediately continue while  
> spooling thread will do writes?

This is another option. I like the kaio because it saves you
yet-another-thread to manage. And it perfectly fits in the
non-blocking operation of the driver thread. One can evaluate
both options and then decide. Somehow I have the feeling that
kaio would be simpler to fit into the current code (I might
be wrong though)

Zoran

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

aio_read/aio_write system calls look like supported ubnder Linux

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 19:26 schrieb Vlad Seryakov:
> 
>> What about separate thread doing all spooling I/O, driver thread  will 
>> send buffer to be written and immediately continue while  spooling 
>> thread will do writes?
> 
> 
> This is another option. I like the kaio because it saves you
> yet-another-thread to manage. And it perfectly fits in the
> non-blocking operation of the driver thread. One can evaluate
> both options and then decide. Somehow I have the feeling that
> kaio would be simpler to fit into the current code (I might
> be wrong though)
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:

> aio_read/aio_write system calls look like supported ubnder Linux
>

yes.  most of modern os's support some kind of kaio.
I have checked solaris, linux and darwin and all do.
the problem with spawning yet another thread is
system resources. each new thread (on solaris)
results in an lwp which is not cheap. therfore I'd
opt to use aio for largefile spooling.

Zoran 

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

Tried it, nsd started crashing and i am guessing that the problem is 
combination of aio_write and mmap.
When is start spooling, i just submit aio_write and return immediately, 
so there are a lot of quick aio_write calls. By the time i reach mmap, 
it looks like it's always ahead of actual writing, so even when i am 
trying to check aio_error/aio_fsync in the NsGetRequest, i still get 
SIGBUS/SIGSEGV when i access reqPtr->content, looks like mmap and 
aio_write are out of sync. And there is no way to wait until all file 
buffers will be flushed, so manual separate thread implementation could 
be the only portable solution. Or i just do not understand aio_xxx 
things completely.

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
> 
>> aio_read/aio_write system calls look like supported ubnder Linux
>>
> 
> yes.  most of modern os's support some kind of kaio.
> I have checked solaris, linux and darwin and all do.
> the problem with spawning yet another thread is
> system resources. each new thread (on solaris)
> results in an lwp which is not cheap. therfore I'd
> opt to use aio for largefile spooling.
> 
> Zoran
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 04:43 schrieb Vlad Seryakov:

> Tried it, nsd started crashing and i am guessing that the problem  
> is combination of aio_write and mmap.
> When is start spooling, i just submit aio_write and return  
> immediately, so there are a lot of quick aio_write calls. By the  
> time i reach mmap, it looks like it's always ahead of actual  
> writing, so even when i am trying to check aio_error/aio_fsync in  
> the NsGetRequest, i still get SIGBUS/SIGSEGV when i access reqPtr- 
> >content, looks like mmap and aio_write are out of sync. And there  
> is no way to wait until all file buffers will be flushed, so manual  
> separate thread implementation could be the only portable solution.  
> Or i just do not understand aio_xxx things completely.

Hm... the idea is to mmap the file when all is read from the client.
So you'd have to aio_write into the file until all is received, then
at some point check the aio_status of the write (non-blocking) and
when it's done, revert to mmap the file. I do not believe you can
mix the aio_ with mmap.

Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 12:06 schrieb Zoran Vasiljevic:

>
> Am 02.01.2006 um 04:43 schrieb Vlad Seryakov:
>
>> Tried it, nsd started crashing and i am guessing that the problem  
>> is combination of aio_write and mmap.
>> When is start spooling, i just submit aio_write and return  
>> immediately, so there are a lot of quick aio_write calls. By the  
>> time i reach mmap, it looks like it's always ahead of actual  
>> writing, so even when i am trying to check aio_error/aio_fsync in  
>> the NsGetRequest, i still get SIGBUS/SIGSEGV when i access reqPtr- 
>> >content, looks like mmap and aio_write are out of sync. And there  
>> is no way to wait until all file buffers will be flushed, so  
>> manual separate thread implementation could be the only portable  
>> solution. Or i just do not understand aio_xxx things completely.
>
> Hm... the idea is to mmap the file when all is read from the client.
> So you'd have to aio_write into the file until all is received, then
> at some point check the aio_status of the write (non-blocking) and
> when it's done, revert to mmap the file. I do not believe you can
> mix the aio_ with mmap.

What I still do not know is: if you are allowed to issue multiple
outstanding aio_writes one after another... It might be that you
need to check wether the first aio_write is done before you attempt
the next one.

Zoran

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

I did very simple test, replaced write with aio_write and at the end 
checked aio_error/aio_return, they all returned 0 so mmap should work 
because file is synced. when i was doing aio_write i used aio_offset, so 
each aio_write would put data into separate region on the file.

Unfortunately i removed modified driver.c by accident, so i will have to 
do it again but something is not right in simply replacing write with 
aio_write and mmap. Without mmap, GetForm will have to read the file 
manually and parse it, it makes it more complicated and still, if writes 
are not complete we may get SIGBUS again.

The problem i see here i cannot be sure that all writes are completed, 
aio_error/aio_return are used to check only one last operation, not the 
batch of writes.

Zoran Vasiljevic wrote:
> 
> Am 02.01.2006 um 12:06 schrieb Zoran Vasiljevic:
> 
>>
>> Am 02.01.2006 um 04:43 schrieb Vlad Seryakov:
>>
>>> Tried it, nsd started crashing and i am guessing that the problem  is 
>>> combination of aio_write and mmap.
>>> When is start spooling, i just submit aio_write and return  
>>> immediately, so there are a lot of quick aio_write calls. By the  
>>> time i reach mmap, it looks like it's always ahead of actual  
>>> writing, so even when i am trying to check aio_error/aio_fsync in  
>>> the NsGetRequest, i still get SIGBUS/SIGSEGV when i access reqPtr- 
>>> >content, looks like mmap and aio_write are out of sync. And there  
>>> is no way to wait until all file buffers will be flushed, so  manual 
>>> separate thread implementation could be the only portable  solution. 
>>> Or i just do not understand aio_xxx things completely.
>>
>>
>> Hm... the idea is to mmap the file when all is read from the client.
>> So you'd have to aio_write into the file until all is received, then
>> at some point check the aio_status of the write (non-blocking) and
>> when it's done, revert to mmap the file. I do not believe you can
>> mix the aio_ with mmap.
> 
> 
> What I still do not know is: if you are allowed to issue multiple
> outstanding aio_writes one after another... It might be that you
> need to check wether the first aio_write is done before you attempt
> the next one.
> 
> Zoran
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 1/2/06, Vlad Seryakov <vl...@cr...> wrote:
> I did very simple test, replaced write with aio_write and at the end
> checked aio_error/aio_return, they all returned 0 so mmap should work
> because file is synced. when i was doing aio_write i used aio_offset, so
> each aio_write would put data into separate region on the file.
>
> Unfortunately i removed modified driver.c by accident, so i will have to
> do it again but something is not right in simply replacing write with
> aio_write and mmap. Without mmap, GetForm will have to read the file
> manually and parse it, it makes it more complicated and still, if writes
> are not complete we may get SIGBUS again.
>
> The problem i see here i cannot be sure that all writes are completed,
> aio_error/aio_return are used to check only one last operation, not the
> batch of writes.


We might have to drop mmap support anyway.  We're trying t accommodate
large files, so we may run out of address space.

This summer I was running thttpd to serve some iso images.  I started
getting busy errors when very little bandwidth was being used, and
with only 3 concurrent users.  It turns out that a couple of extra
images were published, and the users were all downloading different
ones.  As thttpd uses mmap to read the files, the process ran out of
address space.  It doesn't take much...

We still have to support mmap for reading in the case that a module
calls Ns_ConContent(), but otherwise we can use read/write etc.

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

As i see it by default nsd does not use mmaps, only for uploads, yes it 
might need to review this again, mmap for big movie/iso files is not 
appropriate and i am using nsd for those kinds of files most of the time.

Stephen Deasey wrote:
> On 1/2/06, Vlad Seryakov <vl...@cr...> wrote:
> 
>>I did very simple test, replaced write with aio_write and at the end
>>checked aio_error/aio_return, they all returned 0 so mmap should work
>>because file is synced. when i was doing aio_write i used aio_offset, so
>>each aio_write would put data into separate region on the file.
>>
>>Unfortunately i removed modified driver.c by accident, so i will have to
>>do it again but something is not right in simply replacing write with
>>aio_write and mmap. Without mmap, GetForm will have to read the file
>>manually and parse it, it makes it more complicated and still, if writes
>>are not complete we may get SIGBUS again.
>>
>>The problem i see here i cannot be sure that all writes are completed,
>>aio_error/aio_return are used to check only one last operation, not the
>>batch of writes.
> 
> 
> 
> We might have to drop mmap support anyway.  We're trying t accommodate
> large files, so we may run out of address space.
> 
> This summer I was running thttpd to serve some iso images.  I started
> getting busy errors when very little bandwidth was being used, and
> with only 3 concurrent users.  It turns out that a couple of extra
> images were published, and the users were all downloading different
> ones.  As thttpd uses mmap to read the files, the process ran out of
> address space.  It doesn't take much...
> 
> We still have to support mmap for reading in the case that a module
> calls Ns_ConContent(), but otherwise we can use read/write etc.
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
>
> > aio_read/aio_write system calls look like supported ubnder Linux
> >
>
> yes.  most of modern os's support some kind of kaio.
> I have checked solaris, linux and darwin and all do.


Hmm, not sure about that...

Linux aio_read() is implemented within glibc with via threads.  The
2.6 Linux epoll() is great for socket file descriptors, but doesn't
work with files.  Same for realtime signals in 2.4.  The kernel stuff,
io_submit() etc. only works on files opened with O_DIRECT, i.e. no
file system buffering. io_*() via libaio, and epoll() are not
portable.

The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
from 10.3 on, but again this is not portable.  This didn't work with
threads in the past.  Don't know if/when that got fixed...

Solaris 10 has some brand new port_*() calls. Again, not portable.=20
Not sure how aio_read() etc. are implemented on older versions of
Solaris.

I think Windows has pretty good support for AIO, including to files.
Obviously, not portable.


If we only consider the problem of how to spool large uploads to disk,
then AIO is the most direct solution.  It would allow the driver
thread to handle everything.  But this doesn't help us with upload
statistics or access control etc. so we still have to tackle those
problems with some other solution.

The biggest problem I see with AIO is that it will be a real pain to
implement.  At the minimum you're going to have to come up with some
kind of higher level abstraction, then add a different implementation
for each system we want to support, then add a fallback mechanism for
the systems which don't have AIO to offer.

Oh, and it would be nice to handle network IO with that too, which may
mean using a different implementation depending on descriptor type.

Looks tough...  :-(



> the problem with spawning yet another thread is
> system resources. each new thread (on solaris)
> results in an lwp which is not cheap. therfore I'd
> opt to use aio for largefile spooling.
>
> Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 08:36 schrieb Stephen Deasey:

> On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>>
>> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
>>
>>> aio_read/aio_write system calls look like supported ubnder Linux
>>>
>>
>> yes.  most of modern os's support some kind of kaio.
>> I have checked solaris, linux and darwin and all do.
>
>
> Hmm, not sure about that...
>
> Linux aio_read() is implemented within glibc with via threads.  The
> 2.6 Linux epoll() is great for socket file descriptors, but doesn't
> work with files.  Same for realtime signals in 2.4.  The kernel stuff,
> io_submit() etc. only works on files opened with O_DIRECT, i.e. no
> file system buffering. io_*() via libaio, and epoll() are not
> portable.
>
> The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
> from 10.3 on, but again this is not portable.  This didn't work with
> threads in the past.  Don't know if/when that got fixed...
>
> Solaris 10 has some brand new port_*() calls. Again, not portable.
> Not sure how aio_read() etc. are implemented on older versions of
> Solaris.
>
> I think Windows has pretty good support for AIO, including to files.
> Obviously, not portable.

Look:

On Darwin (10.4+)

aio_cancel(2)            - cancel an outstanding asynchronous I/O  
operation (REALTIME)
aio_error(2)             - retrieve error status of asynchronous I/O  
operation (REALTIME)
aio_read(2)              - asynchronous read from a file (REALTIME)
aio_return(2)            - retrieve return status of asynchronous I/O  
operation (REALTIME)
aio_suspend(2)           - suspend until asynchronous I/O operations  
or timeout complete (REALTIME)
aio_write(2)             - asynchronous write to a file (REALTIME)

On Solaris:

On Linux:

aio_return (3)       - get return status of asynchronous I/O operation
aio.h (0p)           - asynchronous input and output (REALTIME)
aio_cancel (3)       - cancel an outstanding asynchronous I/O request
aio_error (3p)       - retrieve errors status for an asynchronous I/O  
operation (REALTIME)
aio_error (3)        - get error status of asynchronous I/O operation
aio_suspend (3)      - wait for asynchronous I/O operation or timeout
aio_read (3p)        - asynchronous read from a file (REALTIME)
aio_return (3p)      - retrieve return status of an asynchronous I/O  
operation (REALTIME)
aio_write (3p)       - asynchronous write to a file (REALTIME)
aio_cancel (3p)      - cancel an asynchronous I/O request (REALTIME)
aio_fsync (3)        - asynchronous file synchronization
aio_fsync (3p)       - asynchronous file synchronization (REALTIME)
aio_write (3)        - asynchronous write
aio_suspend (3p)     - wait for an asynchronous I/O request (REALTIME)
aio_read (3)         - asynchronous read

On Solaris (2.8+)

# apropos aio
aio             aio (3head)     - asynchronous input and output
aio_cancel      aio_cancel (3rt)    - cancel asynchronous I/O request
aio_error       aio_error (3rt) - retrieve errors status for an  
asynchronous I/O operation
aio_fsync       aio_fsync (3rt) - asynchronous file synchronization
aio_read        aio_read (3rt)  - asynchronous read from a file
aio_req         aio_req (9s)    - asynchronous I/O request structure
aio_return      aio_return (3rt)    - retrieve return status of an  
asynchronous I/O operation
aio_suspend     aio_suspend (3rt)   - wait for asynchronous I/O request
aio_write       aio_write (3rt) - asynchronous write to a file
aiocancel       aiocancel (3aio)    - cancel an asynchronous operation
aioread         aioread (3aio)  - read or write asynchronous I/O  
operations
aiowait         aiowait (3aio)  - wait for completion of asynchronous  
I/O operation
aiowrite        aioread (3aio)  - read or write asynchronous I/O  
operations
libaio          libaio (3lib)   - the asynchronous I/O library

What we'd need is available on all above platforms:

    aio_write
    aio_cancel
    aio_suspend

So, what is the problem?

>
>
> If we only consider the problem of how to spool large uploads to disk,
> then AIO is the most direct solution.  It would allow the driver
> thread to handle everything.  But this doesn't help us with upload
> statistics or access control etc. so we still have to tackle those
> problems with some other solution.

This is right and I said that already: the upload statistics and/or
access ctrl etc, needs to be solved additionally by some other means.

>
> The biggest problem I see with AIO is that it will be a real pain to
> implement.  At the minimum you're going to have to come up with some
> kind of higher level abstraction, then add a different implementation
> for each system we want to support, then add a fallback mechanism for
> the systems which don't have AIO to offer.

As I see, all platforms we support now have needed aio_* calls. It might
be that we'd have to write some glue-code if API's are not 100%  
compatible
(which I believe they are):

Darwin:    int aio_write(struct aiocb *iocb);
Solaris:   int aio_write(struct aiocb *aiocbp);
Linux:     int aio_write(struct aiocb *aiocbp);


>
> Oh, and it would be nice to handle network IO with that too, which may
> mean using a different implementation depending on descriptor type.

Why network traffic? The only thing we really need there now is to
make spooling of the large-file-upload non-blocking, or?

>
> Looks tough...  :-(

Depends. I do not believe that it is that difficult. Admittently
I havent done anything in that direction yet, but judging from
the man-page reading it is all really simple. Might be that the
problems will expose when we dig more deeply inthere, but just
from the surface it looks ok.

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 1/2/06, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 02.01.2006 um 08:36 schrieb Stephen Deasey:
>
> > On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
> >>
> >> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
> >>
> >>> aio_read/aio_write system calls look like supported ubnder Linux
> >>>
> >>
> >> yes.  most of modern os's support some kind of kaio.
> >> I have checked solaris, linux and darwin and all do.
> >
> >
> > Hmm, not sure about that...
> >
> > Linux aio_read() is implemented within glibc with via threads.  The
> > 2.6 Linux epoll() is great for socket file descriptors, but doesn't
> > work with files.  Same for realtime signals in 2.4.  The kernel stuff,
> > io_submit() etc. only works on files opened with O_DIRECT, i.e. no
> > file system buffering. io_*() via libaio, and epoll() are not
> > portable.
> >
> > The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
> > from 10.3 on, but again this is not portable.  This didn't work with
> > threads in the past.  Don't know if/when that got fixed...
> >
> > Solaris 10 has some brand new port_*() calls. Again, not portable.
> > Not sure how aio_read() etc. are implemented on older versions of
> > Solaris.
> >
> > I think Windows has pretty good support for AIO, including to files.
> > Obviously, not portable.
>
> Look:
>
> On Darwin (10.4+)
>
> aio_cancel(2)            - cancel an outstanding asynchronous I/O
> operation (REALTIME)
> aio_error(2)             - retrieve error status of asynchronous I/O
> operation (REALTIME)
> aio_read(2)              - asynchronous read from a file (REALTIME)
> aio_return(2)            - retrieve return status of asynchronous I/O
> operation (REALTIME)
> aio_suspend(2)           - suspend until asynchronous I/O operations
> or timeout complete (REALTIME)
> aio_write(2)             - asynchronous write to a file (REALTIME)
>
> On Solaris:
>
> On Linux:
>
> aio_return (3)       - get return status of asynchronous I/O operation
> aio.h (0p)           - asynchronous input and output (REALTIME)
> aio_cancel (3)       - cancel an outstanding asynchronous I/O request
> aio_error (3p)       - retrieve errors status for an asynchronous I/O
> operation (REALTIME)
> aio_error (3)        - get error status of asynchronous I/O operation
> aio_suspend (3)      - wait for asynchronous I/O operation or timeout
> aio_read (3p)        - asynchronous read from a file (REALTIME)
> aio_return (3p)      - retrieve return status of an asynchronous I/O
> operation (REALTIME)
> aio_write (3p)       - asynchronous write to a file (REALTIME)
> aio_cancel (3p)      - cancel an asynchronous I/O request (REALTIME)
> aio_fsync (3)        - asynchronous file synchronization
> aio_fsync (3p)       - asynchronous file synchronization (REALTIME)
> aio_write (3)        - asynchronous write
> aio_suspend (3p)     - wait for an asynchronous I/O request (REALTIME)
> aio_read (3)         - asynchronous read
>
> On Solaris (2.8+)
>
> # apropos aio
> aio             aio (3head)     - asynchronous input and output
> aio_cancel      aio_cancel (3rt)    - cancel asynchronous I/O request
> aio_error       aio_error (3rt) - retrieve errors status for an
> asynchronous I/O operation
> aio_fsync       aio_fsync (3rt) - asynchronous file synchronization
> aio_read        aio_read (3rt)  - asynchronous read from a file
> aio_req         aio_req (9s)    - asynchronous I/O request structure
> aio_return      aio_return (3rt)    - retrieve return status of an
> asynchronous I/O operation
> aio_suspend     aio_suspend (3rt)   - wait for asynchronous I/O request
> aio_write       aio_write (3rt) - asynchronous write to a file
> aiocancel       aiocancel (3aio)    - cancel an asynchronous operation
> aioread         aioread (3aio)  - read or write asynchronous I/O
> operations
> aiowait         aiowait (3aio)  - wait for completion of asynchronous
> I/O operation
> aiowrite        aioread (3aio)  - read or write asynchronous I/O
> operations
> libaio          libaio (3lib)   - the asynchronous I/O library
>
> What we'd need is available on all above platforms:
>
>     aio_write
>     aio_cancel
>     aio_suspend
>
> So, what is the problem?


http://www.gnu.org/software/libc/manual/html_node/Configuration-of-AIO.html

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 17:23 schrieb Stephen Deasey:

>> So, what is the problem?
>
>
> http://www.gnu.org/software/libc/manual/html_node/Configuration-of- 
> AIO.html
>

Still, where is the problem? The fact that Linux implements them as
userlevel thread does not mean much to me.

On Solaris, each thread you start results in an LWP in the kernel.
At least this is so if you use the naviserver thread abstraction.
This is not cheap, hence I'm not very fond of starting threads
with or without the Tcl interp, as all threads are actually expensive.
(the former are more expensive though).

On Linux, especially with NPTL, threads are rather cheap, so
there you can easily add yet-another-thread and handle the spooling
with a dedicated thread or pool of threads.

But... what is the MAIN reason against AIO? Why would not that be
a platform-neutral simple way of doing a non-blocking writing from
within our driver-thread? Provided we have a simple and workable
abstract solution (aio_xxx family of routines) which is thread-safe,
why would we bother about the per-platform implementation of it?

???

Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 19:13 schrieb Vlad Seryakov:

> I did very simple test, replaced write with aio_write and at the  
> end checked aio_error/aio_return, they all returned 0 so mmap  
> should work because file is synced. when i was doing aio_write i  
> used aio_offset, so each aio_write would put data into separate  
> region on the file.
>
> Unfortunately i removed modified driver.c by accident, so i will  
> have to do it again but something is not right in simply replacing  
> write with aio_write and mmap. Without mmap, GetForm will have to  
> read the file manually and parse it, it makes it more complicated  
> and still, if writes are not complete we may get SIGBUS again.
>
> The problem i see here i cannot be sure that all writes are  
> completed, aio_error/aio_return are used to check only one last  
> operation, not the batch of writes.

Hmmm... that means that before writing the next chunk, you should  
check the
status of the last and skip if still  not written?

OTOH, how come you get SIGBUS? And where? Normally this might be only
if you mmap the file as some structure, not char array?

Zoran