Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
>
> > aio_read/aio_write system calls look like supported ubnder Linux
> >
>
> yes.  most of modern os's support some kind of kaio.
> I have checked solaris, linux and darwin and all do.


Hmm, not sure about that...

Linux aio_read() is implemented within glibc with via threads.  The
2.6 Linux epoll() is great for socket file descriptors, but doesn't
work with files.  Same for realtime signals in 2.4.  The kernel stuff,
io_submit() etc. only works on files opened with O_DIRECT, i.e. no
file system buffering. io_*() via libaio, and epoll() are not
portable.

The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
from 10.3 on, but again this is not portable.  This didn't work with
threads in the past.  Don't know if/when that got fixed...

Solaris 10 has some brand new port_*() calls. Again, not portable.=20
Not sure how aio_read() etc. are implemented on older versions of
Solaris.

I think Windows has pretty good support for AIO, including to files.
Obviously, not portable.


If we only consider the problem of how to spool large uploads to disk,
then AIO is the most direct solution.  It would allow the driver
thread to handle everything.  But this doesn't help us with upload
statistics or access control etc. so we still have to tackle those
problems with some other solution.

The biggest problem I see with AIO is that it will be a real pain to
implement.  At the minimum you're going to have to come up with some
kind of higher level abstraction, then add a different implementation
for each system we want to support, then add a fallback mechanism for
the systems which don't have AIO to offer.

Oh, and it would be nice to handle network IO with that too, which may
mean using a different implementation depending on descriptor type.

Looks tough...  :-(



> the problem with spawning yet another thread is
> system resources. each new thread (on solaris)
> results in an lwp which is not cheap. therfore I'd
> opt to use aio for largefile spooling.
>
> Zoran