naviserver-devel — Developer discussion

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 18:24 schrieb Vlad Seryakov:

> Agreed on that as well, let's do it

Conclusion:

   ns_register_filter when method urlPattern script ?arg1 arg2 ...?"

The script definition must thus match:

proc script {when args} {...}

For calls w/o args, we are compatbile with AS.
For calls w/ args, we are not.

Who will do that? Stephen?

Cheers,
Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 12:06 schrieb Zoran Vasiljevic:

>
> Am 02.01.2006 um 04:43 schrieb Vlad Seryakov:
>
>> Tried it, nsd started crashing and i am guessing that the problem  
>> is combination of aio_write and mmap.
>> When is start spooling, i just submit aio_write and return  
>> immediately, so there are a lot of quick aio_write calls. By the  
>> time i reach mmap, it looks like it's always ahead of actual  
>> writing, so even when i am trying to check aio_error/aio_fsync in  
>> the NsGetRequest, i still get SIGBUS/SIGSEGV when i access reqPtr- 
>> >content, looks like mmap and aio_write are out of sync. And there  
>> is no way to wait until all file buffers will be flushed, so  
>> manual separate thread implementation could be the only portable  
>> solution. Or i just do not understand aio_xxx things completely.
>
> Hm... the idea is to mmap the file when all is read from the client.
> So you'd have to aio_write into the file until all is received, then
> at some point check the aio_status of the write (non-blocking) and
> when it's done, revert to mmap the file. I do not believe you can
> mix the aio_ with mmap.

What I still do not know is: if you are allowed to issue multiple
outstanding aio_writes one after another... It might be that you
need to check wether the first aio_write is done before you attempt
the next one.

Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 04:43 schrieb Vlad Seryakov:

> Tried it, nsd started crashing and i am guessing that the problem  
> is combination of aio_write and mmap.
> When is start spooling, i just submit aio_write and return  
> immediately, so there are a lot of quick aio_write calls. By the  
> time i reach mmap, it looks like it's always ahead of actual  
> writing, so even when i am trying to check aio_error/aio_fsync in  
> the NsGetRequest, i still get SIGBUS/SIGSEGV when i access reqPtr- 
> >content, looks like mmap and aio_write are out of sync. And there  
> is no way to wait until all file buffers will be flushed, so manual  
> separate thread implementation could be the only portable solution.  
> Or i just do not understand aio_xxx things completely.

Hm... the idea is to mmap the file when all is read from the client.
So you'd have to aio_write into the file until all is received, then
at some point check the aio_status of the write (non-blocking) and
when it's done, revert to mmap the file. I do not believe you can
mix the aio_ with mmap.

Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 08:36 schrieb Stephen Deasey:

> On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>>
>> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
>>
>>> aio_read/aio_write system calls look like supported ubnder Linux
>>>
>>
>> yes.  most of modern os's support some kind of kaio.
>> I have checked solaris, linux and darwin and all do.
>
>
> Hmm, not sure about that...
>
> Linux aio_read() is implemented within glibc with via threads.  The
> 2.6 Linux epoll() is great for socket file descriptors, but doesn't
> work with files.  Same for realtime signals in 2.4.  The kernel stuff,
> io_submit() etc. only works on files opened with O_DIRECT, i.e. no
> file system buffering. io_*() via libaio, and epoll() are not
> portable.
>
> The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
> from 10.3 on, but again this is not portable.  This didn't work with
> threads in the past.  Don't know if/when that got fixed...
>
> Solaris 10 has some brand new port_*() calls. Again, not portable.
> Not sure how aio_read() etc. are implemented on older versions of
> Solaris.
>
> I think Windows has pretty good support for AIO, including to files.
> Obviously, not portable.

Look:

On Darwin (10.4+)

aio_cancel(2)            - cancel an outstanding asynchronous I/O  
operation (REALTIME)
aio_error(2)             - retrieve error status of asynchronous I/O  
operation (REALTIME)
aio_read(2)              - asynchronous read from a file (REALTIME)
aio_return(2)            - retrieve return status of asynchronous I/O  
operation (REALTIME)
aio_suspend(2)           - suspend until asynchronous I/O operations  
or timeout complete (REALTIME)
aio_write(2)             - asynchronous write to a file (REALTIME)

On Solaris:

On Linux:

aio_return (3)       - get return status of asynchronous I/O operation
aio.h (0p)           - asynchronous input and output (REALTIME)
aio_cancel (3)       - cancel an outstanding asynchronous I/O request
aio_error (3p)       - retrieve errors status for an asynchronous I/O  
operation (REALTIME)
aio_error (3)        - get error status of asynchronous I/O operation
aio_suspend (3)      - wait for asynchronous I/O operation or timeout
aio_read (3p)        - asynchronous read from a file (REALTIME)
aio_return (3p)      - retrieve return status of an asynchronous I/O  
operation (REALTIME)
aio_write (3p)       - asynchronous write to a file (REALTIME)
aio_cancel (3p)      - cancel an asynchronous I/O request (REALTIME)
aio_fsync (3)        - asynchronous file synchronization
aio_fsync (3p)       - asynchronous file synchronization (REALTIME)
aio_write (3)        - asynchronous write
aio_suspend (3p)     - wait for an asynchronous I/O request (REALTIME)
aio_read (3)         - asynchronous read

On Solaris (2.8+)

# apropos aio
aio             aio (3head)     - asynchronous input and output
aio_cancel      aio_cancel (3rt)    - cancel asynchronous I/O request
aio_error       aio_error (3rt) - retrieve errors status for an  
asynchronous I/O operation
aio_fsync       aio_fsync (3rt) - asynchronous file synchronization
aio_read        aio_read (3rt)  - asynchronous read from a file
aio_req         aio_req (9s)    - asynchronous I/O request structure
aio_return      aio_return (3rt)    - retrieve return status of an  
asynchronous I/O operation
aio_suspend     aio_suspend (3rt)   - wait for asynchronous I/O request
aio_write       aio_write (3rt) - asynchronous write to a file
aiocancel       aiocancel (3aio)    - cancel an asynchronous operation
aioread         aioread (3aio)  - read or write asynchronous I/O  
operations
aiowait         aiowait (3aio)  - wait for completion of asynchronous  
I/O operation
aiowrite        aioread (3aio)  - read or write asynchronous I/O  
operations
libaio          libaio (3lib)   - the asynchronous I/O library

What we'd need is available on all above platforms:

    aio_write
    aio_cancel
    aio_suspend

So, what is the problem?

>
>
> If we only consider the problem of how to spool large uploads to disk,
> then AIO is the most direct solution.  It would allow the driver
> thread to handle everything.  But this doesn't help us with upload
> statistics or access control etc. so we still have to tackle those
> problems with some other solution.

This is right and I said that already: the upload statistics and/or
access ctrl etc, needs to be solved additionally by some other means.

>
> The biggest problem I see with AIO is that it will be a real pain to
> implement.  At the minimum you're going to have to come up with some
> kind of higher level abstraction, then add a different implementation
> for each system we want to support, then add a fallback mechanism for
> the systems which don't have AIO to offer.

As I see, all platforms we support now have needed aio_* calls. It might
be that we'd have to write some glue-code if API's are not 100%  
compatible
(which I believe they are):

Darwin:    int aio_write(struct aiocb *iocb);
Solaris:   int aio_write(struct aiocb *aiocbp);
Linux:     int aio_write(struct aiocb *aiocbp);


>
> Oh, and it would be nice to handle network IO with that too, which may
> mean using a different implementation depending on descriptor type.

Why network traffic? The only thing we really need there now is to
make spooling of the large-file-upload non-blocking, or?

>
> Looks tough...  :-(

Depends. I do not believe that it is that difficult. Admittently
I havent done anything in that direction yet, but judging from
the man-page reading it is all really simple. Might be that the
problems will expose when we dig more deeply inthere, but just
from the surface it looks ok.

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 09:21 schrieb Stephen Deasey:

>
> POST request with small amounts of data are really common.  Think of a
> user logging in via a web form.  Couple of bytes.
>
> The way you've coded it at the mo (and I realize it's just a first
> cut), all requests with more than 0 bytes of body content will cause
> the upload stats code to fire.  That's why I suggested we may want to
> have some lower threshold, or restrict it to certain URLs via
> urlspecific data.  We don't want to track every form submission.

No. That is of course not good.

>>
>> One problem i see with lazy uploads that if you have multiple clients
>> doing large POSTs, spawning multiple clients for a long time reading
>> that content will waste resources, each conn thread is heavy, with  
>> Tcl
>> interp.
>
>
> Not necessarily.  You can create another thread pool and then ensure
> that those threads never run Tcl code.  Remember, Tcl interps are
> allocated lazily on first use. We could also adjust stacksize per
> thread pool.

But then, how are you going to interpret the stats? You need to
run some Tcl code for that and this will load the Tcl interp?


>
>
>> Using driver thread reading small chunks from the connections
>> and putting it into file will keep everything smooth. But with small
>> uploads on fast network this may not be an issue, so it needs a
>> compromize solution here, may be configurable options. Currently,
>> spooling into file can be disabled/enabled, lazy spooling may be
>> implemented similar way. Actually, lazy file spooling can be easily
>> done, because even Ns_ConnContent calls SockRead which does  
>> spooling, we
>> just need to introduce an option that tells how much we should  
>> spool in
>> the main thread and then continue in the conn thread.
>
>
> We already have maxreadahead, which is the amount of data read by the
> driver thread into a memory buffer.  I think you're either happy
> letting the driver thread block writing to disk, or you're not.  Why
> would you set a threshold on this?

Right. We already have that knob: maxreadahead. It is the question
what happens when we reach that point. At the moment, extra data is
spooled into the file and file is mmaped at the end of reading
when all the content from the client is received. I'm not that happy
with the driver thread pumping the data into the file as this might
affect the overall performance, therefore I suggested kernel AIO.
This feature of OS is developed precisely for that purpose.

Zoran

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 02.01.2006 um 09:30 schrieb Stephen Deasey:
>
> Right.  So the thread handling the upload would lock/unlock every
> upload_size/10Kbytes or seconds_to_upload.  The frequency of locking
> for the every 10Kbytes case would increase (and hence chance of
> blocking) with the capacity of your and your clients connection.
>
> Another thread would lock/unlock every time the client requests an
> update on the progress, say once per second.
>
> Multiply the above by the number of concurrent clients.
>
>
> That's the price you pay for upload stats and that's OK.  I wouldn't
> want that overhead when I'm not interested in the stats though...

In which case you'd:

   a. have no threads asking for the stats
   b. have only the upload thread do locks every 10k or such

so the locking will decrease even more. Furthermore, if you
set the upload_size to zero then we can entirely skip that
hence no locking would occur.

I believe that this is fair for everybody. People needing
that stats can turn it on. People never interested in that
can leave the default (no stats) and all are happy?

Zoran

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 31.12.2005 um 19:03 schrieb Vlad Seryakov:
>
> > Could config option, like 1 second or 10Kbytes
>
> Yup. For example. This could reduce locking attempts.
> Seems fine to me.


Right.  So the thread handling the upload would lock/unlock every
upload_size/10Kbytes or seconds_to_upload.  The frequency of locking
for the every 10Kbytes case would increase (and hence chance of
blocking) with the capacity of your and your clients connection.

Another thread would lock/unlock every time the client requests an
update on the progress, say once per second.

Multiply the above by the number of concurrent clients.


That's the price you pay for upload stats and that's OK.  I wouldn't
want that overhead when I'm not interested in the stats though...

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Vlad Seryakov <vl...@cr...> wrote:
>
> > I think we talked about this before, but I can't find it in the
> > mailing list archive.  Anyway, the problem with recording the upload
> > process is all the locking that's required.  You could minimize this,
> > e.g. by only recording uploads above a certain size, or to a certain
> > URL.
>
> Yes, that is true, but at the same time, GET requests do not carry the
> body, so no locking willhappen. POST uploads are very slow processes in
> a sense that user expects to wait until it finishes and locking will
> happen only for uploads. It is possible further minimize it by doing
> locking only several times instead of on every read but avoiding locks
> is not possible.


POST request with small amounts of data are really common.  Think of a
user logging in via a web form.  Couple of bytes.

The way you've coded it at the mo (and I realize it's just a first
cut), all requests with more than 0 bytes of body content will cause
the upload stats code to fire.  That's why I suggested we may want to
have some lower threshold, or restrict it to certain URLs via
urlspecific data.  We don't want to track every form submission.


> > It reminds me of a similar problem we had.  Spooling large uploads to d=
isk:
> >
> > https://sourceforge.net/mailarchive/forum.php?thread_id=3D7524448&forum=
_id=3D43966
> >
> > Vlad implemented the actual spooling, but moving that work into the
> > conn threads, reading lazily, is still to be done.
> >
> > Lazy uploading is exactly the hook you need to track upload progress.
> > The client starts to upload a file.  Read-ahead occurs in the driver
> > thread, say 8k.  Control is passed to a conn thread, which then calls
> > Ns_ConnContent().  The remaining content is read from the client, in
> > the context of the conn thread and so not blocking the driver thread,
> > and perhaps the content is spooled to disk.
> >
> > To implement upload tracking you would register a proc for /upload
> > which instead of calling Ns_ConnContent(), calls Ns_ConnRead()
> > multiple times, recording the number of bytes read in the upload
> > tracking cache, and saving the data to disk or wherever.
> >
> > A lot more control of the upload process is needed, whether it be to
> > control size, access, to record stats, or something we haven't thought
> > of yet.  If we complete the work to get lazy reading from the client
> > working, an upload tracker will be an easy module to write.
> >
>
> One problem i see with lazy uploads that if you have multiple clients
> doing large POSTs, spawning multiple clients for a long time reading
> that content will waste resources, each conn thread is heavy, with Tcl
> interp.


Not necessarily.  You can create another thread pool and then ensure
that those threads never run Tcl code.  Remember, Tcl interps are
allocated lazily on first use. We could also adjust stacksize per
thread pool.


> Using driver thread reading small chunks from the connections
> and putting it into file will keep everything smooth. But with small
> uploads on fast network this may not be an issue, so it needs a
> compromize solution here, may be configurable options. Currently,
> spooling into file can be disabled/enabled, lazy spooling may be
> implemented similar way. Actually, lazy file spooling can be easily
> done, because even Ns_ConnContent calls SockRead which does spooling, we
> just need to introduce an option that tells how much we should spool in
> the main thread and then continue in the conn thread.


We already have maxreadahead, which is the amount of data read by the
driver thread into a memory buffer.  I think you're either happy
letting the driver thread block writing to disk, or you're not.  Why
would you set a threshold on this?

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Gustaf Neumann <ne...@wu...> wrote:
> Zoran Vasiljevic wrote:
> >
> > But that would occupy the conn thread for ages, right?
> > I can imagine several slow-line large-file uploads could
> > consume many of those. Wasn't that the reason everything
> > was moved in the driver thread for the 4.0 ?
> >
> > What I have/had in mind is aync writes from the driver thread.
> > Most of the OS'es have this feature (kaio) so we can employ it.
> > The question of locking, however still remains in such case.
> > So the decision has to be made on: what is cheaper? Locking
> > or spooling to disk out of the conn thread? I have no real-life
> > experience but I'm inclined to believe that spooling out of the
> > conn thread would be more costly.
> >
> > What do you think?
> i have not much looked into the c-code, so please forgive, if my
> mail here is obviously wrong or naive.
>
> One thing, which i dislike about the aolserver for file uploads is that
> there is no hook for early permission or quota checking; one has to
> upload the whole file before aolserver is able to give some feedback.
> i see this even more important than the progress bar.


I agree. This is important.


> What about the following model:
>
> A) HTTP header (up to empty line) -> driver thread
>
> B) Then passing control+socket to a separate receiving thread that
>   1) issues a callback at the start (e.g. for permission checking)
>   2) does async i/o to receive the message body (e.g. PUT, POST request)
>     here, one can issue as well callbacks per async block, and decide
>     fom tcl, whether one would like to spool to memory or disc.
>
> C) When end of message received, pass control to connection thread as usu=
al.
>
> The receiving thread is like a connection thread with tcl stuff; when the
> receiving thread is implemented via libthtread, there won't be probably
> much locking. Only one receiving thread will be necessary.
> The overall model could be quite compatible with the aolserver, since the
> connection tread is invoked when all data is received. It would be necess=
ary
> to pass context information from the receiving thread to the connection
> thread
> to avoid double permission checking.
>
> It could as well make sense, to pass control to the receiving thread only
> for chunked content. Maybe, the receiving thread is not needed at all,
> since the driver thread could handle everything as well.


This is similar to the way the old Flash web server worked.  It had a
pool of IO threads which would make sure data on disk was in memory,
possibly blocking, before the main thread would try to write it out to
the client.  Multiple threads are used to get more throughput out of
the disk.  As one thread blocks, another can submit a request.

Your single thread helper mechanism above may not work the disk as
hard as multiple threads, but it does mean that the driver thread
doesn't have to block waiting for disk.

You've also moved quota checking etc. into the helper thread.  This
doesn't so much matter if there's just one, but with more than one you
have to remember that a Tcl interp will be allocated for each, and
they'll be just as heavyweight as normal conn threads.

We have a similarly problem with normal downloads, i.e. writing data
to the client, an mp3 file for example, ties up a heavyweight conn
thread.  A generic pool of IO threads might be useful in a number of
places.  Of course, this is just a simulation of AIO to disk, but see
my other email for why AIO is not so easy.


AOLserver HEAD has a new filter type, NS_FILTER_PREQUEUE, which gets
called just before the connection is queued for processing by a conn
thread.  The unique thing about this filter type is that the
registered code runs in the context of the driver thread.=20
Unfortunately that's after all the data has been read and spooled, but
we could implement it such that they ran just after all headers have
been received.

If you want to check quotas by querying a database then then this
doesn't help -- your db query will block the driver thread.  These
filters will have to be coded carefully.  We could also implement
non-blocking db calls, and that's certainly something I'm interested
in, but it doesn't help right now.

However, if the code you want to run is short and non blocking,
prequeue filters we be a good way to run quota checks and such without
having to involve a conn thread at all.


Prequeue filters don't help with upload tracking, nor with actually
spooling large uploads to disk.

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
>
> > aio_read/aio_write system calls look like supported ubnder Linux
> >
>
> yes.  most of modern os's support some kind of kaio.
> I have checked solaris, linux and darwin and all do.


Hmm, not sure about that...

Linux aio_read() is implemented within glibc with via threads.  The
2.6 Linux epoll() is great for socket file descriptors, but doesn't
work with files.  Same for realtime signals in 2.4.  The kernel stuff,
io_submit() etc. only works on files opened with O_DIRECT, i.e. no
file system buffering. io_*() via libaio, and epoll() are not
portable.

The BSDs (FreeBSD first, others later) have kqueue, and so does Mac/OS
from 10.3 on, but again this is not portable.  This didn't work with
threads in the past.  Don't know if/when that got fixed...

Solaris 10 has some brand new port_*() calls. Again, not portable.=20
Not sure how aio_read() etc. are implemented on older versions of
Solaris.

I think Windows has pretty good support for AIO, including to files.
Obviously, not portable.


If we only consider the problem of how to spool large uploads to disk,
then AIO is the most direct solution.  It would allow the driver
thread to handle everything.  But this doesn't help us with upload
statistics or access control etc. so we still have to tackle those
problems with some other solution.

The biggest problem I see with AIO is that it will be a real pain to
implement.  At the minimum you're going to have to come up with some
kind of higher level abstraction, then add a different implementation
for each system we want to support, then add a fallback mechanism for
the systems which don't have AIO to offer.

Oh, and it would be nice to handle network IO with that too, which may
mean using a different implementation depending on descriptor type.

Looks tough...  :-(



> the problem with spawning yet another thread is
> system resources. each new thread (on solaris)
> results in an lwp which is not cheap. therfore I'd
> opt to use aio for largefile spooling.
>
> Zoran

Re: [naviserver-devel] Upload Progress

[Stephen D. <sd...@gm...>]

On 1/1/06, Vlad Seryakov <vl...@cr...> wrote:
> Another solution to reduce locking, just to allocate maxconn structures
> with its own mutex and perform locking for particular struct only, so
> writers will not block other writers, only writer/reader. More memory
> but less concurrency.
>
> I will try this tomorrow.


But how do you get access to the right conn structure?  You'll have to
use a hash table of URLs, which will have to be locked.  That's one
lock/unlock at the start and end of each upload, and one lock/unlock
for every query of the upload progress, say one per second or so.

The thing about building upload tracking into the server is that it
doesn't help solve other problems, such as the extra control needed
before permitting a large upload. If hooks are added for that, then
upload tracking could be implemented using the same mechanism.  It
would allow you to track uniqueness of the uploader via cookie for
example.

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

Another solution to reduce locking, just to allocate maxconn structures 
with its own mutex and perform locking for particular struct only, so 
writers will not block other writers, only writer/reader. More memory 
but less concurrency.

I will try this tomorrow.

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 18:20 schrieb Vlad Seryakov:
> 
>> Another possible solution can be, pre-allocating maxconn upload  
>> structs and update them without locks, it is integer anyway, so no  
>> need to lock, 4 byte write is never innterupted, usually it is 1  CPU 
>> instruction(true for Intel, maybe not for Sparc).
> 
> 
> Nope. Need locking even for integers. I've seen crazy things...
> 
> OTOH, I'm not that concerned about the locking if it is short-lived
> and not frequently done. I believe the "not frequently done" is the
> hardest thing to judge.
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

Tried it, nsd started crashing and i am guessing that the problem is 
combination of aio_write and mmap.
When is start spooling, i just submit aio_write and return immediately, 
so there are a lot of quick aio_write calls. By the time i reach mmap, 
it looks like it's always ahead of actual writing, so even when i am 
trying to check aio_error/aio_fsync in the NsGetRequest, i still get 
SIGBUS/SIGSEGV when i access reqPtr->content, looks like mmap and 
aio_write are out of sync. And there is no way to wait until all file 
buffers will be flushed, so manual separate thread implementation could 
be the only portable solution. Or i just do not understand aio_xxx 
things completely.

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:
> 
>> aio_read/aio_write system calls look like supported ubnder Linux
>>
> 
> yes.  most of modern os's support some kind of kaio.
> I have checked solaris, linux and darwin and all do.
> the problem with spawning yet another thread is
> system resources. each new thread (on solaris)
> results in an lwp which is not cheap. therfore I'd
> opt to use aio for largefile spooling.
> 
> Zoran
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 20:12 schrieb Vlad Seryakov:

> aio_read/aio_write system calls look like supported ubnder Linux
>

yes.  most of modern os's support some kind of kaio.
I have checked solaris, linux and darwin and all do.
the problem with spawning yet another thread is
system resources. each new thread (on solaris)
results in an lwp which is not cheap. therfore I'd
opt to use aio for largefile spooling.

Zoran 

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

aio_read/aio_write system calls look like supported ubnder Linux

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 19:26 schrieb Vlad Seryakov:
> 
>> What about separate thread doing all spooling I/O, driver thread  will 
>> send buffer to be written and immediately continue while  spooling 
>> thread will do writes?
> 
> 
> This is another option. I like the kaio because it saves you
> yet-another-thread to manage. And it perfectly fits in the
> non-blocking operation of the driver thread. One can evaluate
> both options and then decide. Somehow I have the feeling that
> kaio would be simpler to fit into the current code (I might
> be wrong though)
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 19:26 schrieb Vlad Seryakov:

> What about separate thread doing all spooling I/O, driver thread  
> will send buffer to be written and immediately continue while  
> spooling thread will do writes?

This is another option. I like the kaio because it saves you
yet-another-thread to manage. And it perfectly fits in the
non-blocking operation of the driver thread. One can evaluate
both options and then decide. Somehow I have the feeling that
kaio would be simpler to fit into the current code (I might
be wrong though)

Zoran

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

> 
> Also, the blocking of the driver thread during spooling
> into the file should be taken care. I wanted to look into
> the kernel async IO (as it is available on Darwin/Sol/Linux).
> 

What about separate thread doing all spooling I/O, driver thread will 
send buffer to be written and immediately continue while spooling thread 
will do writes?


-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 19:03 schrieb Vlad Seryakov:

> Could config option, like 1 second or 10Kbytes

Yup. For example. This could reduce locking attempts.
Seems fine to me.

Also, the blocking of the driver thread during spooling
into the file should be taken care. I wanted to look into
the kernel async IO (as it is available on Darwin/Sol/Linux).

Zoran 

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

Could config option, like 1 second or 10Kbytes

Zoran Vasiljevic wrote:
> 
> Am 31.12.2005 um 18:20 schrieb Vlad Seryakov:
> 
>> Another possible solution can be, pre-allocating maxconn upload  
>> structs and update them without locks, it is integer anyway, so no  
>> need to lock, 4 byte write is never innterupted, usually it is 1  CPU 
>> instruction(true for Intel, maybe not for Sparc).
> 
> 
> Nope. Need locking even for integers. I've seen crazy things...
> 
> OTOH, I'm not that concerned about the locking if it is short-lived
> and not frequently done. I believe the "not frequently done" is the
> hardest thing to judge.
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 18:20 schrieb Vlad Seryakov:

> Another possible solution can be, pre-allocating maxconn upload  
> structs and update them without locks, it is integer anyway, so no  
> need to lock, 4 byte write is never innterupted, usually it is 1  
> CPU instruction(true for Intel, maybe not for Sparc).

Nope. Need locking even for integers. I've seen crazy things...

OTOH, I'm not that concerned about the locking if it is short-lived
and not frequently done. I believe the "not frequently done" is the
hardest thing to judge.

Zoran

Re: [naviserver-devel] Naviserver and OACS (compatibility)?

[Vlad S. <vl...@cr...>]

Agreed on that as well, let's do it

Stephen Deasey wrote:
> On 12/31/05, Zoran Vasiljevic <zv...@ar...> wrote:
> 
>>Am 31.12.2005 um 01:15 schrieb Gustaf Neumann:
>>
>>
>>> Class Filter
>>> Filter instproc preauth args { .... }
>>> Filter instproc postauth args { .... }
>>>
>>
>>Damn, you are right. In which case we'd do:
>>
>>     ns_register_filter <reason> GET /junk myfilter myarg1 myarg2 ...
>>
>>and it would call the filter as:
>>
>>     myfilter <reason> myarg2 myarg3
>>
>>Understand. This would allow you to:
>>
>>Class TheFilter
>>TheFilter proc preauth   {arg1 arg2} {...}
>>TheFilter proc postauth  {arg1 arg2 arg3} {...}
>>
>>ns_register_filter preauth  GET /junk TheFilter arg1 arg2
>>ns_register_filter postauth GET /junk TheFilter arg1 arg2 arg3
>>
>>and we'd call
>>
>>TheFilter preauth arg1 arg2
>>TheFilter postauth arg1 arg2 arg3
>>
>>The proc way would be:
>>
>>proc TheFilter {reason args} {...}
>>
>>and it would naturally fit in, as with OO syntax.
>>
>>Well, this certainly makes sense to me. This is clean and elegant.
>>Hm... this will however be an incompatible change to the way AOL
>>server is handling the task, but will be way more sexy.
>>
>>This is what I mean (others can comment): I'd go after this change.
>>For the people not using any optional arguments, there would be no
>>compatibility issues. For those using the arguments, they'd have to
>>rewrite the code. But it is for the better, not worse.
>>
>>Stephen, Vlad, what do you think? I have no problem with that
>>as we are not using any of the optional args in our code.
>>We might examine to modify ns_register_proc the same way.
> 
> 
> 
> The order:
> 
>     TheFilter preauth arg1 arg2
> 
> makes the most sense to me. The <when> is not an optional arg, it's
> always appended.  Required args always come before optional args in
> proc specs...
> 
> If we changed this, we should also change Ns_TclEvalCallback() in the
> same way.  A proc registered with ns_serverrootproc may have the host
> appended as the last arg. It uses Ns_TclEvalCallback() directly.
> 
> ns_register_proc doesn't append information of it's own.  But it does
> insist on appending a blank arg if none were supplied during
> registration.  I'd like to remove that as it's easy to write code
> which works with either case.
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

> 
> What I have/had in mind is aync writes from the driver thread.
> Most of the OS'es have this feature (kaio) so we can employ it.
> The question of locking, however still remains in such case.
> So the decision has to be made on: what is cheaper? Locking
> or spooling to disk out of the conn thread? I have no real-life
> experience but I'm inclined to believe that spooling out of the
> conn thread would be more costly.

We d locking in nsd in a lot of places anyway, locking itself is not the 
issue if it does it for very short time, of course, everything is 
serialized.

The other solution could be special-formatted query arguments that will 
tell to do upload statistics, looks like a hack but why not, it is 
web/application server, it can introduce specific protocol extensions.

something like: /upload/url?arg1=1&arg2=2;ns_upload_stats=1

Another possible solution can be, pre-allocating maxconn upload structs 
and update them without locks, it is integer anyway, so no need to lock, 
4 byte write is never innterupted, usually it is 1 CPU instruction(true 
for Intel, maybe not for Sparc).

In the request for upload stat thread, we scan that array for matching 
url and return upload stats. So driver thread is writing only, other 
thread are reading only.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Upload Progress

[Vlad S. <vl...@cr...>]

> I think we talked about this before, but I can't find it in the
> mailing list archive.  Anyway, the problem with recording the upload
> process is all the locking that's required.  You could minimize this,
> e.g. by only recording uploads above a certain size, or to a certain
> URL.

Yes, that is true, but at the same time, GET requests do not carry the 
body, so no locking willhappen. POST uploads are very slow processes in 
a sense that user expects to wait until it finishes and locking will 
happen only for uploads. It is possible further minimize it by doing 
locking only several times instead of on every read but avoiding locks 
is not possible.

> It reminds me of a similar problem we had.  Spooling large uploads to disk:
> 
> https://sourceforge.net/mailarchive/forum.php?thread_id=7524448&forum_id=43966
> 
> Vlad implemented the actual spooling, but moving that work into the
> conn threads, reading lazily, is still to be done.
> 
> Lazy uploading is exactly the hook you need to track upload progress. 
> The client starts to upload a file.  Read-ahead occurs in the driver
> thread, say 8k.  Control is passed to a conn thread, which then calls
> Ns_ConnContent().  The remaining content is read from the client, in
> the context of the conn thread and so not blocking the driver thread,
> and perhaps the content is spooled to disk.
> 
> To implement upload tracking you would register a proc for /upload
> which instead of calling Ns_ConnContent(), calls Ns_ConnRead()
> multiple times, recording the number of bytes read in the upload
> tracking cache, and saving the data to disk or wherever.
> 
> A lot more control of the upload process is needed, whether it be to
> control size, access, to record stats, or something we haven't thought
> of yet.  If we complete the work to get lazy reading from the client
> working, an upload tracker will be an easy module to write.
> 

One problem i see with lazy uploads that if you have multiple clients 
doing large POSTs, spawning multiple clients for a long time reading 
that content will waste resources, each conn thread is heavy, with Tcl 
interp. Using driver thread reading small chunks from the connections 
and putting it into file will keep everything smooth. But with small 
uploads on fast network this may not be an issue, so it needs a 
compromize solution here, may be configurable options. Currently, 
spooling into file can be disabled/enabled, lazy spooling may be 
implemented similar way. Actually, lazy file spooling can be easily 
done, because even Ns_ConnContent calls SockRead which does spooling, we 
just need to introduce an option that tells how much we should spool in 
the main thread and then continue in the conn thread.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] Offtopic

[Bernd E. <eid...@we...>]

Hi you Navicoders,

my best wishes to you all for (today,tomorrow :-) and 2006!

Bernd.

Re: [naviserver-devel] Upload Progress

[Zoran V. <zv...@ar...>]

Am 31.12.2005 um 15:27 schrieb Gustaf Neumann:

>  Maybe, the receiving thread is not needed at all,
> since the driver thread could handle everything as well.

This is what I ment. Driver thread is very hot as it
does not (should not?) find iself into any blocking at all.
At the moment it does, as it writes into a mmaped file
when somebody uploads a thing larger that would fit in
a predefined buffer.  But this should be avoided. Precisely
that's the reason why I'm contemplating the kaio as this would
allow the driver thread to run at highest possible speed.

The locking.

In order to get any meaningful statistics, there have to
be places to hook into that processing which will inevitably
lead to locks. But, OTOH, how frequent would that be?
We are talking about very short and relatively seldom lock
attempts. I can't say it out of the sleeve, but I somehow
have a feeling that locking would not be an issue here.
Or?

Other kind of processing (like quota enforcing etc).

This would/could be pretty difficult as I wouldn't like to run
any Tcl code from within the driver thread. So, if this need
be done, than we must thing of some bit toggling and signalling
the driver thread to abort/stop/whatever the upload process if needed.

Just my 2 cents.

Cheers
Zoran