Re: [naviserver-devel] Tcl nonblocking file IO in NaviServer?

[Gustaf N. <ne...@wu...>]

On 06.06.19 23:09, Gustaf Neumann wrote:
> Not sure, what your performance constraints are, but open/close
> is pretty fast these days (measured under linux on openacs.org):
>     % time {set F [open /tmp/foo a]; puts $F x; close $F} 1000
>     55.333 microseconds per iteration
>
> if this is too slow for you, one can experiment with ramdisks,
> and other already mentioned approaches.
>
i did some more experiments and implemented a "ns_asyncwrite"
wrapper for NaviServer's internal function:

     time {ns_asyncwrite 55 hello\n} 1000
     0.957 microseconds per iteration

This is pretty cool. Since the AsyncWriterThread is build around
a queue, this is also protected against concurrent write() operations.
this means, one can do ~1mio simple write operations per second
in the thread protected way, independent from file system blocks
(measured on the same machine with the same load as above).

The only disadvantage is that one needs commands for opening/closing
based on the numeric file descriptors (not on Tcl file handles), which
means at the end small wrappers on top of NaviServer's 
ns_write()/ns_close().
Working with the numeric file descriptors has the advantage that it can be
passed around between interpreters/threads, etc., so it is well suited
for multi threaded applications, the FDs of open files can be kept
in nsvs, etc.).

I see also potentials for this in some of our projects (e.g. writing 
application
specific log-files e.g. for ingest in elasticsearch, etc.).

If nobody objects, i'll try to bring this to a state for committing this 
the next
days.

-g