[ naviserver-Feature Requests-1202462 ] Rewrite to use Tcl VFS filesystem abstraction

[SourceForge.net <no...@so...>]

Feature Requests item #1202462, was opened at 2005-05-15 22:14
Message generated for change (Comment added) made by vasiljevic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=719009&aid=1202462&group_id=130646

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: C-API
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Zoran Vasiljevic (vasiljevic)
Assigned to: Zoran Vasiljevic (vasiljevic)
Summary: Rewrite to use Tcl VFS filesystem abstraction

Initial Comment:
We should really invest some time and rewrite all
(most?) of the
internal usages of open(), stat() (etc) so we can take
advantage of
Tcl virtual filesystem wappers. 


----------------------------------------------------------------------

>Comment By: Zoran Vasiljevic (vasiljevic)
Date: 2005-06-17 21:22

Message:
Logged In: YES 
user_id=95086

OK. I will proceede as agreed.

FYI, the fastpath code will use mmap only if
   fastpath.map is turned on
   fastpath cache is disabled 
      or
   entry is too large for the cache.

Only under those circumstances, the mmap is attempted.



----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-06-17 21:15

Message:
Logged In: YES 
user_id=87254

Basically, YES.

I don't think you need the vfs configuration flag.  The
check to see if mmap is enabled comes before the vfs flag in
the fastpath code, so no matter what you set vfs to, it may
be overidden by mmap.

You should drop the vfs flag and make vfs the default if
mmap is not enabled (as that is currently the only other
option).  i.e. also drop the fallback to normal open() etc.
if vfs is not enabled.

The original concern was that the file descriptor would not
be available so that more advanced IO calls could be used,
not that Tcl_VFSOpen() is noticably slower than open() etc.
 But you've shown how to deal with that, so I think just use
Tcl VFS throughout, for simplicity.

The same applies to the adp code, just use Tcl VFS.  Any
possible minor slowdown when accessing the file sytem should
be hidden by the ADP memory cache anyway.

So, the instruction for folks who want to use the Tcl VFS
hooks to overide normal FS processing would be to make sure
mmap (currently, plus any future optimisation) is turned off.



----------------------------------------------------------------------

Comment By: Zoran Vasiljevic (vasiljevic)
Date: 2005-06-17 11:58

Message:
Logged In: YES 
user_id=95086

This is how I plan to proceede on this topic.

1. Rewrite all filesystem-specific calls to use Tcl pendants
 except thoser related to ADP and fastpath processing.
This would induce no speed penalties to the request
processing code.

2. Add fastpath.vfs flag which will, if set, use Tcl VFS
path *after*
exhausting all other possiblities (mmaping, direct access to
file).
This way one can override a page from the container file by
providing
the page on the filesystem. The vfs toggle will bet set to
false by default.

3. Change ADP to use TclVFS in the same manner: first trying the
file on the filesystem (as done now) and if it fails, using
Tcl VFS 
wrappers.

This way the VFS is a second-class citizen and would only be
used
as a fallback if the file cannot be otherwise found on the
filesystem
and if the VFS toggle has been flipped.

Please give YES or NO (if NO, tell why)



----------------------------------------------------------------------

Comment By: Zoran Vasiljevic (vasiljevic)
Date: 2005-05-16 13:43

Message:
Logged In: YES 
user_id=95086

Oh yes. The packaging is what I'm (mostly) after. 

The only parts where mmap is used is fastpath.c.
I do not know what/where is sendfile...

So, effectively, the mmap has to be *enabled* to be
used, right? Well, there is nothing wrong in leaving the
mmap inplace. Actually, it should be also done in the 
way that Windows platform can use it (nees some wrapping
but its easy done). 

But, it mmap is not enabled? Then the server uses standard
open/read/close stuff from the OS. Now, what's wrong  in
using Tcl API equivalents of that? In that case, it is transparent
where the file actually comes from, filesystem, zip/tarfile, etc...
Do you think this could hit us performance-wise. I do not know.
Maybe, but it's worth trying it out.  

Also, there are whole lotta places where the server uses OS
calls to the filesystem which has nothing to do with serving
pages (logfiles, access files, config files, etc...). These are 
not sensitive performance-wise and those places should/could
be rewritten so we can bootstrap from zipfile if needed.
Or be entirely wrapped in a startkit.

What do you think?

----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-05-16 13:30

Message:
Logged In: YES 
user_id=87254


I'd be reluctant to let go of mmap and sendfile, considering
our niche as a performace server.

You can already use the vfs to read zip files etc., it's the
single file packaging your after, right?


----------------------------------------------------------------------

Comment By: Zoran Vasiljevic (vasiljevic)
Date: 2005-05-16 08:55

Message:
Logged In: YES 
user_id=95086

The advantage is that you can for example, assemble all of the server
distro in a starkit and distribute it as a wrapped application.
Also, all sorts of filesystem wrappers (tar, zip, metakit, http, ...)
are already implemeted in TclVFS so you can serve pages out of 
the tar/zip files directly, for example.

No, mmap kind of things would be not possible since all would be 
going thru Tcl API. But, where is the mmap used? I believe only 
in fastpath and only if you explicitly enable it. And, only for Unix
port. 





----------------------------------------------------------------------

Comment By: Stephen Deasey (sdeasey)
Date: 2005-05-16 01:25

Message:
Logged In: YES 
user_id=87254

What would be the advantage of using the Tcl VFS?  Would it
be just as fast?  Could we still use performance enhancemts
such as mmap() and sendfile()?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=719009&aid=1202462&group_id=130646