Menu
▾
▴
naviserver-devel
|
From: <vl...@cr...> - 2005-05-30 02:00:37
|
Perfect -------- Original Message -------- To: nav...@li... From: Zoran Vasiljevic <zv...@ar...> Subject: Re: [Re: [naviserver-devel] RFE #1202462] Am 29.05.2005 um 15:24 schrieb vl...@cr...: > If that will be configurable, let's say with --with-vfs during > configure or somehow to have the ability using low-level or Tcl > interfaces, i have no problems with it. The cases when direct calls > would be necessary for high performance sites, but in cases when > performance is acceptable Tcl Vfs can be used. OK. This is fine with me. I will fix the mmap usage so it can be used from windows as well. In the same time, I'll define Ns_XXX wrappers for doing file stuff and those will be ifdef'ed USE_TCL_VFS. The USE_TCL_VFS will be undefined by default. Is this OK so? Zoran ------------------------------------------------------- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 _______________________________________________ naviserver-devel mailing list nav...@li... https://lists.sourceforge.net/lists/listinfo/naviserver-devel |
|
From: Zoran V. <zv...@ar...> - 2005-05-30 09:16:44
|
Am 30.05.2005 um 01:59 schrieb vl...@cr...: > Perfect > Even more "perfect": instead of compile-time, you could select vfs operation as a ns/server config option. I'd have to write Ns_FSStat, Ns_FSOpen, Ns_FSDirents etc wrappers *anyway*. So instead of ifdefing pieces of code in there, I can make the judgement on the basis of checking the config setup which is one quick operation. This is more friendly and would allow people to easily experiment with the performance and pros/cons added with the vfs interface w/o the hassle of recompiling all. I think this is the proper way to go: ns_section ns/server ns_param usetclvfs (false|true) I'd put this process-wide instead of virtual-server-wide because I do not know if I have access to the current virtual server from all places where those calls are made. Zoran |
|
From: Stephen D. <sd...@gm...> - 2005-05-30 10:01:15
|
On 5/30/05, Zoran Vasiljevic <zv...@ar...> wrote: >=20 > Am 30.05.2005 um 01:59 schrieb vl...@cr...: >=20 > > Perfect > > >=20 > Even more "perfect": >=20 > instead of compile-time, you could select vfs operation > as a ns/server config option. >=20 > I'd have to write Ns_FSStat, Ns_FSOpen, Ns_FSDirents > etc wrappers *anyway*. So instead of ifdefing pieces of > code in there, I can make the judgement on the basis of > checking the config setup which is one quick operation. > This is more friendly and would allow people to easily > experiment with the performance and pros/cons added with > the vfs interface w/o the hassle of recompiling all. What's the return value of Ns_FSOpen? |
|
From: Zoran V. <zv...@ar...> - 2005-05-30 11:02:53
|
Am 30.05.2005 um 12:01 schrieb Stephen Deasey: > > What's the return value of Ns_FSOpen? > Something that can be fed to Ns_FSRead and/or Ns_FSClose for example. I haven't started the implementation yet but this is first what comes to mind. Presumably, it will be Tcl_Channel in VFS or a regular OS filehandle w/o VFS. Is there any specific (hidden) reason why you took this particular exmaple? If you think about mmap, this should be wrapped entirely so you'd give a file to mmap and optionally the address to map it to. All the rest will be done by the implementation. Zoran |
|
From: Stephen D. <sd...@gm...> - 2005-06-01 12:10:40
|
On 5/30/05, Zoran Vasiljevic <zv...@ar...> wrote: >=20 > Am 30.05.2005 um 12:01 schrieb Stephen Deasey: >=20 > > > > What's the return value of Ns_FSOpen? > > >=20 > Something that can be fed to Ns_FSRead and/or Ns_FSClose > for example. I haven't started the implementation yet > but this is first what comes to mind. >=20 > Presumably, it will be Tcl_Channel in VFS or a regular > OS filehandle w/o VFS. The return type would be unknown unless the caller also used #ifdef? The issue is not preferring open over Tcl_VFSOpen, and it's not mmap per se. mmap is just one example of a function which requires a file descriptor or real files in the file system. sendfile is another, as are readv/writev, epoll, kqueue, inotify, etc. The Tcl VFS is a lowest common denominator abstraction over file systems. It provides the common things, but can't provide implementations of everything, nor can it return a useful file descriptor from a Tcl_Channel, at least in the case where everything is contained within one big file. I think Ns_TclGetOpenFd is going to fail. I don't see anything wrong with using Tcl_VFS* directly to read the config file, write to a log file etc., and the tcl commands in nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's the stuff in nsd/fastpath.c that's important. File descriptors need to be available for use with modern, high performance system calls with a minimum of conditional compilation/execution. This also has the potential to be a never-ending bug hunt. Absolutely every call to the file system has to be converted to Tcl_VFS* or Ns_VFS*, including all modules and the libraries they wrap, or people will be surprised... I'm not against this feature, but it's pretty esoteric and the implementation is invasive so the quality bar needs to be set high.=20 I'm sure you can do it :-) |
|
From: Zoran V. <zv...@ar...> - 2005-06-01 18:20:07
|
Am 01.06.2005 um 14:10 schrieb Stephen Deasey: > On 5/30/05, Zoran Vasiljevic <zv...@ar...> wrote: > >> >> Am 30.05.2005 um 12:01 schrieb Stephen Deasey: >> >> >>> >>> What's the return value of Ns_FSOpen? >>> >>> >> >> Something that can be fed to Ns_FSRead and/or Ns_FSClose >> for example. I haven't started the implementation yet >> but this is first what comes to mind. >> >> Presumably, it will be Tcl_Channel in VFS or a regular >> OS filehandle w/o VFS. >> > > > The return type would be unknown unless the caller also used #ifdef? The return value is of course opaque and should be returned to the FS abstraction layer. But I read on... > > I don't see anything wrong with using Tcl_VFS* directly to read the > config file, write to a log file etc., and the tcl commands in > nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's > the stuff in nsd/fastpath.c that's important. File descriptors need > to be available for use with modern, high performance system calls > with a minimum of conditional compilation/execution. > > This also has the potential to be a never-ending bug hunt. Absolutely > every call to the file system has to be converted to Tcl_VFS* or > Ns_VFS*, including all modules and the libraries they wrap, or people > will be surprised... > I'm not against this feature, but it's pretty esoteric and the > implementation is invasive so the quality bar needs to be set high. > I'm sure you can do it :-) > Of course, all you say holds. If you need to go to the max when accesing files, then Tcl VFS will not be the fastest kid on the block. But what about this: we make a Ns_FS wrappers and put them in place, even in the fastpath and adp code. We also add Ns_FSGetOSHandle which will then return invalid handle to the caller if the file is not located on the native filesystem. This is trivial and costs no performance. Now, the caller can accomodate for that and decide what to do. A very good example is the actual fastpath code which does the same with the mmap. It works-around the fact that mmap is not defined in the server config and uses simple open/read/close sequence. This way you'd be free to add whatever fancy roaring implementation (kaio for example) to get the max-speed, if you like. We need only cover access to files, no sockets, pipes, etc. What we'd need is just a handful of calls: open, close, stat, seek, read, write, opendir, readdir, closedir, getcwd (did I forgot something?) Most of them would use the opaque file-handle from which you can obtain the OS handle to make any private speed optimizations. If the handle cannot be obtained, then you provide the best-effort fallback. What would/could be wrong with that? Zoran |
|
From: Stephen D. <sd...@gm...> - 2005-06-01 19:27:23
|
On 6/1/05, Zoran Vasiljevic <zv...@ar...> wrote: >=20 > Am 01.06.2005 um 14:10 schrieb Stephen Deasey: >=20 > > On 5/30/05, Zoran Vasiljevic <zv...@ar...> wrote: > > > >> > >> Am 30.05.2005 um 12:01 schrieb Stephen Deasey: > >> > >> > >>> > >>> What's the return value of Ns_FSOpen? > >>> > >>> > >> > >> Something that can be fed to Ns_FSRead and/or Ns_FSClose > >> for example. I haven't started the implementation yet > >> but this is first what comes to mind. > >> > >> Presumably, it will be Tcl_Channel in VFS or a regular > >> OS filehandle w/o VFS. > >> > > > > > > The return type would be unknown unless the caller also used #ifdef? >=20 > The return value is of course opaque and should be returned > to the FS abstraction layer. But I read on... >=20 > > > > I don't see anything wrong with using Tcl_VFS* directly to read the > > config file, write to a log file etc., and the tcl commands in > > nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's > > the stuff in nsd/fastpath.c that's important. File descriptors need > > to be available for use with modern, high performance system calls > > with a minimum of conditional compilation/execution. > > > > This also has the potential to be a never-ending bug hunt. Absolutely > > every call to the file system has to be converted to Tcl_VFS* or > > Ns_VFS*, including all modules and the libraries they wrap, or people > > will be surprised... >=20 > > I'm not against this feature, but it's pretty esoteric and the > > implementation is invasive so the quality bar needs to be set high. > > I'm sure you can do it :-) > > >=20 > Of course, all you say holds. If you need to go to the max when > accesing files, then Tcl VFS will not be the fastest kid on the block. >=20 > But what about this: we make a Ns_FS wrappers and put them in place, > even in the fastpath and adp code. We also add Ns_FSGetOSHandle which > will then return invalid handle to the caller if the file is not located > on the native filesystem. This is trivial and costs no performance. > Now, the caller can accomodate for that and decide what to do. > A very good example is the actual fastpath code which does the same > with the mmap. It works-around the fact that mmap is not defined in > the server config and uses simple open/read/close sequence. >=20 > This way you'd be free to add whatever fancy roaring implementation > (kaio for example) to get the max-speed, if you like. > We need only cover access to files, no sockets, pipes, etc. > What we'd need is just a handful of calls: >=20 > open, close, stat, seek, read, write, opendir, readdir, closedir, > getcwd >=20 > (did I forgot something?) >=20 > Most of them would use the opaque file-handle from which you can > obtain the OS handle to make any private speed optimizations. If the > handle cannot be obtained, then you provide the best-effort fallback. >=20 > What would/could be wrong with that? Sounds good. |
|
From: Zoran V. <zv...@ar...> - 2005-06-01 20:47:14
|
Am 01.06.2005 um 21:27 schrieb Stephen Deasey: > > Sounds good. > OK. I will start the work and give diffs for review when I come to some point. Zoran |
|
From: Zoran V. <zv...@ar...> - 2005-06-14 13:20:43
|
Am 01.06.2005 um 20:19 schrieb Zoran Vasiljevic:
>> I don't see anything wrong with using Tcl_VFS* directly to read the
>> config file, write to a log file etc., and the tcl commands in
>> nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's
>> the stuff in nsd/fastpath.c that's important. File descriptors need
>> to be available for use with modern, high performance system calls
>> with a minimum of conditional compilation/execution.
>>
>> This also has the potential to be a never-ending bug hunt.
>> Absolutely
>> every call to the file system has to be converted to Tcl_VFS* or
>> Ns_VFS*, including all modules and the libraries they wrap, or people
>> will be surprised...
>
>> I'm not against this feature, but it's pretty esoteric and the
>> implementation is invasive so the quality bar needs to be set high.
>> I'm sure you can do it :-)
>
I have thought about all this...
We do have consensus on fact that fastpath code is actually the
core of the performance respective the filesystem access. All
other parts where files are read and/or written is not speed
relevant. Correct?
Now, if I simplify the above task by making the *rest* of the
server VFS aware (replacing all native FS calls with Tcl wrappers)
and add one additional config-option to fastpath code like the
mmap is today?
So, one who needs to serve pages out of container files, would
simply toggle the:
ns/server/$server/fastpath/vfs
switch to 1 (the default would be zero: not use vfs).
This way, the task is pretty simple and straight-forward.
What do you think?
Zoran
|
|
From: Vlad S. <vl...@cr...> - 2005-06-14 13:40:31
|
My question is: will it affect service Tcl pages as well. For example using templating system it opens .tcl and .adp files. Will Vfs slow down the oevrall performance? Zoran Vasiljevic wrote: > > Am 01.06.2005 um 20:19 schrieb Zoran Vasiljevic: > >>> I don't see anything wrong with using Tcl_VFS* directly to read the >>> config file, write to a log file etc., and the tcl commands in >>> nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's >>> the stuff in nsd/fastpath.c that's important. File descriptors need >>> to be available for use with modern, high performance system calls >>> with a minimum of conditional compilation/execution. >>> >>> This also has the potential to be a never-ending bug hunt. Absolutely >>> every call to the file system has to be converted to Tcl_VFS* or >>> Ns_VFS*, including all modules and the libraries they wrap, or people >>> will be surprised... >> >> >>> I'm not against this feature, but it's pretty esoteric and the >>> implementation is invasive so the quality bar needs to be set high. >>> I'm sure you can do it :-) >> >> > > > I have thought about all this... > > We do have consensus on fact that fastpath code is actually the > core of the performance respective the filesystem access. All > other parts where files are read and/or written is not speed > relevant. Correct? > > Now, if I simplify the above task by making the *rest* of the > server VFS aware (replacing all native FS calls with Tcl wrappers) > and add one additional config-option to fastpath code like the > mmap is today? > So, one who needs to serve pages out of container files, would > simply toggle the: > > ns/server/$server/fastpath/vfs > > switch to 1 (the default would be zero: not use vfs). > This way, the task is pretty simple and straight-forward. > > What do you think? > > Zoran > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: NEC IT Guy Games. How far can you > shotput > a projector? How fast can you ride your desk chair down the office luge > track? > If you want to score the big prize, get to know the little guy. Play to > win an NEC 61" plasma display: http://www.necitguy.com/?r=20 > _______________________________________________ > naviserver-devel mailing list > nav...@li... > https://lists.sourceforge.net/lists/listinfo/naviserver-devel -- Vlad Seryakov 571 262-8608 office vl...@cr... http://www.crystalballinc.com/vlad/ |
|
From: Zoran V. <zv...@ar...> - 2005-06-14 13:49:07
|
Am 14.06.2005 um 15:39 schrieb Vlad Seryakov: > My question is: will it affect service Tcl pages as well. For > example using templating system it opens .tcl and .adp files. Will > Vfs slow down the oevrall performance? I expected this question! I believe the overhead of processing an adp page (or Tcl procedure) will *by far* exceede the actual open/read/close vs. the VFS equivalent. I think we are there in sub-msec range whereas the actual adp/tcl execution falls into the msec to 10's or 100's of msec. I do not believe that this is going to be of any issue. We can test this by adding a wrapper in the adpcode where the file is read-in and checking the performance. Note: the entire Tcl core is using those wrappers. That is, whatever you do with files from Tcl, everything passes thru VFS layer. Zoran |
|
From: Stephen D. <sd...@gm...> - 2005-06-15 06:55:02
|
On 6/14/05, Zoran Vasiljevic <zv...@ar...> wrote: >=20 > Am 01.06.2005 um 20:19 schrieb Zoran Vasiljevic: >=20 > >> I don't see anything wrong with using Tcl_VFS* directly to read the > >> config file, write to a log file etc., and the tcl commands in > >> nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's > >> the stuff in nsd/fastpath.c that's important. File descriptors need > >> to be available for use with modern, high performance system calls > >> with a minimum of conditional compilation/execution. > >> > >> This also has the potential to be a never-ending bug hunt. > >> Absolutely > >> every call to the file system has to be converted to Tcl_VFS* or > >> Ns_VFS*, including all modules and the libraries they wrap, or people > >> will be surprised... > > > >> I'm not against this feature, but it's pretty esoteric and the > >> implementation is invasive so the quality bar needs to be set high. > >> I'm sure you can do it :-) > > >=20 >=20 > I have thought about all this... >=20 > We do have consensus on fact that fastpath code is actually the > core of the performance respective the filesystem access. All > other parts where files are read and/or written is not speed > relevant. Correct? >=20 > Now, if I simplify the above task by making the *rest* of the > server VFS aware (replacing all native FS calls with Tcl wrappers) > and add one additional config-option to fastpath code like the > mmap is today? > So, one who needs to serve pages out of container files, would > simply toggle the: >=20 > ns/server/$server/fastpath/vfs >=20 > switch to 1 (the default would be zero: not use vfs). > This way, the task is pretty simple and straight-forward. >=20 > What do you think? How will you load modules at startup? dlopen() etc. take a file name. |
|
From: Zoran V. <zv...@ar...> - 2005-06-15 07:07:33
|
Am 15.06.2005 um 08:54 schrieb Stephen Deasey:
>
> How will you load modules at startup? dlopen() etc. take a file name.
>
I will start with this and see if it suffices:
Tcl_FSLoadFile dynamically loads a binary code file into
memory and
returns the addresses of two procedures within that file, if
they are
defined. The appropriate function for the filesystem to
which pathPtr
belongs will be called. If that filesystem does not
implement this
function (most virtual filesystems will not, because of OS
limitations
in dynamically loading binary code), Tcl will attempt to copy
the file
to a temporary directory and load that temporary file.
Zoran
|
|
From: Stephen D. <sd...@gm...> - 2005-06-15 07:26:57
|
On 6/15/05, Zoran Vasiljevic <zv...@ar...> wrote: >=20 > Am 15.06.2005 um 08:54 schrieb Stephen Deasey: > > > > How will you load modules at startup? dlopen() etc. take a file name. > > >=20 > I will start with this and see if it suffices: >=20 > Tcl_FSLoadFile dynamically loads a binary code file into > memory and > returns the addresses of two procedures within that file, if > they are > defined. The appropriate function for the filesystem to > which pathPtr > belongs will be called. If that filesystem does not > implement this > function (most virtual filesystems will not, because of OS > limitations > in dynamically loading binary code), Tcl will attempt to copy > the file > to a temporary directory and load that temporary file. Oh, that'll work. How about registering your own handler on /* just as the core server does at startup? That way the fastpath code won't be called unless you explicitly call it. You can send files using Ns_ConnSendChannel(). |
|
From: Zoran V. <zv...@ar...> - 2005-06-15 07:42:59
|
Am 15.06.2005 um 09:26 schrieb Stephen Deasey: > > How about registering your own handler on /* just as the core server > does at startup? That way the fastpath code won't be called unless > you explicitly call it. You can send files using > Ns_ConnSendChannel(). > I believe there are some side-effect on overloading the entire root with a custom handler... I cannot recall exactly, but I know there are. Zoran |
|
From: Zoran V. <zv...@ar...> - 2005-06-15 08:06:50
|
Am 15.06.2005 um 09:26 schrieb Stephen Deasey:
> On 6/15/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
>>
>> Am 15.06.2005 um 08:54 schrieb Stephen Deasey:
>>
>>>
>>> How will you load modules at startup? dlopen() etc. take a file =20
>>> name.
>>>
>>>
>>
>> I will start with this and see if it suffices:
>>
>> Tcl_FSLoadFile dynamically loads a binary code file into
>> memory and
>> returns the addresses of two procedures within that file, if
>> they are
>> defined. The appropriate function for the filesystem to
>> which pathPtr
>> belongs will be called. If that filesystem does not
>> implement this
>> function (most virtual filesystems will not, because of OS
>> limitations
>> in dynamically loading binary code), Tcl will attempt to copy
>> the file
>> to a temporary directory and load that temporary file.
>>
>
>
> Oh, that'll work.
>
> How about registering your own handler on /* just as the core server
> does at startup? That way the fastpath code won't be called unless
> you explicitly call it. You can send files using
> Ns_ConnSendChannel().
I believe we are basically talking about this code in fastpath.c:
if (servPtr->fastpath.mmap
&& NsMemMap(file, stPtr->st_size, NS_MMAP_READ, &fmap) =20
=3D=3D NS_OK) {
result =3D Ns_ConnReturnData(conn,status, =20
fmap.addr,fmap.size, type);
NsMemUmap(&fmap);
} else {
fd =3D open(file, O_RDONLY | O_BINARY);
if (fd =3D=3D -1) {
Ns_Log(Warning, "fastpath: open(%s) failed: %s", file,
strerror(errno));
goto notfound;
}
result =3D Ns_ConnReturnOpenFd(conn, status,type, fd,stPtr-=20=
>st_size);
close(fd);
}
I recently added wrappers for platform-neutral mmap. I can imagine
something like:
if (servPtr->fastpath.mmap
&& NsMemMap(file, stPtr->st_size, NS_MMAP_READ, &fmap) =20
=3D=3D NS_OK) {
result =3D Ns_ConnReturnData(conn,status, =20
fmap.addr,fmap.size, type);
NsMemUmap(&fmap);
=10=10=10=10=10=10=10=10=10=10=10=10=10=10=10=10} else if =
(servPtr->fastpath.vfs
&& NsVfsLoad(file, buf) =3D=3D NS_OK) {
result =3D Ns_ConnReturnData(conn, status, buf, stPtr-=20
>st_size, type);
} else {
fd =3D open(file, O_RDONLY | O_BINARY);
if (fd =3D=3D -1) {
Ns_Log(Warning, "fastpath: open(%s) failed: %s", file,
strerror(errno));
goto notfound;
}
result =3D Ns_ConnReturnOpenFd(conn, status,type, fd,stPtr-=20=
>st_size);
close(fd);
}
This will induce absolutely no performance hit and it is about only
place where speed matters, AFAIK.
Zoran
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X