naviserver-devel — Developer discussion

[naviserver-devel] vfs changes applied

[Zoran V. <zv...@ar...>]

Hi!

I have applied the TclVFS changes to the CVS tree.
The previous state is tagged as "before-tclvfs"
if you need to wholesale backoff the changes for
some reason.

I have tried (my best) to locate all possible uses
of OS file-related calls and re-writte them with
Tcl_FS counterparts.

Exceptions are: log.c, tclfile.c and nslog/nslog.c
where this was not possible due to interdependence
of channels and threads in Tcl.

Special cases are: adpeval.c, adpreqest.c, fastpath.c
and urlopen.c. I have followed different logic there.
First, the OS call is applied (access, stat, open, read,
write etc). If this call failed, the Tcl_FS conunterpart
is applied. If this also failed, the error is thrown.
This way the VFS is second-class citizen in parts sensible
to performance. I'm pretty sure that you will not hit any
performance problem this way.

The modload.c is entirely rewritten to utilitze Tcl_FSLoadFile
which is *very* clever beast. As a side-effect, some public
C-API calls have been removed, while obsolete (I already
reported that in one of my previous email).

Please test the new code and if you find something awkward,
tell me immediately so I can fix it.

 From this point, NS should be able to serve pages from
any Tcl_VFS compatible filesystem. Moreover, it should be
capable of starting out of starpack, etc... This is STILL NOT
supported NOR tested, but I will soon get this done as well.

My target is to be able to wrap-up a distribution consisting
of a single executable file containing all the shared libs
and data needed to run a web-site and/or application.

Cheers
Zoran 

[naviserver-devel] Darwin DNS

[Zoran V. <zv...@ar...>]

No luck: this (NetBSD) system does not have any MT-safe call.
What I did:

During the configure I emit the warning if compiling under Darwin.

checking for getaddrinfo in -lsocket... no
checking for getnameinfo in -lsocket... no
checking for getaddrinfo... yes
checking for getnameinfo... yes
checking for gethostbyname_r... no
checking for gethostbyaddr_r... no
configure: WARNING: DNS queries will use non-threadsafe calls which  
could result in server instability

In dns.c I added

#ifdef __APPLE__
    Ns_Cs cs;
    Ns_CsEnter(&cs);
    ...
    Ns_CsLeave(&cs);
#endif

to cope with this *at least* within our own program. This does not
guarantee that this is MT-safe because any of the rest of the
system may call those calls anytime, hence we get problems anyways.

Zoran

Re: [naviserver-devel] How to disable HAVE_GETNAMEINFO ?

[Zoran V. <zv...@ar...>]

Am 06.10.2005 um 20:54 schrieb Stephen Deasey:

>
>
> Around line 195 of configure.in there's this test:
>
> has_threadsafe_dns=no
> AC_CHECK_LIB(socket, getaddrinfo)
> AC_CHECK_LIB(socket, getnameinfo)
> AC_CHECK_FUNCS(getaddrinfo getnameinfo)
> if test "${ac_cv_func_getaddrinfo}" = "yes" \
>     -a "${ac_cv_func_getnameinfo}" = "yes" ; then
>     has_threadsafe_dns=yes
> fi
> if test "${has_threadsafe_dns}" != "yes" ; then
>     AC_HAVE_GETHOSTBYNAME_R
>     AC_HAVE_GETHOSTBYADDR_R
>     if test "${ac_cv_func_gethostbyname_r}" = "yes" \
>         -a "${ac_cv_func_gethostbyaddr_r}" = "yes" ; then
>         has_threadsafe_dns=yes
>     fi
> fi
> if test "${has_threadsafe_dns}" != "yes" ; then
>   AC_MSG_WARN([dns queries will use non-threadsafe calls which could
> result in server instability])
> fi
>
>
> Maybe you could check if $system == Darwin after the first check and
> set has_threadsafe_dns = no.
>
> At least this way all the checking is confined to the configure script
> and not scattered throughout the source.
>
> Even better would be to test for the brokenness rather than the
> platform.  Not sure what what the bug is or how easy it would be to
> check for.  Then when Apple fix this we start using getnameinfo again
> automatically.
>

OK. I found the problem with getnameinfo and also found out that none
of the apropriate calls on Darwin are thread safe... I will have to
check out how to resolve this...

Thanks for the hint. I do not understand how I did not see that
one... I grep'ed all the sources...

Zoran

Re: [naviserver-devel] How to disable HAVE_GETNAMEINFO ?

[Stephen D. <sd...@gm...>]

On 10/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi !
>
> I'm confused. Very.
>
> I need to disable the configure  machinery to
> define the HAVE_GETNAMEINFO and HAVE_GETADDRINFO
> for Darwin builds. Reason: those are broken on
> Darwin, and upon that, also MT-unsafe. OTOH,
> the classic gethostbyname/gethostbyaddr are
> working and are protected by critical section
> so I think it is better to use those.
>
> BUT: I do not see a clear way how to influence
> configure NOT to use/define those above !
>
> I COULD just
>
> #ifdef __APPLE
> # undef HAVE_GETADDRINFO
> # undef HAVE_GETNAMEINFO
> #endif
>
> but this is not elegant.
>
> Any ideas?
>
> Zoran
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>



Around line 195 of configure.in there's this test:

has_threadsafe_dns=3Dno
AC_CHECK_LIB(socket, getaddrinfo)
AC_CHECK_LIB(socket, getnameinfo)
AC_CHECK_FUNCS(getaddrinfo getnameinfo)
if test "${ac_cv_func_getaddrinfo}" =3D "yes" \
    -a "${ac_cv_func_getnameinfo}" =3D "yes" ; then
    has_threadsafe_dns=3Dyes
fi
if test "${has_threadsafe_dns}" !=3D "yes" ; then
    AC_HAVE_GETHOSTBYNAME_R
    AC_HAVE_GETHOSTBYADDR_R
    if test "${ac_cv_func_gethostbyname_r}" =3D "yes" \
        -a "${ac_cv_func_gethostbyaddr_r}" =3D "yes" ; then
        has_threadsafe_dns=3Dyes
    fi
fi
if test "${has_threadsafe_dns}" !=3D "yes" ; then
  AC_MSG_WARN([dns queries will use non-threadsafe calls which could
result in server instability])
fi


Maybe you could check if $system =3D=3D Darwin after the first check and
set has_threadsafe_dns =3D no.

At least this way all the checking is confined to the configure script
and not scattered throughout the source.

Even better would be to test for the brokenness rather than the
platform.  Not sure what what the bug is or how easy it would be to
check for.  Then when Apple fix this we start using getnameinfo again
automatically.

[naviserver-devel] How to disable HAVE_GETNAMEINFO ?

[Zoran V. <zv...@ar...>]

Hi !

I'm confused. Very.

I need to disable the configure  machinery to
define the HAVE_GETNAMEINFO and HAVE_GETADDRINFO
for Darwin builds. Reason: those are broken on
Darwin, and upon that, also MT-unsafe. OTOH,
the classic gethostbyname/gethostbyaddr are
working and are protected by critical section
so I think it is better to use those.

BUT: I do not see a clear way how to influence
configure NOT to use/define those above !

I COULD just

#ifdef __APPLE
# undef HAVE_GETADDRINFO
# undef HAVE_GETNAMEINFO
#endif

but this is not elegant.

Any ideas?

Zoran

Re: [naviserver-devel] Before I check in

[Zoran V. <zv...@ar...>]

Am 05.10.2005 um 20:07 schrieb Stephen Deasey:

> I wonder if some will also need to look for global variables, to
> checkl for module version or something...?
>
> Anyway, if Tcl implements simillar functionality, we can just migrate
> to the standard Tcl way of doing things.  No need to wrap
>


Tcl_FSLoadFile will allow you to checkup up to two symbols
while loading the shared file. Among others it also loads
the shared-lib or a bundle (mac osx beast) in one shot
from the file or from memory. I use this in modload.c
as the sole load/lookup function.

If everybody would use Ns_ModuleLoad, then we need no
other calls.

OK, so I will scrap the mentioned calls out of the +
pubic API.

Cheers
Zoran

Re: [naviserver-devel] Before I check in

[Stephen D. <sd...@gm...>]

Probably ok to remove.  Some modules (nsdb) implement their own
loading mechanism for e.g. drivers.  But they seem to only use
LoadModule (or whatever it's called, not at my computer right now).

I wonder if some will also need to look for global variables, to
checkl for module version or something...?

Anyway, if Tcl implements simillar functionality, we can just migrate
to the standard Tcl way of doing things.  No need to wrap



On 10/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
> ... just a quick question.
>
> Anybody using those C_API's:
>
> Ns_ModuleSymbol(CONST char *file, CONST char *name)
>     NS_GNUC_NONNULL(1) NS_GNUC_NONNULL(2);
>
> Ns_ModuleGetSymbol(CONST char *name)
>     NS_GNUC_NONNULL(1);
>
> ?
>
> Because I *removed* them out, given the fact that
> we now use Tcl_FsLoadFile() and you do not
> need those any more.
>
> One could simulate those (unneeded calls) with
> Tcl_FSLoadFile but I thought to save myself some work.
>
> In the server core, only the modload.c is using
> those calls. I do not know why they got exported on
> the C-API level in the first place.
>
> Cheers
> Zoran
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Power Architecture Resource Center: Free content, downloads, discussions,
> and more. http://solutions.newsforge.com/ibmarch.tmpl
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

Re: [naviserver-devel] Range: header parsing info

[Zoran V. <zv...@ar...>]

Am 04.10.2005 um 22:00 schrieb Stephen Deasey:
>
> The tests contain examples of valid and invalid range headers.
> There's a also reference to the RFC.
>
> See: tests/http_byteranges.test.
>
> I think they're actually pretty complete, so if the tests pass you
> haven't broken anything  :-).    But if you see some missing cases,
> feel free to add them.
>

Great! To be honest, I believe the initial ParseRange never passed
all those tests... I have rewritten it partially and all tests
pass OK now.

Still double/triple checking all before commiting...

Zoran

[naviserver-devel] Before I check in

[Zoran V. <zv...@ar...>]

... just a quick question.

Anybody using those C_API's:

Ns_ModuleSymbol(CONST char *file, CONST char *name)
     NS_GNUC_NONNULL(1) NS_GNUC_NONNULL(2);

Ns_ModuleGetSymbol(CONST char *name)
     NS_GNUC_NONNULL(1);

?

Because I *removed* them out, given the fact that
we now use Tcl_FsLoadFile() and you do not
need those any more.

One could simulate those (unneeded calls) with
Tcl_FSLoadFile but I thought to save myself some work.

In the server core, only the modload.c is using
those calls. I do not know why they got exported on
the C-API level in the first place.

Cheers
Zoran

Re: [naviserver-devel] Range: header parsing info

[Stephen D. <sd...@gm...>]

On 10/4/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
> Am 04.10.2005 um 09:37 schrieb Zoran Vasiljevic:
>
> > Hi,
> >
> > Vlad, you have made the ParseRange() call in the fastpath.c.
> > Can you help me understand the "Range:" header layout?
> > Is it something like:
> >
> >    Range: bytes=3Da-b,c-d,e-f
> >    or
> >    Range: bytes=3D-a
> >
> > Or, can it only be:
> >
> >    Range: bytes=3Da-b
> >    or
> >    Range: bytes=3D-a
> >
> > I have some trouble understanding this from the code.
> >
>
> Hm... after reading the code *again* (is ths true?)
>
>      /*
>       * Parse the "Range: bytes=3D" header. The header may contain zero
>       * or more range_spec's separated by ',' like:
>       *
>       *     bytes=3D[range_spec][,range_spec]...
>       *
>       * where range_spec could be one of:
>       *     first_byte_pos-last_byte_pos
>       *     -suffix_length
>       *
>       * The byte positions are inclusive; byte count starts at zero.
>       */


The tests contain examples of valid and invalid range headers.=20
There's a also reference to the RFC.

See: tests/http_byteranges.test.

I think they're actually pretty complete, so if the tests pass you
haven't broken anything  :-).    But if you see some missing cases,
feel free to add them.

Re: [naviserver-devel] TclVFS changes done (at last!)

[Zoran V. <zv...@ar...>]

Am 04.10.2005 um 23:25 schrieb Stephen Deasey:

> Try to commit the thread-safety fixes for nslog seperately, if
> possible.  We don't want to loose this important fix in the noise of
> generic fs changes.

OK.

>
> It looks like Tcl has [file link -sybolic a b].  Will this not replace
> ns_symlink?  No truncate though, that sucks.  TclX has ftruncate...
>

It has [file link symbolic] but is incompatible with what
ns_symlink is doing. Look:

lexxsrv:nscp 2> file mkdir /tmp/foo
lexxsrv:nscp 3> cd /tmp/foo
lexxsrv:nscp 4> exec touch bar
lexxsrv:nscp 5> ns_symlink bar barlink
lexxsrv:nscp 6> file link -symbolic tcllink bar
bar
lexxsrv:nscp 7> exec ls -l
-rw-rw-rw-   1 root  wheel   0 Oct  5 09:40 bar
lrwxrwxrwx   1 root  wheel   3 Oct  5 09:40 barlink -> bar
lrwxrwxrwx   1 root  wheel  20 Oct  5 09:42 tcllink -> /private/tmp/ 
foo/bar

You see? Thats why I wrote (tcl/compat.tcl):

#
# ns_symlink --
#
#   This is still implement in the server code. The reason is that
#   the Tcl [file link] command always creates link target with
#   absolute path to the linked file; nsd/tclfile.c:NsTclSymlinkObjCmd()
#

> ns_symlink?  No truncate though, that sucks.  TclX has ftruncate...

Well, not only truncate, but:


#
# ns_truncate --
#
#   This is still implement in the server code. The reason is that
#   the Tcl has no portable equivalent; nsd/ 
tclfile.c:NsTclFTruncateObjCmd()
#

#
# ns_ftruncate --
#
#   This is still implement in the server code. The reason is that
#   the Tcl has no portable equivalent; nsd/ 
tclfile.c:NsTclTruncateObjCmd()
#

#
# ns_mktemp --
#
#   This is still implement in the server code. The reason is that
#   the Tcl has no portable equivalent; nsd/ 
tclfile.c:NsTclMkTempObjCmd()
#

#
# ns_tempnam --
#
#   This is still implement in the server code. The reason is that
#   the Tcl has no portable equivalent; nsd/ 
tclfile.c:NsTclTempNamObjCmd()
#

I will now checkin the nslog. Then I will tag before-tcl-vfs
and then checking the VFS changes. At the moment the VFS changes
mean: no direct OS calls related to file-operations. All (most)
is done using Tcl wrappers. Exceptions are: temporary filenames,
pipes and sockets.

Cheers
Zoran

Re: [naviserver-devel] TclVFS changes done (at last!)

[Stephen D. <sd...@gm...>]

On 10/4/05, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi friends,
>
> I have made the TclVFS changes finally, believe or or not!
>
> Before wholesale commit, I will however tag the current
> CVS state as before-tcl-vfs (any better idea?) so you/we
> can backoff if we found some serious, non-fixable problem.
>
> I have replaced most of the internal direct OS calls
> with TclVFS pendants. Things dealing with temporary/debug
> files are however left unchanged.
>
> The tcl/compat.tcl now contains most of the ns_xxxxxx
> file-fiddling calls once done in C and actually augmented
> with the modern [file] command implementation. Please
> inspect and see if I missed something.
> Generally, things there are *removed* from the C-code.
> Those Tcl commands are really ment for back-compat and
> should really not be used; use the [file] instead.
> I haven't done the ns_symlink(a) and ns_truncate(b)
> because of the:
> (a) potential incombatiblity and
> (b) lack of appropriate functionality on the Tcl level
>
> Majority of VFS changes are rather unimportant for the
> speed *except* the adpeval.c and fastpath.c.
> There, I first try the OS then TclVFS path.  I believe
> you will see no performance penalty whoatsoever.
> That is, if you do not use Tcl virtual filesystem mappings,
> everything should work as before.
> Vlad, I rewrote the ParseRanges() in order to better
> understand it. I hope I did not screw anything. Do a
> quick glance there (after I commit) and doublecheck.
>
> Oh yes, if interested, I can give you diffs of files
> which have been changed in advance. Anybody?
>
> Caveat emptor: I did rewrite the nslog/nslog.c to use
> Tcl_Obj interface and have found that this part was
> *utterly* MT-unsafe, generally speaking. This has been
> now improved. Please take care to inspect your access.log
> files and check if everything is still as it should be.
>
> This is the list of changed files so far:
>
> M include/ns.h
> M include/nscheck.h
> M nsd/adpeval.c
> M nsd/adprequest.c
> M nsd/binder.c
> M nsd/config.c
> M nsd/conn.c
> M nsd/driver.c
> M nsd/fastpath.c
> M nsd/init.c
> M nsd/log.c
> M nsd/modload.c
> M nsd/nsconf.c
> M nsd/nsmain.c
> M nsd/pidfile.c
> M nsd/rollfile.c
> M nsd/tclcmds.c
> M nsd/tclfile.c
> M nsd/tclimg.c
> M nsd/urlopen.c
> M nslog/nslog.c
> M tcl/compat.tcl
>
> Again, please pay attention to nsd/adpeval.c
> and nsd/fastpath.c.
>
> Any objections to commit?


Sounds like a plan.

Try to commit the thread-safety fixes for nslog seperately, if
possible.  We don't want to loose this important fix in the noise of
generic fs changes.

It looks like Tcl has [file link -sybolic a b].  Will this not replace
ns_symlink?  No truncate though, that sucks.  TclX has ftruncate...

[naviserver-devel] TclVFS changes done (at last!)

[Zoran V. <zv...@ar...>]

Hi friends,

I have made the TclVFS changes finally, believe or or not!

Before wholesale commit, I will however tag the current
CVS state as before-tcl-vfs (any better idea?) so you/we
can backoff if we found some serious, non-fixable problem.

I have replaced most of the internal direct OS calls
with TclVFS pendants. Things dealing with temporary/debug
files are however left unchanged.

The tcl/compat.tcl now contains most of the ns_xxxxxx
file-fiddling calls once done in C and actually augmented
with the modern [file] command implementation. Please
inspect and see if I missed something.
Generally, things there are *removed* from the C-code.
Those Tcl commands are really ment for back-compat and
should really not be used; use the [file] instead.
I haven't done the ns_symlink(a) and ns_truncate(b)
because of the:
(a) potential incombatiblity and
(b) lack of appropriate functionality on the Tcl level

Majority of VFS changes are rather unimportant for the
speed *except* the adpeval.c and fastpath.c.
There, I first try the OS then TclVFS path.  I believe
you will see no performance penalty whoatsoever.
That is, if you do not use Tcl virtual filesystem mappings,
everything should work as before.
Vlad, I rewrote the ParseRanges() in order to better
understand it. I hope I did not screw anything. Do a
quick glance there (after I commit) and doublecheck.

Oh yes, if interested, I can give you diffs of files
which have been changed in advance. Anybody?

Caveat emptor: I did rewrite the nslog/nslog.c to use
Tcl_Obj interface and have found that this part was
*utterly* MT-unsafe, generally speaking. This has been
now improved. Please take care to inspect your access.log
files and check if everything is still as it should be.

This is the list of changed files so far:

M include/ns.h
M include/nscheck.h
M nsd/adpeval.c
M nsd/adprequest.c
M nsd/binder.c
M nsd/config.c
M nsd/conn.c
M nsd/driver.c
M nsd/fastpath.c
M nsd/init.c
M nsd/log.c
M nsd/modload.c
M nsd/nsconf.c
M nsd/nsmain.c
M nsd/pidfile.c
M nsd/rollfile.c
M nsd/tclcmds.c
M nsd/tclfile.c
M nsd/tclimg.c
M nsd/urlopen.c
M nslog/nslog.c
M tcl/compat.tcl

Again, please pay attention to nsd/adpeval.c
and nsd/fastpath.c.

Any objections to commit?

Cheers
Zoran

Re: [naviserver-devel] Range: header parsing info

[Zoran V. <zv...@ar...>]

Am 04.10.2005 um 09:37 schrieb Zoran Vasiljevic:

> Hi,
>
> Vlad, you have made the ParseRange() call in the fastpath.c.
> Can you help me understand the "Range:" header layout?
> Is it something like:
>
>    Range: bytes=a-b,c-d,e-f
>    or
>    Range: bytes=-a
>
> Or, can it only be:
>
>    Range: bytes=a-b
>    or
>    Range: bytes=-a
>
> I have some trouble understanding this from the code.
>

Hm... after reading the code *again* (is ths true?)

     /*
      * Parse the "Range: bytes=" header. The header may contain zero
      * or more range_spec's separated by ',' like:
      *
      *     bytes=[range_spec][,range_spec]...
      *
      * where range_spec could be one of:
      *     first_byte_pos-last_byte_pos
      *     -suffix_length
      *
      * The byte positions are inclusive; byte count starts at zero.
      */

Zoran

[naviserver-devel] Range: header parsing info

[Zoran V. <zv...@ar...>]

Hi,

Vlad, you have made the ParseRange() call in the fastpath.c.
Can you help me understand the "Range:" header layout?
Is it something like:

    Range: bytes=a-b,c-d,e-f
    or
    Range: bytes=-a

Or, can it only be:

    Range: bytes=a-b
    or
    Range: bytes=-a

I have some trouble understanding this from the code.

Thanks,
Zoran

[naviserver-devel] Tcl/Tk 2005 Conference - Schedules available now, Registration is open

[<aku...@sh...>]

Tcl/Tk 2005 Conference Schedule & Registration
==============================================

The 12th Tcl/Tk Conference Schedules are available.

The tutorials and paper presentation schedules have been 
finalized and are available at:

http://www.tcl.tk/community/tcl2005/tut2005.html
http://www.tcl.tk/community/tcl2005/schedule.html

The abstracts for the selected papers are available at:

http://www.tcl.tk/community/tcl2005/abstracts.html

The conference dinner will be on Wednesday evening.  
Blueteam will be providing a social hour with drinks
and munchies on Thursday evening.

Registration is open for tutorials and technical 
sessions at:

http://www.tcl.tk/community/tcl2005/reg.html


Program Committee:
==================
 Donal  Fellows       University of Manchester
 Clif   Flynt         Noumena Corp.
 Ron    Fox           NSCL Michigan State University
 Jeff   Hobbs         ActiveState Corp.
 Steve  Landers       Digital Smarties
 Gerald Lester        HMS Software
 Cyndy  Lilagan       Eolas Technologies Inc.
 Arjen  Markus        WL | Delft Hydraulics

--
Sincerely,
	Andreas Kupries <aku...@sh...>
			<http://www.purl.org/NET/akupries/>
-------------------------------------------------------------------------------

Re: [naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Am 30.07.2005 um 12:34 schrieb Stephen Deasey:

>
> I just wonder if the system is too simple.  In your example man page
> you followed each ccall definition with a description.  In the Tcl man
> pages the definitions are grouped at the top, then the args are
> described, then there's some prose text describing the functions.

This kind of layout is also possible to achieve, but I deliberately
avoided it because I find it extemely difficult to follow if you
have about 20 calls decribed in one page.

We can emit arg definitions a'la Tcl-man in the SYNOPSIS section,
this is no big deal.

>
> If the source looked like:
>
> ccall int FooFunc {{int arg}} {
>     p { description... }
>     p { more... }
> }
>
> I could see how a tree was being built up, and how formatting
> functions could walk it and output in any order, and use the data to
> synthesize lists and tables etc.  But with the simple macros embedded
> in literal text, it looks like the formatters will be limited to the
> order in which things are presented in the source man page.  Or am I
> way off here..?


You are not, but getting this done this way would mean even more
chages in the way how text is parsed. The doctools maintain some
state between passes and you have to fiddle with this, which I
really wanted to avoid. I already had to make changes to this state
by introducing the ccall formatter... Less changes = better.

I believe with some tweaks on the ccall systax we can get what
we want and avoid too much modifications. I will take some of the
ideas from here and give it another round. Lets see if that'll
be more appealing.

Zoran

Re: [naviserver-devel] Doc with doctools

[Stephen D. <sd...@gm...>]

On 7/30/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 30.07.2005 um 11:50 schrieb Stephen Deasey:
> >
> > Great.  Are the macros expanded recursively?  Can ccal output the type
> > declarations for the benefit of indexes etc.?  Will we be able to
> > document the Tcl cmds as easily?
>=20
> I will first have to examine the TOC and index parts before I can
> say something about that.
>=20
> Tcl commands are already handled by doctools with ease (doctools
> was designed arround documenting Tcl commands primarily).


Yeah, I was looking at your thread docs:

[call [cmd thread::create] [opt -joinable] [opt -preserved] [opt script]]

Same deal as with the ccall, all the [opt] stuff seems redundant.  For
documenting Tcl cmds, it would seem possible to use *exactly* the same
syntax as one would to define a proc:

call name -opt -- arg {arg2 def} args


> >
> > Oh, and can the index be categorized (used to live here:
> > http://aolserver.com/40drafts/ do you remember?) rather than just fill
> > the screen, as the Tcl API docs do:
> > http://www.tcl.tk/man/tcl8.4/TclLib/contents.htm ?
>=20
> Good question. Again, the TOC/Index parts I have not scratched
> yet. But I assume we can make just about anything once we understand
> the guts of it. I've made efforts to understand the formatting engine
> first to be able to get an idea how we could expand it to document
> C-calls nicely. Once this is done (and I believe we're very close to
> it), I will take the rest under examination.
>=20
> >
> > Also, is it possible to do things like mark a function deprecated, and
> > when, so that is doesn't show up in the normal doc index but an index
> > of deprecated functions is created, grouped by version number.
> >
> > How do we specify CONST, and can we document args as in, out or inout?
> >  How about NONNULL args, varargs and format strings?
>=20
> Uhuhuhuhu... args in/out is already handled. You can group args
> definitions
> more/less like Tcl is doing it in their docs (look "man
> Tcl_SetObjResult"
> for example).
>=20
> For the other, dunno. We'll have to see...
>=20
> >
> > I'm just wondering how flexible this thing is.  My mental model is
> > that it just calls [subst] on the man page source, so things like
> > literal text would seem hard to hide or move around...
> >
>=20
> It is pretty flexible, once you understand how it works. As it is
> all in plain Tcl and in just couple of relatively short files, you
> can pretty quickly get an idea how to change it, when needed.
> What I want to minimize is the scope of changes and use the thing
> more/less as-is in order to put a lower acceptance barrier on the
> doctools maintainer(s). If we manage to get at least "ccall" in
> and supprted *by them* it will be a great thing.


Agree.  Would be best to get any changes accepted upstream.

Also, no need to go nuts up front, we can add things later.

I just wonder if the system is too simple.  In your example man page
you followed each ccall definition with a description.  In the Tcl man
pages the definitions are grouped at the top, then the args are
described, then there's some prose text describing the functions.

If the source looked like:

ccall int FooFunc {{int arg}} {
    p { description... }
    p { more... }
}

I could see how a tree was being built up, and how formatting
functions could walk it and output in any order, and use the data to
synthesize lists and tables etc.  But with the simple macros embedded
in literal text, it looks like the formatters will be limited to the
order in which things are presented in the source man page.  Or am I
way off here..?

Re: [naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Am 30.07.2005 um 11:50 schrieb Stephen Deasey:
>
> Great.  Are the macros expanded recursively?  Can ccal output the type
> declarations for the benefit of indexes etc.?  Will we be able to
> document the Tcl cmds as easily?

I will first have to examine the TOC and index parts before I can
say something about that.

Tcl commands are already handled by doctools with ease (doctools
was designed arround documenting Tcl commands primarily).

>
> Oh, and can the index be categorized (used to live here:
> http://aolserver.com/40drafts/ do you remember?) rather than just fill
> the screen, as the Tcl API docs do:
> http://www.tcl.tk/man/tcl8.4/TclLib/contents.htm ?

Good question. Again, the TOC/Index parts I have not scratched
yet. But I assume we can make just about anything once we understand
the guts of it. I've made efforts to understand the formatting engine
first to be able to get an idea how we could expand it to document
C-calls nicely. Once this is done (and I believe we're very close to
it), I will take the rest under examination.

>
> Also, is it possible to do things like mark a function deprecated, and
> when, so that is doesn't show up in the normal doc index but an index
> of deprecated functions is created, grouped by version number.
>
> How do we specify CONST, and can we document args as in, out or inout?
>  How about NONNULL args, varargs and format strings?

Uhuhuhuhu... args in/out is already handled. You can group args  
definitions
more/less like Tcl is doing it in their docs (look "man  
Tcl_SetObjResult"
for example).

For the other, dunno. We'll have to see...

>
> I'm just wondering how flexible this thing is.  My mental model is
> that it just calls [subst] on the man page source, so things like
> literal text would seem hard to hide or move around...
>

It is pretty flexible, once you understand how it works. As it is
all in plain Tcl and in just couple of relatively short files, you
can pretty quickly get an idea how to change it, when needed.
What I want to minimize is the scope of changes and use the thing
more/less as-is in order to put a lower acceptance barrier on the
doctools maintainer(s). If we manage to get at least "ccall" in
and supprted *by them* it will be a great thing.

Zoran

Re: [naviserver-devel] Doc with doctools

[Stephen D. <sd...@gm...>]

On 7/30/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 30.07.2005 um 03:58 schrieb Stephen Deasey:
>=20
> >
> > [ccall [type int] Ns_ConnLocationAppend [type Ns_Conn] [arg *connPtr]
> > [type Ns_DString] [arg dest]]
> >
> > A C-call always has a name, always returns something (even if it's
> > just void), all args are typed and named, etc., and a lot of that can
> > be inferred from position and so on.  But in the above,  the doc
> > writer must redundantly specify [type Ns_Conn] [arg *connPtr].
> >
> > Couldn't the macro ccall just handle a lot of this transparently?
> > Something like:
> >
> > [ccall int Ns_ConnLocationAppend {
> >     {Ns_Conn *connPtr}
> >     {Ns_DString *dest}
> > } {
> >     Describe me here...
> > }]
> >
>=20
> Of course it could. This is how I did it in the first place.
> But, I'm still not sure and will have to investigate this,
> but it appears to me that other parts of the engine (TOC and Index
> generators) can take advantage of those [type] declarations.
> If this is not true, then there is absolutely no reason why
> not to code like that. I made the [ccall=10] resemble the [call]
> but there is no reason why one could/should not make it like:
>=20
>   ccall int Ns_ConnLocationAppend {{Ns_Conn *connPtr} {Ns_DString
> *dest}}
>=20
> I will put this into and give another round.


Great.  Are the macros expanded recursively?  Can ccal output the type
declarations for the benefit of indexes etc.?  Will we be able to
document the Tcl cmds as easily?

Oh, and can the index be categorized (used to live here:
http://aolserver.com/40drafts/ do you remember?) rather than just fill
the screen, as the Tcl API docs do:
http://www.tcl.tk/man/tcl8.4/TclLib/contents.htm ?

Also, is it possible to do things like mark a function deprecated, and
when, so that is doesn't show up in the normal doc index but an index
of deprecated functions is created, grouped by version number.

How do we specify CONST, and can we document args as in, out or inout?
 How about NONNULL args, varargs and format strings?


I'm just wondering how flexible this thing is.  My mental model is
that it just calls [subst] on the man page source, so things like
literal text would seem hard to hide or move around...

Re: [naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Am 30.07.2005 um 03:58 schrieb Stephen Deasey:

>
> [ccall [type int] Ns_ConnLocationAppend [type Ns_Conn] [arg *connPtr]
> [type Ns_DString] [arg dest]]
>
> A C-call always has a name, always returns something (even if it's
> just void), all args are typed and named, etc., and a lot of that can
> be inferred from position and so on.  But in the above,  the doc
> writer must redundantly specify [type Ns_Conn] [arg *connPtr].
>
> Couldn't the macro ccall just handle a lot of this transparently?
> Something like:
>
> [ccall int Ns_ConnLocationAppend {
>     {Ns_Conn *connPtr}
>     {Ns_DString *dest}
> } {
>     Describe me here...
> }]
>

Of course it could. This is how I did it in the first place.
But, I'm still not sure and will have to investigate this,
but it appears to me that other parts of the engine (TOC and Index
generators) can take advantage of those [type] declarations.
If this is not true, then there is absolutely no reason why
not to code like that. I made the [ccall=10] resemble the [call]
but there is no reason why one could/should not make it like:

  ccall int Ns_ConnLocationAppend {{Ns_Conn *connPtr} {Ns_DString =20
*dest}}

I will put this into and give another round.

Zoran
 =20=

Re: [naviserver-devel] Doc with doctools

[Stephen D. <sd...@gm...>]

On 7/27/05, Zoran Vasiljevic <zv...@ar...> wrote:
> Hi,
>=20
> Before I spend more time, just want to give you some
> idea about it. Please download those:
>=20
>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.man


The following two have the same contents:


>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.3
>     http://www.archiware.com/www/downloads/man.macros
>=20
> and use:
>=20
>     nroff -man Ns_ConnHeaders.3 | more
>=20
> to look at it.
>=20
> The ".man" is the source file I used. Out of this file
> we can make wiki, html, nroff etc pp formats with doctools.
>=20
> I have patched the doctools to include new formatting
> command "ccall" which stands for "C-call" and is a
> companion to "call" which formats Tcl ccommands.
>=20
> I did the nroff formatter first. Once we settle on this
> one, I will update the rest of formatters, eventually
> producting a private version of doctools for our needs.
> We can post changes to tcllib folks so they can include
> this in the standard distro if they like.
>=20
> While doing those changes, I now have much better insight
> how doctools are working and I can produce now just about
> any layout you'd like.
>=20
> Lets hear your feedback.
>=20
> Zoran


[ccall [type int] Ns_ConnLocationAppend [type Ns_Conn] [arg *connPtr]
[type Ns_DString] [arg dest]]

A C-call always has a name, always returns something (even if it's
just void), all args are typed and named, etc., and a lot of that can
be inferred from position and so on.  But in the above,  the doc
writer must redundantly specify [type Ns_Conn] [arg *connPtr].

Couldn't the macro ccall just handle a lot of this transparently?=20
Something like:

[ccall int Ns_ConnLocationAppend {
    {Ns_Conn *connPtr}
    {Ns_DString *dest}
} {
    Describe me here...
}]

Re: [naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Am 28.07.2005 um 07:18 schrieb Bernd Eidenschink:

>
> Hi Zoran,
>
> the doctools source looks good, should be manageable!

Yes. This is fairly simple.

>
>
>>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.man
>>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.3
>>
>
> This one seems not to be on your server:
>
>>     http://www.archiware.com/www/downloads/man.macros


Ups? It is now, sorry...

>>
>> and use:
>>
>>     nroff -man Ns_ConnHeaders.3 | more
>>
>
> Returns empty and Midnight Commander (mc) says "Empty output from  
> child
> filter". What do I do wrong?
>

I don't know. Alternatively, you can copy the Ns_ConnHeaders.3 into
/usr/local/man/man3 and just say "man Ns_ConnHeaders"...

Zoran

Re: [naviserver-devel] Doc with doctools

[Bernd E. <eid...@we...>]

Hi Zoran,

the doctools source looks good, should be manageable!

>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.man
>     http://www.archiware.com/www/downloads/Ns_ConnHeaders.3

This one seems not to be on your server:
>     http://www.archiware.com/www/downloads/man.macros
>
> and use:
>
>     nroff -man Ns_ConnHeaders.3 | more

Returns empty and Midnight Commander (mc) says "Empty output from child 
filter". What do I do wrong?

Bernd.

Re: [naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Am 27.07.2005 um 20:08 schrieb Vlad Seryakov:

> Looks interesting. So all the docs will need to be re-written in  
> doctools format.
>

Well, there is not much available out-there. But, giving it a doctool  
looks
we can concentrate on content and make nroff/html/wiki automatically by
a "make htmldoc" or "make mandoc" or whatever...

zoran