naviserver-devel — Developer discussion

[naviserver-devel] Doc with doctools

[Zoran V. <zv...@ar...>]

Hi,

Before I spend more time, just want to give you some
idea about it. Please download those:

    http://www.archiware.com/www/downloads/Ns_ConnHeaders.man
    http://www.archiware.com/www/downloads/Ns_ConnHeaders.3
    http://www.archiware.com/www/downloads/man.macros

and use:

    nroff -man Ns_ConnHeaders.3 | more

to look at it.

The ".man" is the source file I used. Out of this file
we can make wiki, html, nroff etc pp formats with doctools.

I have patched the doctools to include new formatting
command "ccall" which stands for "C-call" and is a
companion to "call" which formats Tcl ccommands.

I did the nroff formatter first. Once we settle on this
one, I will update the rest of formatters, eventually
producting a private version of doctools for our needs.
We can post changes to tcllib folks so they can include
this in the standard distro if they like.

While doing those changes, I now have much better insight
how doctools are working and I can produce now just about
any layout you'd like.

Lets hear your feedback.

Zoran

Re: Fwd: [naviserver-devel] Commits, ns_register_adp

[Bernd E. <eid...@we...>]

Hi Stephen,

thanks for your explanation!

Fwd: [naviserver-devel] Commits, ns_register_adp

[Stephen D. <sd...@gm...>]

On 7/12/05, Bernd Eidenschink <eid...@we...> wrote:
>=20
> Hi Stephen,
>=20
> I just read your commits. Can you please explain the effects of your work=
 to
> me for this:
> >Fix the confusion around the default adp
> >request handler and ns_register_adp and add the missing proc
> >callback info.  ns_info requestprocs now gets things
> >right.
>=20
> and that:
> >ns_register_adp will also now register the default adp
> >request handler if you don't specify an optional file to map.
>=20
> What does this exactly mean?


If you haven't updated yet, login to a command shell (make runtest) and:

% join [ns_info requestprocs] \n

You'll see that the ADP request handlers are not named.  That's weird
since I remember adding the names to nsd/proc.c myself...  What I
actually named was the ADP request handler that gets registered when
you call ns_register_adp, which is subtly different.

ns_register_adp GET /aurl /home/stephen/my.adp

What happens here is that /aurl, /aurl/foo, etc. all end up running
my.adp.  I modified this function to make the adpfile optional.  Now
you can call:

ns_register_adp GET /adps

and normal adp processing will happen for that directory.  You might
want to do this if you register a default handler for the root URL,
but want ADPs in a sub directory.  So here's how it looks now:

% ns_register_proc GET /foo/* foo_handler arg
% ns_register_adp GET /foo/adp/*
% ns_register_adp GET /foo/x.adp /var/x.adp
% join [ns_info requestprocs] \n

test GET * ns:fastget test
test HEAD * ns:fastget test
test POST * ns:fastget test
test GET *.adp ns:adprequest test
test HEAD *.adp ns:adprequest test
test POST *.adp ns:adprequest test
test GET *.tcl ns:tclrequest ns_sourceproc {}
test HEAD *.tcl ns:tclrequest ns_sourceproc {}
test POST *.tcl ns:tclrequest ns_sourceproc {}
test GET /foo* ns:tclrequest foo_handler arg
test GET /foo/adp* ns:adprequest test
test GET /foo/x.adp* ns:adpmap /var/x.adp

Re: [naviserver-devel] Last commit

[Zoran V. <zv...@ar...>]

Am 12.07.2005 um 18:15 schrieb Vlad Seryakov:

> We do a lot of huge uploads and i figured while i have spare time  
> for this and need i will try. I did exactly as you defined in your  
> emails.
> Not protable though but i think usable to see how it will perform.
> i am going to install it on one of my server which does uploads and  
> watch for stability and memory usage.
>
>

Oh, sorry, I see you are using the temp file already...
Zoran

Re: [naviserver-devel] Last commit

[Zoran V. <zv...@ar...>]

Am 12.07.2005 um 18:15 schrieb Vlad Seryakov:

> We do a lot of huge uploads and i figured while i have spare time  
> for this and need i will try. I did exactly as you defined in your  
> emails.
> Not protable though but i think usable to see how it will perform.
> i am going to install it on one of my server which does uploads and  
> watch for stability and memory usage.
>
>

I think we should, for the final solution, perform the filling of the
mmap'ed file outside the driver thread, as Stephen suggested, in
order to avoid blocking it.
The currrent implementation is more/less the same as  AS is doing in
the latest code. Not that it is bad, I mean. It just can/should be  
improved.

Also, the mmaping should create a temp file, extend it and then
mmap. This way you avoid consuming the system swap. I do not know
if this is/could-be an issue later on.

Cheers
Zoran

Re: [naviserver-devel] Last commit

[Zoran V. <zv...@ar...>]

Am 12.07.2005 um 18:15 schrieb Vlad Seryakov:

> We do a lot of huge uploads and i figured while i have spare time  
> for this and need i will try. I did exactly as you defined in your  
> emails.
> Not protable though but i think usable to see how it will perform.
> i am going to install it on one of my server which does uploads and  
> watch for stability and memory usage.
>

Great! Afterwards we can improve it by defining a thread-pool of  
lightweight
threads (w/o Tcl interp) and leave them fiddling with sinking the  
socket until
all is read from remote. This way connection threads will be relieved  
from
too much blocking...

Zoran

Re: [naviserver-devel] Last commit

[Vlad S. <vl...@cr...>]

We do a lot of huge uploads and i figured while i have spare time for 
this and need i will try. I did exactly as you defined in your emails.
Not protable though but i think usable to see how it will perform.
i am going to install it on one of my server which does uploads and 
watch for stability and memory usage.

Zoran Vasiljevic wrote:
> 
> Am 12.07.2005 um 18:01 schrieb Vlad Seryakov:
> 
>>
>> I added spooling into temp for uploads.
>>
> 
> Oohohoho, I wanted to do this ;-)
> I'm curious how you did it...
> 
> Zoran
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by the 'Do More With Dual!' webinar 
> happening
> July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
> core and dual graphics technology at this free one hour event hosted by 
> HP, AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

Re: [naviserver-devel] Last commit

[Zoran V. <zv...@ar...>]

Am 12.07.2005 um 18:01 schrieb Vlad Seryakov:

>
> I added spooling into temp for uploads.
>

Oohohoho, I wanted to do this ;-)
I'm curious how you did it...

Zoran

[naviserver-devel] Last commit

[Vlad S. <vl...@cr...>]

Last commit notification failed at SF:

smtplib.SMTPSenderRefused: (452, 'Space shortage, please try later', 
'ser...@pr...')

I added spooling into temp for uploads.

-- 
Vlad Seryakov
571 262-8608 office
vl...@cr...
http://www.crystalballinc.com/vlad/

[naviserver-devel] Commits, ns_register_adp

[Bernd E. <eid...@we...>]

Hi Stephen,

I just read your commits. Can you please explain the effects of your work to 
me for this:
>Fix the confusion around the default adp
>request handler and ns_register_adp and add the missing proc
>callback info.  ns_info requestprocs now gets things
>right. 

and that:
>ns_register_adp will also now register the default adp 
>request handler if you don't specify an optional file to map.

What does this exactly mean?

Thanks,
Bernd.

Re: [naviserver-devel] Wiki backup

[Stephen D. <sd...@gm...>]

mysqlhotcopy doesn't work as the db is on a remote host.  The -x flag
to mysqldump also doesn't work, no permission...

I'm grabbing a dump file once per day.



On 7/8/05, Bernd Eidenschink <eid...@we...> wrote:
> Hi.
>=20
> just a thought, Stephen:
> Is it possible for you as an Admin to start the Database Update
> from another server via ssh? If so, you could backup and delete
> the database dump in one step and run all that via a cronjob
> from another server?!
> If they have 'mysqlhotcopy', this is what I use on one of our
> servers:
> /usr/bin/mysqlhotcopy $database -u <USERNAME> \
>    -h localhost --password=3D<PW> --addtodest <TARGETDIR>
> or one file via dump
> mysqldump -u <USERNAME> -p<PW> -x --all-databases >dump.sql
> (or --databases db1 db2 ...)
>=20
> 'mysqlhotcopy' would probably run faster and produce smaller
> files (if this is an issue), but 'files', that would have to
> be tarred.
>=20
> As long as we don't upload any files to the Wiki, the database
> backup is all we need I guess.
>=20
> Bernd.
>=20
>=20
> -------------------------------------------------------
> This SF.Net email is sponsored by the 'Do More With Dual!' webinar happen=
ing
> July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
> core and dual graphics technology at this free one hour event hosted by H=
P,
> AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>

[naviserver-devel] Wiki backup

[Bernd E. <eid...@we...>]

Hi.

just a thought, Stephen:
Is it possible for you as an Admin to start the Database Update
from another server via ssh? If so, you could backup and delete
the database dump in one step and run all that via a cronjob
from another server?!
If they have 'mysqlhotcopy', this is what I use on one of our
servers:
/usr/bin/mysqlhotcopy $database -u <USERNAME> \
   -h localhost --password=<PW> --addtodest <TARGETDIR>
or one file via dump
mysqldump -u <USERNAME> -p<PW> -x --all-databases >dump.sql
(or --databases db1 db2 ...)

'mysqlhotcopy' would probably run faster and produce smaller
files (if this is an issue), but 'files', that would have to
be tarred.

As long as we don't upload any files to the Wiki, the database
backup is all we need I guess.

Bernd.

[naviserver-devel] NaviServer 4.99.0 released

[Stephen D. <sd...@gm...>]

4.99.0 is released to the Source Forge file download system.  Bernd
added a link to the wiki.  Seems to have gone smoothly.

I didn't tell Freshmeat.  It's a bit of a stealth release...

Re: [naviserver-devel] Release/Docs

[Bernd E. <eid...@we...>]

> OTOH, have you ever looked at doctools?

Not too much, but as I read here:
http://wiki.tcl.tk/3054

the conversions should be easy:
dtp doc html  your_manpage > your_manpage.html
dtp doc nroff your_manpage > your_manpage.n
dtp doc text  your_manpage > your_manpage.txt

And seems there's also "latex" (and therefor PDF) and "wiki":
http://aspn.activestate.com/ASPN/docs/ActiveTcl/tcllib/doctools/doctools.html

So, if we use doctools, we only would need
a) An autogenerated step to generate html,nroff (and wiki)
b) (Auto)generate an index.html file to all generated single html pages (easy)

I don't know the output of the doctool->wiki conversion, if it creates 
[Wiki-Links] automagically so that we have relations for free. But that does 
not matter. We know our commands so we can regsub and create those relations.

If we start with Wiki, we can create a nice Documentation with easy 
relations/links between the commands and documents. But the
wiki->nroff step does not exist, does it? There are these small little 
nroff-gadgets like ".sp" ("Skip one line vertically") or markup for italics 
for function parameters. We can only produce really nice nroff/groff, if we 
have a really good Wiki template and always use it correctly. At least this 
is my impression.

BTW: How do we fetch our documentation out of Wiki for autogeneration of 
documentation? Maybe we take a database dump and one of us accesses it local? 
Hm...not good. Or spidering it... No.

So, somehow, seems the doctools approach would be easier right now.
And no one will complain that documentation is not on Wiki, if documentation 
is available! :-)

Bernd.

Re: [Re: [naviserver-devel] Range requests]

[Zoran V. <zv...@ar...>]

Am 05.07.2005 um 09:41 schrieb Stephen Deasey:

>
> Does anyone have a small bugfix they need to add?  I'll add the change
> for varargs macros so the Windows compile works.  Anything else?


Not at the moment. I'm happy with what we have now.
Zoran

Re: [naviserver-devel] Release/Docs

[Zoran V. <zv...@ar...>]

Am 05.07.2005 um 09:25 schrieb Stephen Deasey:

> Maybe we should pick one C function and one Tcl command to document
> fully on the wiki, write some wiki macros to make it easy, then try
> converting it into whatever?
>

Eh... It seems that this task is a real PITA! I think somebody
could make a fortune by inventing easy to use collaborative doc
system able to generate just a handful of formats: pdf, nroff, html.

If I understand this correctly, writing wiki pages is also a task
you have to learn (wiki markup). This is the same as with doctools.
Difference: in wiki everybody can do this from the browser, whereas
in doctools you'd have to do it off-line and in a text editor.

Apart that it is appealing to be able to write it on-line and it
seems trivial for bunch of people to do it "at the same time"
I see no other pros for Wiki. Cons are important: no simple interface
to cvs (docs are vital part of the server), lack of usable converters...
It seems like a lot of hacking for me and a great time eater.

OTOH, doctools are already supporting (almost) everything we need.
The cons are: you must edit sources off-line.

Just to be clear: I'm not trying to push doctools vs. wiki.
I'm just not convinced that wiki will save us any time and
trouble. I have a notion that doctools are less problematic
and would bring us to the goal (of having decent html/nroff
docs distributed with the server) faster.

I can imagine the following:
We divide and conquer...

Since I have more experience with doctools than wiki, I will take
a C-API and a Tcl-API call and write source of the doc in doctools.
Then I will convert this to nroff, html and wiki format by means of
the "dtp" processor from the tcllib.

You or Bernd can take the Wiki part and examine what would it mean
to write a converter that would emit something like people are
accustomed to when hitting "man fcopy" or "man lindex" or "man  
Tcl_NewObj".

Then we can share experience (and source-files).
How does this sound?

Re: [Re: [naviserver-devel] Range requests]

[Stephen D. <sd...@gm...>]

On 7/4/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 03.07.2005 um 23:04 schrieb vl...@cr...:
>=20
> > It needs improvements i agree.
> >
> > i am on vaca! tion this week so you all decide without waiting for me.
>=20
> I'd suggest #ifdef USE_BYTE_RANGES undefined by default. You can
> temporarily tweak the Makefile to define it.
>=20
> OTOH, what Stephen is suggesting makes sense. Stripout the range code,
> make a release, add it back again. This way you'd be on the CVS-head as
> you were before. Only the tagged release would have no ranges.


Yes, it's this last one I'd like to do -- go back in time a week or
so, make a release, then fast forward again.  The only difference
should be a file to download on sourceforge.

Does anyone have a small bugfix they need to add?  I'll add the change
for varargs macros so the Windows compile works.  Anything else?

Re: [naviserver-devel] Release/Docs

[Stephen D. <sd...@gm...>]

On 7/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 05.07.2005 um 08:43 schrieb Stephen Deasey:
>=20
> > On 7/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
> >
> >>
> >> Am 05.07.2005 um 02:36 schrieb Stephen Deasey:
> >>
> >>
> >>> http://naviserver.sf.net/wiki/index.php/User:Sdeasey
> >>>
> >>> Neato: templates!  The wiki =3D=3D nroff  :-)
> >>>
> >>
> >> Excuse my ignorance, but: where is nroff?
> >>
> >
> >
> > I was exagerating, sorry.  Templates... with args... macros...
> > nroff...
> >
> >
> > Still, where man.macros (oh, my, God) has:
> >
> >     .OP cmdName dbName dbClass
> >
> > The wiki would have Template:OP, called as:
> >
> >     {{OP:cmdName=3DX|dbName=3DY|dbClass=3DZ}}
> >
> > Which could expand to whatever you like, including HTML comments to
> > aid converting back to nroff etc...
>=20
>=20
> Does this mean that converting from wiki format to nroff
> would be relatively trivial? IOW, I can invent just about
> any Tempate:OP were OP is whatever nroff will support, like
> .CE, .CS, .PP, .SH ?
>=20
> I do not know wiki enough to be able to make any decision,
> but if there is a way of writing it all in wiki and getting
> it converted to decent nroff (and html) without much fuss,
> then I'm all for it.
>=20
> OTOH, have you ever looked at doctools?
>=20
> Zoran


I really don't know what the best thing to do is.  I can see plusses
and minuses either way.  I personally I don't have a lot of use for
paper docs, but I know it's important to some people.

You can already get an XML page dump:

  http://naviserver.sourceforge.net/wiki/index.php/Special:Export/User:Sdea=
sey

Kind of dissapointing, just a big blob of wiki-source.  However, the
macros are preserved.  So the problem basically becomes, can you write
a program which spiders the website and converts {{X:1|2|3}} into .X 1
2 3.

Maybe we should pick one C function and one Tcl command to document
fully on the wiki, write some wiki macros to make it easy, then try
converting it into whatever?

Re: [naviserver-devel] Release/Docs

[Zoran V. <zv...@ar...>]

Am 05.07.2005 um 08:43 schrieb Stephen Deasey:

> On 7/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
>
>>
>> Am 05.07.2005 um 02:36 schrieb Stephen Deasey:
>>
>>
>>> http://naviserver.sf.net/wiki/index.php/User:Sdeasey
>>>
>>> Neato: templates!  The wiki == nroff  :-)
>>>
>>
>> Excuse my ignorance, but: where is nroff?
>>
>
>
> I was exagerating, sorry.  Templates... with args... macros...  
> nroff...
>
>
> Still, where man.macros (oh, my, God) has:
>
>     .OP cmdName dbName dbClass
>
> The wiki would have Template:OP, called as:
>
>     {{OP:cmdName=X|dbName=Y|dbClass=Z}}
>
> Which could expand to whatever you like, including HTML comments to
> aid converting back to nroff etc...


Does this mean that converting from wiki format to nroff
would be relatively trivial? IOW, I can invent just about
any Tempate:OP were OP is whatever nroff will support, like
.CE, .CS, .PP, .SH ?

I do not know wiki enough to be able to make any decision,
but if there is a way of writing it all in wiki and getting
it converted to decent nroff (and html) without much fuss,
then I'm all for it.

OTOH, have you ever looked at doctools?

Zoran

Re: [naviserver-devel] Release/Docs

[Stephen D. <sd...@gm...>]

On 7/5/05, Zoran Vasiljevic <zv...@ar...> wrote:
>=20
> Am 05.07.2005 um 02:36 schrieb Stephen Deasey:
>=20
> > http://naviserver.sf.net/wiki/index.php/User:Sdeasey
> >
> > Neato: templates!  The wiki =3D=3D nroff  :-)
>=20
> Excuse my ignorance, but: where is nroff?


I was exagerating, sorry.  Templates... with args... macros... nroff...


Still, where man.macros (oh, my, God) has:

    .OP cmdName dbName dbClass

The wiki would have Template:OP, called as:

    {{OP:cmdName=3DX|dbName=3DY|dbClass=3DZ}}

Which could expand to whatever you like, including HTML comments to
aid converting back to nroff etc...

Re: [naviserver-devel] Release/Docs

[Zoran V. <zv...@ar...>]

Am 05.07.2005 um 02:36 schrieb Stephen Deasey:

> http://naviserver.sf.net/wiki/index.php/User:Sdeasey
>
> Neato: templates!  The wiki == nroff  :-)

Excuse my ignorance, but: where is nroff?

Re: [naviserver-devel] Release/Docs

[Stephen D. <sd...@gm...>]

http://naviserver.sf.net/wiki/index.php/User:Sdeasey

Neato: templates!  The wiki =3D=3D nroff  :-)

Categories look useful too.  Unfortunately all page names have their
initial letter bashed to upper case, and underscores are converted to
spaces...  :-(

Re: [naviserver-devel] Release/Docs

[Stephen D. <sd...@gm...>]

Don't know if this is useful:


    http://hula-project.org/Wiki_Conversion

    Our idea is that there be a script on the wiki server that autogenerate=
s
    PDFs of the admin guide, the user guide, and any other large-scale
    documentation, suitable for printing and binding and admiring and
    cherishing forever and ever.  -- Nat Friedman

Re: [naviserver-devel] Release/Docs

[Zoran V. <zv...@ar...>]

Am 04.07.2005 um 16:45 schrieb Bernd Eidenschink:

>> This way or another, it seems that if we'd have a wiki->html and
>> wiki->nroff converters at hand, we could trash doctools... But I
>> do not know if any of those already exists. OK, the wiki->html
>> should already be there, otherwise wiki would not work ;-)
>> But, what about wiki->nroff ?
>>
>
> Hm, the existing nroff docs could be transformed like here:
> http://naviserver.sourceforge.net/wiki/index.php/ 
> Template:Documentation_template
> I used the script that is found under the link "hack" at the end.
> (It's a hack. It's a hack.)
>
> The way back from wiki->nroff could be, depending on the nroff  
> quality, more
> easy, if we use an easy and fresh Wiki document (and make use of  
> standard
> HTML-Comments for better parsing).
>
> But it always would be unorthodox. :-)
>

I see this really pragmatic: *I* would not like to invent the wheel.
If there is some good quality wiki->nroff, wiki->html, wiki->(whatever)
this would be the way to go: write all in Wiki and get it converted
to other stuff. This is very appealing.
If not, but somebody (you?) could write working converters for at least
html and nroff conversion, I'd be happy to use them and would also  
opt to
write source docs in wiki. Ideal would be a tclsh app I could hit like

    wiki2html source.wiki   (would produce source.html in the current  
dir)
    wiki2nroff source.wiki  (would produce source.nroff in the  
current dir)

If not, I'd use doctools for already given reasons. It can't be that
bad considering that all of tcllib modules are documented with doctools
(with some notable deficiencies in documenting C-code as I already
mentioned).

Zoran

Re: [naviserver-devel] Release/Docs

[Bernd E. <eid...@we...>]

> This way or another, it seems that if we'd have a wiki->html and
> wiki->nroff converters at hand, we could trash doctools... But I
> do not know if any of those already exists. OK, the wiki->html
> should already be there, otherwise wiki would not work ;-)
> But, what about wiki->nroff ?

Hm, the existing nroff docs could be transformed like here:
http://naviserver.sourceforge.net/wiki/index.php/Template:Documentation_template
I used the script that is found under the link "hack" at the end.
(It's a hack. It's a hack.) 

The way back from wiki->nroff could be, depending on the nroff quality, more 
easy, if we use an easy and fresh Wiki document (and make use of standard 
HTML-Comments for better parsing).

But it always would be unorthodox. :-)