Menu
▾
▴
#1000 communicate with upstream texi2html project
Accepted
nobody
None
Maintainability
2015-11-14
2010-01-25
Anonymous
No
Originally created by: *anonymous
Originally created by: percival.music.ca@gmail.com
Our lilypond-texi2html.init file is getting ridiculous; we keep on fixing
texi2html bugs by introducing extra functions to this file. It would be
nice if somebody could communicate with the texi2html people -- send our
bugfixes and features upstream, prod them into releasing 1.84 or whatever,
and then update our file to work with the later version and remove anything
that we don't need in there.
Originally posted by: v.villenave
[OT but sooo tempting]
And this is our 1000th tracker page! Yay! :)
Originally posted by: markpole...@gmail.com
Is it just a matter of asking the texi2html guys to look
over the lilypond-texi2html.init file or does one of us
need to get involved and explain some of texi2html bugs
that the file fixes?
Also, it looks like texi2html will soon be part of texinfo:
http://lists.nongnu.org/archive/html/texi2html/2010-03/msg00003.html
And texi2html bugs should now be sent to:
bug-texinfo@gnu.org
(start subject line with "[texi2html]: ")
- Mark
Originally posted by: percival.music.ca@gmail.com
One of us definitely needs to be involved. Throwing the entire init file at them
without explanations would be cruel and unusual punishment.
For example, look at texinfo_file_name($). Figure out what we're doing differently
than a normal texinfo_filename($)... I think it's the lc_last() call. Then send a
feature request to the texinfo list asking to have the html filename be lower-cased
(but not the intermediate directories; just the filename). Once that's in the main
texi2html... and once they have a stable release... then we can remove that function
from our file. Oh, and double-check that I'm correct that lc_last() is the only
difference.
Some parts of our file probably shouldn't be merged, like the translations and xref
stuff.
No, this isn't an easy or quick task, but (like all build system issues) it'll only
get worse as time progresses. And at least this could be done by a Frog -- you don't
need to be a lilypond expert, and certainly don't need to know anything about scheme
or C. Just a bit of basic perl knowledge, the ability to compile the docs (to see if
the modified texi2html + our changed init file produce the same html files), and
communicating with the texinfo people.
Labels: Maintainability
Originally posted by: markpole...@gmail.com
I'd prefer this to be done sooner than later, but I'm not
the one for the job. It looks like the final texi2html
release is imminent* and we don't want to miss it. Would
anyone like to volunteer?
*texi2html is now part of texinfo (in CVS at least) and once
the final texi2html version is released, the new version of
texinfo should follow:
http://lists.nongnu.org/archive/html/texi2html/2010-03/msg00003.html
Originally posted by: percival.music.ca@gmail.com
I really, really doubt that anybody is going to volunteer.
*shrug*
Our texi2html init file is a complete mess, but it's no worse than any other part of
our build system. We have collectively decided that it's easier to live with the
mess. Cleaning up the build system would take dozens, if not hundreds, of hours.
I'm not certain that it's *bad* that we've decided not to do anything about it.
Originally posted by: percival.music.ca@gmail.com
(No comment was entered for this change.)
Labels: -Maintainability Type-Maintainability
Originally posted by: percival.music.ca@gmail.com
Since texi2html has been completely rewritten, there is nowhere to "send patches upstream" at the moment.
That said, whenever there is a new version of texi2html, it would be good to
1) migrate our docs to that system
2) send (some/most?) of our custom texi2html settings to the upstream project.
I'm not expecting any action on this for at least another 2 years. I wouldn't be surprised if it passed 5 years.
Summary: communicate with upstream texi2html project
Labels: -Priority-Low -Frog
Originally posted by: pertu...@free.fr
I was the main texi2html developer and I am also the main developer of the new implementation in texinfo. I would like to port the current lilypond init file to the new API for 2 reasons, one is allow lilypond to use the new implementation as soon as possible, the other being using lilypond as a real life case of customization to see what has to be available through the API.
So, could it be possible for you, please, to provide me with
* your current init file
* a tarball of the texi files of lilypond
Thanks in advance
Originally posted by: reinhold...@gmail.com
Hi Patrice,
Thanks for you offer, which I gladly accept (I was the main "offender" of our init script three years ago).
It's actually not that easy to send you our texi files, since most are generated by lilypond-book from .tely files (texinfo with lilypond snippets, which are turned into images by lilypond-book), and some are even auto-generated by our buildsystem (like the whole internals reference).
So, it's probably easiest for you to chec out all of lilypond, build lilypond and then the documentation (see out contributor's guide for detailled information, http://lilypond.org/doc/v2.15/Documentation/contributor/ ):
-) git clone git://git.sv.gnu.org/lilypond.git
-) ./autogen.sh; ./configure
-) make
-) make doc
The texi2html init script is Documentation/lilypond-texi2html.init, and the texinfo sources can be found in the Documentation subdir (the .texi files created by lilypond-book from the .tely files during make doc can be found in the out-www/ subdirs...).
A few words about the largest chunks of our script:
1) We print out a TOC in a <div> on the left side, which is created from the texi2html node list, not from the menu structure.
2) The snippets include several files multiple times (in different sections), so many nodes are duplicate, and -- what's worse -- we don't want any menus there (which just waste space and clutter the page). So, we get thousands of warnings (when you run "make doc") about nodes not being in a menu etc.
3) The parts about the map files (*.xref-map) are about our own splitting (we don't want every node in a separate file, but only each numbered section should start a new file) and about translated node names:
We wanted to translate node names in translated documentation (because the file names should be translated), too. Unfortunately, for xrefs between unrelated manuals, we need to know more (file name where the translated node can be found, which cannot be deduced from the node name because we only split at numbered sections). The build system then extracts all those node names and the corresponding file names and stores them in .xref-map files (in out-ww/xref-maps/ ).
For translated docs so we added @translationof (see any of the translated manuals in Documentation/{cs,de,es,fr,hu,it,ja,nl,zh}/ ), which specifies the original untranslated node name, which will be used for the xrefs.
4) We use our own navigation bars between the nodes. Their structure is defined in several arrays.
5) All those AJAX_SEARCH conditionals are about optionally adding an AJAX search fields (which does a server-side searche of all entries in the texinfo index files). See http://kainhofer.com/~lilypond/Documentation/notation/index.html for an example.
To build the docs with this search enabled, you need to set the AJAX_SEARCH env variable in the make command:
make AJAX_SEARCH=1 doc
6) We have also included translations of the texi2html strings for several more languages.
If you have further questions (that's a rhetorical question, I don't expect anyone to understand that file on his own), feel free to ask here or via private mail.
Cheers,
Reinhold
Cc: pertu...@free.fr
Originally posted by: percival.music.ca@gmail.com
actually, I was just about to suggest working on contributor.texi and/or web.texi. Neither one involves lilypond-book. Admittedly web.texi doesn't use the special table of contents stuff, but it could be good to work on that first; then once web.texi is looking fine, work on contributor.texi (which has the special table of contents but no translations), and only then tackle the other manuals.
I'm putting together a tarball of the relevant files, so stay tuned.
Originally posted by: reinhold...@gmail.com
Oh, I forgot to mention that
-) we post-process the output of texi2html with our scripts/build/www_post.py script to fix several hyperlinks and some missing translations, etc..
-) I also need to mention that we build two versions of all manuals, one without any page splitting ([manualname]-big-page.html) and one with splitting at numbered sections (genereated into the [manualname]/ subdirectory of out-www). That's one of the reasons why we need to fix the links (to the css files etc), because the files have different base dirs.
-) The final output of "make doc" is generated in out-www/offline-root/
-) The script to create the .xref-map is scripts/build/extract_texi_filenames.py
Originally posted by: percival.music.ca@gmail.com
Here's a tarball to get you started with website. Run
make web
and ignore the warnings about missing pictures. The website is functional.
make web-no-init
creates a website without our init file; this version of the website is not particularly functional.
I think this only uses about 30% of the code in lilypond-texi2html.init, but that should still be a good introduction to this stuff.
Some of the parts of our init file would be quite useful upstream (such as naming the html file after the node name, instead of creating files like index_1.html index_2.html). Stuff about the xref files probably aren't too useful... OTOH, it's possible that this could be a useful translation tool for other projects, so it might still be worth it. Kind-of depends on how many other people use texinfo, and how much time you want to spend on this. :)
I think texinfo is a great documentation language; I haven't found anything else that produces decent pdf and html. It's great to see some interest in reviving it!
Originally posted by: pertu...@free.fr
Thanks for all those informations.
This is much more complicated that what I had foreseen, and it is unrealistic to do that for the first release. And there isn't much that is already in latest texi2html or in tp, except for the node name normalization code.
I see 3 areas of work
1. Use the new gettext-like framework for translations
2. The whole @translationof/xrefs translated node names support
3. Remaining customizations
The point requiring more work, in my opinion, is 2. But it is also something that should be upstream.
Originally posted by: percival.music.ca@gmail.com
It would be great if the xrefs could be implemented upstream! Especially if other projects start using it and iron out some bugs before we get to it. :)
Anyway, that sounds like good plan. There's no great rush on our side; the existing system has a few problems, but nothing critical.
Originally posted by: pertu...@free.fr
Three questions. First, I don't really understand the interest of lower casing the file name. What is it for?
Second, why use a mapping of xrefs? Is it to avoid using the redirection files created with NODE_FILES set? Is it because nodes are not really used as targets but sections names are? Is it because of translations?
On the matter of translations, I didn't really understand what @translationof is really used for. I really can't see when the english node name is useful, since everytime a link to the translated node is used?
Originally posted by: percival.music.ca@gmail.com
First answer: I think it's just aesthetics. Most webpages use lower-case letters, so IIRC I asked that the people working on the init file produces lower-case html files. It's been a long time since this work was done so I'm not certain if I actually asked them or not, but it's the kind of thing that I would do.
I would like to keep the lower-case html names, though. I know it's easy to do in code. :)
Second answer: I'm not certain about the translation issues, but I can clarify/emphasize that (in IIRC 10 out of our 11 manuals) we create html names based on numbered/unnumbered status. i.e.
@node foo
@subsubsection foo
gets a new html page, whereas
@node bar
@unnumberedsubsubsec bar
does *not* get a new html page. And yes, we regularly use @unnumberedsubsubsec; that's the default "level" in our notation reference.
In most manual we keep unnumbered for the subsubsec level, but in 1 or 2 manuals we use unnumbered at the subsec level and have new pages on the @section level. The point of the "numbered/unnumbered" distinction is to avoid having to specify different --section= commands in the build system; instead, the splitting is done in the general numbered/unnumbered case. This produces a more complicated init file, of course. :)
I think this could be a useful feature for other projects, so I definitely hope it gets upstream.
Third answer: I'm not totally positive, but I believe that @translationof is only used in our internal "housekeeping" translation scripts -- they check that there's an appropriate @translationof for each @node in the main docs.
Originally posted by: fedel...@gmail.com
Third answer to comment 15: I think that the main purpose of @translationof is keeping the original "@node name" and so avoid the break of ref and crossref links in translated manuals. Without @translationof a translator should:
- fix immediately all the @ref links within that manual pointing to that node (easy)
- fix immediately all the crossref from other manuals to that node (unpractical)
This search in the git source tree might help:
git grep -n translationof scripts/*
BTW, if you remove @translationof from a node in a translated page, the URL name will be @node (i.e. the translated title). So another purpose of @translationof seems translating the node text but keeping the english node name in the URL (of course, the translated page will have the localized .xx.html extension).
My personal opinion is that the URL should be translated (because of search engines and user interface). If @translationof didn't have any other function, it would be useless IMHO.
Maybe we are following some GNU convention for URLs? I see that gnu.org pages have page-title-in-english.xx.html
Originally posted by: dak@gnu.org
How are the language changing links on the bottom of manual pages supposed to work if the URL name is translated?
Originally posted by: fedel...@gmail.com
Yes, good point.
I'd be curious to see how other projects using texinfo (and possibly the latest texi2any) deal with translation. Do you know any?
Texinfo manual doesn't seem to be translated in any language.
Originally posted by: gra...@percival-music.ca
Here's some recent comments I made on lilypond-devel https://lists.gnu.org/archive/html/lilypond-devel/2015-07/msg00036.html :
If somebody has basic knowledge of perl (which isn't hard to acquire) and texinfo (essentially just knowing it's a doc format with @commands), and is good at diagnosing and communicating problems, then I would cautiously estimate 20 hours for this task. 10 hours of investigation and rewriting by oneself, plus another 10 hours of creating minimal examples, sending them to the texinfo list to ask for help, then integrating the solutions into the updated thing.
This could very feasibly be done by somebody new to lilypond development.
oh, two slight snags:
1) once it was updated to the new texinfo, our build scripts will probably need some slight adjustment. That is not do-able by somebody new to lilypond, but I'd estimate 1 hour from a current lilypond developer to get that done.
2) it is possible (or even likely) that texinfo will need to be changed in order for us to do what we want. I'm certain that if somebody has a minimal example of the problem, and a compelling justification for why we want to achieve the desired output, the texinfo people will be happy to add whatever features or bugfixes
are required in texinfo. But this *would* add another delay for being able to build lilypond on a stock distribution with texinfo 5.x.
Originally posted by: gra...@percival-music.ca
You know, in the worst case, this could be done by somebody with **no** programming skill at all.
- I am willing to mentor this project.
- Your goal would be to compare the HTML output of
texi2html5 --options test.html
to our website. Make a list of things that are different.
- We pick one item, then you write a polite email to the texinfo people asking them to add this as a feature, or how to achieve it with custom hooks.
(ideally, you would have some basic programming ability and could make a first attempt at doing this yourself, but we can try skipping this step)
- Once we have an answer (either "yes, great idea! we've added that in texinfo 5.1.4" or "here's how to do it"), we move on to another item.
The most important thing here is to get some momentum. I mean, once we've isolated a specific programming task, we could ask for help from lilypond-devel. For example: "I need a perl function which takes a string like 'Installing on Windows' and returns 'installing-on-windows.html'". Once we've isolated a sub-problem to that extent, it won't be hard to get help with the actual programming.
Originally posted by: mgdd...@googlemail.com
I do have basic knowledge of perl and my knowledge of texinfo is basically what you write as requirement in #20 above.
At this time I can't build the documentation because of the problem mentioned here:
https://lists.gnu.org/archive/html/lilypond-devel/2015-03/msg00084.html
Thus if I get some tutoring to get me started I'm willing to help fixing this problem.
For the record:
I'm on Arch Linux and that comes with texinfo 5.x
Originally posted by: gra...@percival-music.ca
(reposted from correct account)
Hi mgdde42,
At the moment me and another person are working on this off-list. We have a public repo here:
https://github.com/barrykp/lilypond-texinfo
If you contact me off-list, I can put you in touch the other guy, and we can work on this together.
Originally posted by: lily...@orange.fr
I've adapted the build on a specific branch, but got stuck by uninitialized values in a test and others things I'm unable to deal with.
Please fin enclosed the patches towards liylypond-texi2html.init and the logs.
Originally posted by: pkx1...@gmail.com
(No comment was entered for this change.)
Labels: Patch-new
Cc: lily...@orange.fr