#48 Making .tex files

[Jonathan Dushoff]

The latex rule mechanisms make sure all .tex files are up to date before trying to make any real targets. Sometimes, this might be good, because the make rules might change and we might need to make something we didn't think we needed to (although this is not absolutely clear). Other times, it is bad, because when we can't make .tex files, we are unable to view files that have nothing to do with them. An extreme case is source files, which cannot be viewed in a directory with broken rules to make .tex files.

[Lee Worden]

we have $(WW_THIS_PROJECT_SOURCE_FILES) now (https://sourceforge.net/p/workingwiki/feature-requests/54), so we should use that to distinguish .tex source files from .tex files that are intermediate targets. Or maybe rewrite the makefile some other way so that it doesn't need to.

[Lee Worden]

working on this. Considering renaming the $(tex_d_files) variable to $(WW_TEX_D_FILES). That would break any existing projects that modify that variable. Is that an issue?

[Lee Worden]

Dushoff says no so I think we're safe.

• assigned_to: Lee Worden
• milestone: lalashan-yushan_site-specific --> refactored_workingwiki_(with_projectengine)

[Lee Worden]

OK, it's changed in $(RESOURCES)/makefile-before and -after. WW now only makes .d files for .tex files that are source files (and are in the main project directory). Projects can change that behavior by changing $(WW_TEX_D_FILES).

I don't know if that actually addresses this bug report, though.

[Lee Worden]

See also https://sourceforge.net/p/workingwiki/bugs/133/, which is very similar to this ticket.

[Lee Worden]

From JD, reposted from https://sourceforge.net/p/workingwiki/bugs/86 to here by me:

I have a suggested work-around now, which seems to work very nicely for
making pdfs from auto-generated .tex. I would like to test a bit more and
then suggest we add it to the makefiles (or the site makefiles). I'm pretty
surprised, though, that it didn't solve the related LatexML problem. Before
I proceed; any idea why (or what the LatexML error on this page means)?

http://lalashan.mcmaster.ca/theobio/projects/index.php/Pseudo_auto_tex

[Lee Worden]

In response to that: yes, it's because the .post.d wasn't being included. I've proposed a fix by editing that page.

[Anonymous]

I'm confused, because my indirect way of making manuscript.post.d fixed the problem with manuscript.post.pdf, so it presumably created manuscript.post.d.

In any case, while I don't understand why yours works and mine doesn't, I do agree that, given the possible jump-starting problems, yours is just as good, and therefore we can just do it that way.

JD

[Lee Worden]

In regard to the main problem, which is that you can't make anything when you can't make one of the .d files, I wonder if there's a way to put some kind of failsafe into the process of making .d files itself. Something like:

ifeq ($(origin RECURSIVE_DOT_D_MAKE),undefined)
%.d : /proc/uptime
    - $(make) RECURSIVE_DOT_D_MAKE=1 $@
else
%.d : %.tex
    $(DO_THINGS)
endif

so that when the .d can't be made, because the prerequisite can't be made, the inner make fails but the outer make moves along happily.

If that isn't enough because make needs the .d to exist, we could use

    - $(make) RECURSIVE_DOT_D_MAKE=1 $@ || touch $@

or something.

[Jonathan Dushoff]

This is related to (but slicker than) my suggestion of a two-step make long ago.

[Lee Worden]

I don't know which is slicker, but yes it does seem related, good point.

[Lee Worden]

Still, though, I think it would probably be best to adopt the "advanced" strategy from http://mad-scientist.net/make/autodep.html: have the .d files made as a side effect of compiling the .tex file, and don't provide a rule to make them directly. Then everything will behave when the .tex is broken, except for actually trying to use the .tex.

[Lee Worden]

See also https://sourceforge.net/p/workingwiki/bugs/261: Nested LaTeX dependencies

[Lee Worden]

I should really do it the way http://mad-scientist.net/make/autodep.html says, but it's confusing!

[Lee Worden]

Copied over from #261:

Hmm, if we use chained rules like

(in manuscript.d)

manuscript.latex.pdf : content.tex

(in content.d)

content.tex : abstract.tex ; touch $@

together with

(in $(RESOURCES)/makefile-after)

%.d : %.tex

and abstract.tex is to be made from a .dmu file that doesn't work, then what happens? Make won't be able to make abstract.d and content.d, so the whole project will break and be unable to make anything. This is the problem of

We can get make to run by not using content.d and abstract.d (by not including them in $(WW_TEX_D_FILES)), but then the pdf file won't depend on the .dmu. But if we do include them, breaking the DMU code breaks the whole project.

So this is an argument for eliminating the %.d : %.tex, and if we switch to not using chaining:

manuscript.latex.pdf : content.tex abstract.tex

we'll avoid the can't make problem, but we'll still have it if we try to make manuscript.pdf from manuscript.dmu. This is not actually an argument against or for chaining, but against having make update the .d files automatically. So it belongs in bug #48, not here. I'll copy it over there.

[Lee Worden]

Upon reading the mad scientist link more carefully, understanding it better, and making a working prototype latex version of it in a test directory, I'm pretty convinced it's the way to go. The design pattern looks like this:

• One .d file per target, not one per .tex source file. More precisely, from X.tex and some number of other things you can make X.pdf, X.latex.pdf, X.pdflatex.pdf, X.xelatex.pdf, X.latexml.html, X.latexml.xhtml, etc., and for that constellation of targets we'll have one dependency file X.d.
• That file will flatten the list of all dependencies, even though they may be invoked in a long recursive chain of \input, \bibliography, and \includegraphics commands, into a few simple lists of prerequisites. When any of them changes, it's time to run latex or latexml on the main file X.tex, and this straightforwardly encodes that fact in a simple and effective way.
• The .d file will be made as a byproduct of running latex, latexml, etc., not in a separate step as we have been doing. This eliminates the sorrows of failure to make a .d file, and will only leave us with the sorrows of failure to make a .pdf or .html target, which are unavoidable.
• .d files will be included using "-include", so make will only read in .d files that exist, and will not complain about any that don't. Because they will no longer be declared as targets of rules, it won't try to update them before reading them in.
• Thus we will no longer have a .d file for each .tex file, and we won't try to include one for each .tex file, or one for each .tex source file, or whatever. We only have one for each target, which we don't really have a way to enumerate. I think we should just have it include all the .d files that exist, without trying to predict which ones to look for. Doing that means they should probably be called something more specific, so that other makefile authors can define other kinds of dependency files for non-latex code without us annoying them by including their makefiles and taking away their power of choice. So maybe they should be called %.tex-dependencies or something.
• At that, as I think Jonathan suggested, maybe we should tuck them away in a .workingwiki/ directory or something.

I welcome comments and criticisms before I make it so.

[Lee Worden]

I mean, before I commit to making it so.

[Lee Worden]

Upon reading the mad scientist link more carefully, understanding it better, and making a working prototype latex version of it in a test directory, I'm pretty convinced it's the way to go. The design pattern looks like this:

One .d file per target, not one per .tex source file. More precisely, from X.tex and some number of other things you can make X.pdf, X.latex.pdf, X.pdflatex.pdf, X.xelatex.pdf, X.latexml.html, X.latexml.xhtml, etc., and for that constellation of targets we'll have one dependency file X.d.
That file will flatten the list of all dependencies, even though they may be invoked in a long recursive chain of \input, \bibliography, and \includegraphics commands, into a few simple lists of prerequisites. When any of them changes, it's time to run latex or latexml on the main file X.tex, and this straightforwardly encodes that fact in a simple and effective way.
The .d file will be made as a byproduct of running latex, latexml, etc., not in a separate step as we have been doing. This eliminates the sorrows of failure to make a .d file, and will only leave us with the sorrows of failure to make a .pdf or .html target, which are unavoidable.
.d files will be included using "-include", so make will only read in .d files that exist, and will not complain about any that don't. Because they will no longer be declared as targets of rules, it won't try to update them before reading them in.
Thus we will no longer have a .d file for each .tex file, and we won't try to include one for each .tex file, or one for each .tex source file, or whatever. We only have one for each target, which we don't really have a way to enumerate. I think we should just have it include all the .d files that exist, without trying to predict which ones to look for. Doing that means they should probably be called something more specific, so that other makefile authors can define other kinds of dependency files for non-latex code without us annoying them by including their makefiles and taking away their power of choice. So maybe they should be called %.tex-dependencies or something.
At that, as I think Jonathan suggested, maybe we should tuck them away in a .workingwiki/ directory or something.

I welcome comments and criticisms before I make it so.

JD - do you want time to review this, or shall I go ahead?

[Jonathan Dushoff]

One big concern would be: what happens if a .tex file goes away? Maybe use existing .tex.d files that are associated with existing .tex files?

Otherwise it sounds very good, modulo the question of whether there's an easy way to make the recursively flattened file.

BTW, I suggest calling them base.tex.d, rather than base.tex or base.ww_tex_dependency_rule. If you know what I mean.

[Lee Worden]

JD and I have resolved the concern about removed .tex files on the phone: ms.tex.d only tells how to make ms.pdf and related products, so having it remain after ms.tex is removed seems okay. I'll go ahead and make the changes.

[Lee Worden]

ps. the flattened list of dependencies turned out to be easy to do - you'll see the while loop in makefile-before once I've checked it in. I wrote and tested it in the Toronto airport when I was early for my flight back to SF.

[Lee Worden]

The new make rules need more complete testing before committing them. There is a ticket somewhere on here with a whole list of good tests for making sure .tex files compile right. I need to find those and check all of them. Right now I just noticed that it chokes when there's a .tex and .bib but the .tex.d hasn't been created yet - it knows how to compile the .tex but not how to make the .bbl.

[Lee Worden]

I found the tests - they're on [#86]. Embarking on testing now.

[Lee Worden]

OK

• passing test 1 (after some fixing)
• passing test 2
• passing test 3
• passing test 4
• passing test 5
• passing test 6 (latex only)
• passing test 7
• passing test 8 (which used to fail)
• therefore test 9 is not needed (you don't have to do WW_TEX_D_FILES += ms.d)
• passing test 10 without any added WW_TEX_D_FILES code