#312 Ghost .d dependency

[Jonathan Dushoff]

Yesterday, Chyun fixed a broken file link in a latex-markup document in a very complicated project. The project continued to demand the non-existent file. Removing .workingwiki/ did not fix the problem. I worked on it for a long time, and got confused, but what apparently fixed the problem was removing the .log file associated with the .d file, followed by removing .workingwiki/

lalashan.mcmaster.ca/theobio/circumcision/index.php?title=MC_risk_behaviors_Uganda_manuscript

[Jonathan Dushoff]

It may be worth mentioning that the project is apparently missing a .bib file that it should need to compile the document. The document is compiling anyway. All very confusing.

[Jonathan Dushoff]

I'm wondering if deleting manuscript.log at some clever time would have solved this problem, since it seems like a long-outdated manuscript.log was the root cause.

[Jonathan Dushoff]

Meaning, whether there's an appropriate time in the make rules to delete manuscript.log (maybe after using it to make the .d, or after calling the .d in an attempt to prepare to make the .tex file)

[Jonathan Dushoff]

At the time of problem, project also had the deprecated make directive

LATEX=pdflatex.

Now changed to

DEFAULT_LATEX=pdflatex.

[Jonathan Dushoff]

I should also mention that the reason I didn't simply clear the project directory is that I had latex links to big stats projects which would have taken a very long time to rebuild. I don't know if we want a ticket to discuss whether this sort of thing can be dealt with.

In retrospect, to be selfish and solve the problem instead of reporting it, I should have just made my own clean rule

clean:
    /bin/rm -f *.* .workingwiki/*.*

[Lee Worden]

This is weird and confusing to me. Do you have any more detailed info
about what happens with the .log file relative to the .bib and latex?

By log file you mean manuscript.log, right, not any .make.log file? The
only thing that WW's makefile code does with latex's .log files is when
it greps them to see if there's a message saying to rerun latex to get
the citations right, and to find out whether it created a .dvi or .xdv
file rather than .pdf. They are not used in making .d files.

http://lalashan.mcmaster.ca/theobio/projects/index.php/Special:GetProjectFile?filename=makefile-before-latex

http://lalashan.mcmaster.ca/theobio/projects/index.php/Special:GetProjectFile?filename=makefile-after-latex

There may be something weird going on with the .log file that I can't
imagine, but it also seems like the problem might have been fixed by
something else that happened non-transparently around the same time as
you deleted the .log, since I can't see how a .log file would affect
what the makefiles try to make, or what latex tries to do.

[Jonathan Dushoff]

Yes, it was manuscript.log. I found it using grep: it was the only file in the project (aside from make logs) that still had the bad file name.

All I can do is continue to observe and report, I guess.

JD

[Jonathan Dushoff]

Here's an MRE: http://lalashan.mcmaster.ca/theobio/projects/index.php/Dependency_sandbox

[Lee Worden]

I see, it would update the .d file at the same time that it reruns pdflatex. But it doesn't get that far, because it can't make the prerequisites, which include fine.Rout.png, according to the current contents of line.d.

[Lee Worden]

But removing .workingwiki/ should take care of that, though once that's gone you might have to remove line.pdf or sync line.tex to force it to remake, since it will think that line.tex is the only prerequisite.

[Lee Worden]

Is there a way to list the image files as prerequisites, but allow it to go ahead and run latex if they can't be made?

[Lee Worden]

On 07/18/2013 11:25 AM, XXXXX wrote:

Hi Lee,

Here's a problem I just encountered that Jonathan helped me solve.

I have a file "file.tex" which initially included another file "file2.tex" via the latex command "\input{file2.tex}". I had the pdf displayed by calling the project file "file.pdf" (I did not write any explicit makefile)

Then I decided not to include file2.tex, and deleted "\input{file2.tex}". Saved it. But the wiki would keep on building file.pdf WITH file2.tex. Synchronizing it did nothing. Only deleting the file .workingwiki as directed by JD, solved the problem.

Let me know if you need more details.

Hi XXXXX, thank you.

Yes, I guess this is bug #312, which Jonathan recently discovered: https://sourceforge.net/p/workingwiki/bugs/312/

Possibly he advised you to write to me so that I'll know that it's affecting you as well.

I haven't figured out what to do about it. It's because the makefiles do latex compilation in the sequence:

• build all the prerequisites of the pdf target
• run latex and bibtex (and latex, and latex...) to create the pdf output
• update the list of prerequisites (which is stored in the file .workingwiki/file.d)

They used to update the prerequisites first, but it caused so many problems that I changed the sequence. This problem is a consequence of the new sequence.

LW

[Lee Worden]

Here's one possible solution: using the wildcard function to find out which files currently exist. Something like

file.pdf : $(wildcard figure1.png file2.tex)
    $(MAKE) figure1.png file2.tex
    $(run-latex)

[Lee Worden]

Some challenges with that technique:

Currently the commands for %.pdf : %.tex are in a central makefile, thus they're generic. This is good because it provides commands to be used the first time, when the .d file doesn't exist yet. However, I'm not sure how I would add this $(MAKE) figure1.png file2.tex to the commands, because the list of prerequisites is different for each .tex file and there may be multiple.

Also note the command should probably be -$(MAKE) -k figure1.png file2.tex, so that it will try to make all of them even if some fail, and so that the making will continue in case of failure. (Actually it only needs to try to make the ones that don't exist, since the others will have been made as prerequisites, but that's harder.)

Note as well, that while this design protects against failure due to files that don't exist, it doesn't help if the .d file refers to a file that is no longer needed, but exists and can't be successfully updated.

More to the point, what we really need is to remake if any of the prerequisites is new or has new prerequisites (whether the files exist or not), but go ahead and run the commands if any of them can't be made. I don't think this wildcard thing really answers that.

[Lee Worden]

JD and I discussed this on the phone. I think the solution I like now is to rebuild the .d file before checking the prerequisites, by using recursion - something like

ifeq ($(IN_RECURSION),)
%.pdf : /proc/uptime
    $(rebuild-d-file $*.tex)
    $(MAKE) IN_RECURSION=1 $@
else
%.pdf : %.tex
    $(run-latex)
endif

in the main makefile, and

ifneq ($(IN_RECURSION),)
file.pdf : figure1.png file2.pdf
endif

in the .d file. This way when the .tex is changed, the .d file will get updated before it gets used, and the outdated prerequisites won't be used.

[Lee Worden]

Hmm, I am trying to write an MRE in the form of a test case on http://lalashan.mcmaster.ca/theobio/projects/index.php/WorkingWiki/Nested_LaTeX_Tests, but it's not working! I mean, it's not failing when it should. I have a file with an \include of a nonexistent file, and I replace it with one that doesn't have that; the test is

test-14 : /proc/uptime
    make clean
    cp test-14-with.tex ms.tex
    -make ms.pdf
    -make ms.latexml.html
    cat .workingwiki/ms.tex.d
    cp test-14-without.tex ms.tex
    make ms.pdf
    make ms.latexml.html
    echo 'passed' > $@

The .d file after compiling the "with" file is

ms.latex.pdf : ms.tex whoami.tex 
ms.pdflatex.pdf ms.xelatex.pdf : ms.tex whoami.tex 
ms.bbl :: 
ms.latexml.xhtml ms.latexml.html5 ms.latexml.html : ms.tex whoami.tex 
ms.latexml.xml : ms.tex whoami.tex 
ms.tex whoami.tex  :

and after compiling the "without" file is

ms.latex.pdf : ms.tex 
ms.pdflatex.pdf ms.xelatex.pdf : ms.tex 
ms.bbl :: 
ms.latexml.xhtml ms.latexml.html5 ms.latexml.html : ms.tex 
ms.latexml.xml : ms.tex 
ms.tex  :

So why doesn't it fail at the second "make ms.pdf"?

[Lee Worden]

Running make with debug output reveals

     No implicit rule found for `whoami.tex'.
     Finished prerequisites of target file `whoami.tex'.
    Must remake target `whoami.tex'.
    Successfully remade target file `whoami.tex'.

Wherefore is this "Successfully remade"??

[Lee Worden]

The plot thickens! By my tests, the makefiles perform as intended when the .tex file used to refer to a file that doesn't exist and there is no rule to make it.

But they do fail when there's a reference to a file that doesn't exist, has a rule, and can't be made.

That is, I get a failure when I remove \includegraphics{nonexistent.Rout.png} from my .tex file, but not when I remove \input{nonexistent.tex} or \includegraphics{nonexistent.png}. Which means that there's something I don't understand about XXXXX's report, above.

[Lee Worden]

I'm testing out the recursive-make thing in an offline project. I'm having a version of the same problem there, where it somehow decides that, say, ms.pdf is updated when it hasn't actually done the steps and the file isn't there. WTF? I think I came across this before somewhere...