#73 Off-by-one error in display of \every... token registers

[Bruno Le Floch]

The string displayed when tracing the automatic insertion of \every... token registers is wrong. Namely,

\begingroup % Just there to minimize tracing output
  \tracingmacros=2
  \tracingonline=1
  \everyvbox{\relax} % or whatever, non-empty.
  \vbox{}
\endgroup
\end

displays \everyjob->\relax instead of \everyvbox->\relax in the tracing information. Only tracing is affected, hence the low priority. This appears to be an off-by-one error, as similar examples with other token registers also print the wrong string:

• \everypar is shown as \everymath
• \everymath is shown as \everydisplay (this is what confused me while chasing a bug in a macro)
• \everydisplay is shown as \everyhbox
• \everyhbox is shown as \everyvbox
• \everyvbox is shown as \everyjob
• ...
• \everyeof is shown as \toks0.

[Khaled Hosny]

I tried to debug this, it seems to be caused by the print_cmd_chr change in 8fda78c, but removing the +1 will break the tracing of \XeTeXinterchartoks. I don’t fully understand this change or the related code, so no idea how to fix it yet.

[Bruno Le Floch]

Hello,

Here is a possible bug fix. I have not tested yet. Debugging this is
complicated because I have to jump among tex.web, etex.ch, and
xetex.ch. Is there a xetex.web somewhere?

I suspect that omitting the following two changes in xetex.ch (i.e.,
do not put a "-1" or "+1" offset) should fix the bug and not introduce
new ones.

@x
[...irrelevant to bug...]
@d eTeX_text_offset=output_routine_loc-output_text
@y
[...irrelevant to bug...]
@d eTeX_text_offset=output_routine_loc-output_text-1 {1 more to make
space for the |inter_char_text|}
@z

and

@x
othercases print_cmd_chr(assign_toks,t-output_text+output_routine_loc)
@y
othercases print_cmd_chr(assign_toks,t-output_text+output_routine_loc+1)
@z

I suspect that these offsets by 1 were introduced (the first,
correctly, the second half-wrongly) before the introduction of
XeTeX_inter_char_loc in

@x
@d etex_toks=etex_toks_base+1 {end of \eTeX's token list parameters}
@y
@d XeTeX_inter_char_loc=every_eof_loc+1 {not really used, but serves as a flag}
@d etex_toks=XeTeX_inter_char_loc+1 {end of \eTeX's token list parameters}
@z

Regards,
Bruno

On 4/27/14, Khaled Hosny khaledhosny@users.sf.net wrote:

• Group: v0.99991 --> Future

[Bruno Le Floch]

A related bug: with \tracingassigns positive, the assignment \XeTeXinterchartoks 1 2 {foo} gives

{changing ?=?}
{into ?=?}

which I believe is a related issue. I understand some of it (currently writing it up).

[Khaled Hosny]

The Git repository do have a xetex.web containing the majority of the WEB code. See my comment above regarding the +1 offset.

[Bruno Le Floch]

Ok, I now claim to understand what's going on, and suggest a fix below. Sadly my setup is a mess right now (ancient version of Ubuntu), I don't have autoreconf, and I cannot build XeTeX to test.

Three changes to do: remove the -1 in the definition of eTeX_text_offset, remove the +1 in the print_cmd_chr that you mention, and change a value to inter_char_text=17. The third change is best done by replacing

@d inter_char_text=16 {text from \\XeTeXinterchartoks}

by the following line (to be placed just below the definition of every_eof_text):

@d inter_char_text=XeTeX_inter_char_loc-eTeX_text_offset
  {|token_type| code for \.{\\XeTeXinterchartoks}}

---

Explanation: TeX needs two sets of integers to be in sync: from tex.web,

The codes for |output_text|, |every_par_text|, etc., are equal to a constant
plus the corresponding codes for token list parameters |output_routine_loc|,
|every_par_loc|, etc.

In tex.web this is ensured simply by the fact that in both sets, the integers are contiguous, from output_text=6 to write_text=15 (excluded) and from output_routine_loc=local_base+1 to toks_base=local_base+10 (excluded).

Then, etex.ch introduces another parameter every_eof_loc=local_base+10 in the second list and changes toks_base=local_base+11 by 1. In parallel, it sets every_eof_text=15 and write_text=16, but does that in a way that is slightly more robust against adding commands. Namely, eTeX_text_offset is defined as the difference between the two sets of parameters, simply computed as output_routine_loc-output_text, and every_eof_text and write_text are defined as the corresponding every_eof_loc and toks_base, minus the offset.

Next comes the changes to get xetex.web. First, there is the insertion of backed_up_char=4, which shifts the ..._text parameters by 1, leading to values output_text=7 to every_eof_text=16. This is not a problem for the eTeX changes: the offset between the two sets of number changes by 1, but is still equal to the same expression output_routine_loc-output_text. Next, we want to add to the end of both lists one new parameter for \XeTeXinterchartoks.

On the one hand, add XeTeX_inter_char_loc=local_base+11 and shift to toks_base=local_base+12. This is done correctly. On the other hand, we want to have every_eof_text=16, inter_char_text=17, and write_text=18 (i.e., to insert inter_char_text). Here comes the bug: the values that XeTeX currently uses are instead 17, 16, 19 in this order. Why? The erroneous shift by 1 in every_eof_text and write_text comes from the erroneous shift by 1 in the eTeX_text_offset. The erroneous value 16 for inter_char_text is simply hard-coded.

Because of this interchange between 16 and 17, the two sets of numbers didn't match, and the tracing code could not work. Presumably, the code changes were tested by looking at XeTeXinterchartoks: the shift by 1 in print_cmd_chr reflects that, since it gives the right output for this command only. Putting the various ..._text parameters in the right order fixes all those aspects.

One open question is to know whether the change also fixes the display of \tracingassigns=2\XeTeXinterchartoks1 2{ }. It might be an unrelated issue. I won't investigate now. Sorry for the length.

[Bruno Le Floch]

Another advisable change (for ancient Pascals which want the various cases to come in order): swap the two

inter_char_text: print_nl("<XeTeXinterchartoks> ");
every_eof_text: print_nl("<everyeof> ");

and swap the order of backed_up_char,backed_up a couple lines above. This is most likely unnecessary, but somewhat more consistent.

[Khaled Hosny]

Thanks, both fixes applied as 7a4fbc454151a74578ec2133cc80f16be7974f92 and 61f0dc4396db75d934bf1c1c91e3cb4624a73289. The \tracingassigns issues seems to be because \XeTeXinterchartoks is not handled in show_sa procedure, but it is not clear me how it should be handled.

[Just A. Man]

I can confirm that the original bug is gone: running xetex on

\begingroup % Just there to minimize tracing output
  \tracingmacros=2
  \tracingonline=1
  \everyvbox{\relax} % or whatever, non-empty.
  \vbox{}
\endgroup
\end

yields

(./mwe.tex
\everyvbox->\relax 
[1] )

on tty and in the log as expected.