See a question on TeX.stackexchange, and especially Khaled Hosny's answer. It appears that XeTeX is not using the correct font to decide how to turn a math list into a horizontal list. Namely, it seems to be using the last used font (or last defined? or last selected with a font command?) rather than the font in which the math formula should be typeset (typically controlled by \textfont
etc.).
The problem occurs when mixing OpenType and non-OpenType fonts, more particularly when nesting math mode (even within a \setbox...\hbox{$...$}
construction, which should not have global consequences).
\global\font\xxx"[latinmodern-math.otf]"\relax \global\textfont 0\xxx \newbox\mybox \def\ignore#1{\setbox\mybox\hbox{#1}} \def\showhbox#1{\setbox0\hbox{#1}\showbox0\relax} \tracingall \showhbox{${\ \over\ }\ignore{a}$} \showhbox{${\ \over\ }\ignore{$\mathchar`a$}$} \showhbox{${\ \over\ }\ignore{a}$} \bye
Anonymous
Should be fixed by 9785737384bfe6080b8231f666ddf5b3861c957a, I think; please test.