mk_states_c builds the per-state "which keyword tokens can the parser accept
here" table that the lexer uses (via FGLPARSE_allow_token_state) to decide
whether a word is a reserved word or an identifier. It emitted a state's group
when it saw the NEXT "State" header, and the END block only wrote the
terminator - so the highest numbered state was never emitted.
With the current grammar that is 4732 states (0-4731) described by 4731
entries, which left token_groups[4731] pointing at the terminating 0 and a NULL
dereference for anything asking about that state. It has been harmless only
because state 4731 happens to be a $default reduce state, where bison reduces
without asking for a lookahead so the lexer never queries it. Any grammar
change that makes the top numbered state one that needs a lookahead turns it
into a crash.
The per-state flush is now a function called from both the state header and
END, so the last state is emitted like every other. The table goes from 4731 to
4732 entries; the group dedup is unchanged at 607 distinct groups, and the
entries for states 0-4730 keep their existing positions, so nothing else moves.
Also added a bounds check. An out of range state now returns 1 (treat the word
as a keyword) rather than indexing past the table: if the table and the parser
ever get out of step, the old behaviour was to walk off the end of the array,
and the next worst thing would be silently turning every keyword in that state
into an identifier.
The same two changes are made in mk_states_opt.c, which generates the bsearch
variant. Note that variant is currently generated but not linked - 4glc links
rules/generated/mk_states.o, the linear scan version - so the live path is the
one in mk_states_c.in.
No behaviour change, as expected for a state that was never consulted: the full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.
