Aubit 4GL SVN-Code

Honour DBMONEY for MONEY formatting.

The runtime never read DBMONEY, so the currency symbol and decimal
separator were fixed at "$" and ".". It is now parsed the way Informix
defines it, [front][. or ,][back], into a money format alongside the
existing per context separators.

DBMONEY=DM, -> DM1234,56
DBMONEY=£. -> £1234.56
DBMONEY=,DM -> 1234,56DM

resource.c carries an internal default of "#." for DBMONEY, so only a
value differing from that counts as user set - MONEY keeps following
A4GL_NUMERIC otherwise, and default output is unchanged.

A symbol longer than one byte (DM, or £ in UTF-8) takes the extra from
the blanks left of the placeholder, leaving the digits where they are.
Where the field has no room the symbol is left off rather than pushing
digits out of it.

Suite: 1099 run, 32 failed, unchanged from r13025. Test 240 still fails
on its existing USING alignment bug, with byte identical output.

Add test 1906: NCHAR/NVCHAR semantics on multi byte data.

Covers character based length, subscripts and truncation, locale aware
case folding, and the NVARCHAR cases fixed in r13024. Skips itself if
the machine has no UTF-8 locale.

Fix NCHAR/NVCHAR handling of multi byte (UTF-8) data.

Bytes, characters and display columns were all being treated as one
number, which only holds for ASCII. nchar.c now has one function per
quantity and each caller uses the one it needs.

- subscripts and padding now work in characters; truncation no longer
cuts a UTF-8 sequence in half
- A4GL_wcswidth counted characters despite its name, so screen and
report layout now ask for columns instead
- upshift/downshift fold non-ASCII via towupper/towlower
- NCHAR was invisible to the ODBC driver, so those columns came back
empty; fgl_sizes[] also ran off the end for NCHAR/NVCHAR/INT8/SERIAL8
- NVARCHAR was compiled as DTYPE_CHAR, so no character aware path ever
applied to it. Now a real DTYPE_NVCHAR, along with the operators,
conversions and pop_var2/alloc entries that were missing behind it
- a locale named in the environment but not installed no longer aborts
the program at startup

Adds A4GL_COLLATE (off by default) to order comparisons with strcoll.

ASCII behaviour is unchanged. Suite: 1098 run, 32 failed, against 33
before the change.

4glc: two parsing hot spots - 4glc is now about a third faster

Profiled 4glc compiling a 7737 line module (callgrind, since perf is not
permitted here). Two things dominated, and neither was what I expected.

1. A4GL_get_current_comments() declared

char buff[200000]="";

An initialiser on an array that size makes the compiler zero all 200KB on
entry, and the function is called once per command - 53512 times in that
module, about 10GB of memset, 41% of total run time. Everything below
treats buff as a C string (strlen/strcat/strdup), so terminating the first
byte is all that is needed. The redundant strcpy(buff,"") that immediately
followed it goes too.

2. FGLPARSE_allow_token_state() was the linear scan version. The token groups
average about 120 entries, and the lexer calls this for every word that
spells a reserved word, so it was 23% of what remained. states_optimised.c
already generates the same table sorted with a bsearch lookup - it just was
not linked, because WANTKW_C pointed at mk_states.c. It now points at
states_optimised.c, using the absolute path the existing build rule for
that file defines.

7737 lines: 0.58s -> 0.39s
instructions: 21.6G -> 12.9G after (1); (2) took the remainder down
further

Both verified. The comment change was A/B'd on the path that actually uses
comments - A4GL_LEXTYPE=WRITE, which stores them in the .dat for fgllint -
and the output is byte identical apart from the embedded compile timestamp
(one byte, at offset 361, 28 seconds apart between the two runs). The full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

What is left is the semantic value stack. The %union carries char str[12288],
so sizeof(YYSTYPE) is 12KB, and bison copies that on every shift
("*++yyvsp = yylval") and every reduction ("yyval = yyvsp[1-yylen]"). That is
now 54% of the remaining time, and YYINITDEPTH 200 also puts a 2.4MB array on
the stack. Making str a pointer would remove most of it, but it touches every
$$.str in the .rule files, so it is not a change to make casually.

4glc: build the keyword state table from bison --xml, keep y.output as fallback

The per-state "which tokens can the parser accept here" table was scraped out
of bison's y.output, which is a human-readable report rather than an interface
- hence the %empty stripping, the dot-position scanning and the per-version
format workarounds, and hence a failure mode where a format change yields a
well-formed but wrong table and keywords silently become identifiers.

bison's --xml report carries exactly the same information, including the
<lookaheads> sets that are the whole reason y.output was needed in the first
place (the parser's own yypact/yycheck/yytable cannot supply them - see the
comment at the top of mk_states_c.in). It is one element per line, so it is
read with anchored patterns instead of prose parsing.

--xml is not in every bison and passing -x to one without it is fatal, so
configure now checks and sets BISON_XML_FLAG, which reaches the bison command
line as YACC_XML. mk_states_c uses the XML only if bison actually produced
one, and otherwise takes the original y.output route, which is preserved
unchanged. The two routes were run against each other on this grammar and
produce byte identical output, so the fallback is a real equivalent rather
than a degraded mode.

The script is now two small extractors - one for XML, one for y.output -
feeding a shared emitter over a flat "S <state> / T <token>" stream, so the
table building logic exists once.

Also made the group dedup deterministic. The key was built by "for (a in arr)",
whose order is unspecified, so the same set of tokens could produce different
key strings between runs: states that should have shared a group got separate
ones, and the generated file was not reproducible. Sorting the key first drops
the group count from 607 to 578 and makes repeated runs byte identical.

Verified: both routes produce the same mk_states.c, full build clean, and the
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run with
test 1413 still fixed. Test 706 shows up in this run and not the last, but it
passes on its own on both builds - it is flaky in batch, and the generated
table here is byte identical so parser behaviour cannot have changed.

4glc: record why the keyword/identifier table is scraped from y.output

Reading bison's y.output to build the per-state token table looks like it
should be unnecessary - yypact/yycheck/yytable are sitting right there in
y.tab.c, and bison uses exactly that scan in yysyntax_error() to list expected
tokens. I tried it, and it does not work.

Those tables only carry a state's EXPLICIT actions. A token that is acceptable
through a default reduction has no entry at all. State 29 is the clearest case:

1025 module_globals_section: actual_globals_section . [$end, KW_CSTART, ...
MAIN, FORMHANDLER]
GLOBALS shift, and go to state 12
$default reduce using rule 1025

Only GLOBALS has a yypact entry, but MAIN, FUNCTION, DEFINE and the rest are
perfectly legal there - the parser reduces rule 1025 first and shifts them in
the next state. A yypact based implementation was compiled alongside the
existing one and compared over all 4732 x 2324 (state, token) pairs: they
disagree on 146338 of them, and nearly all of those are the yypact version
being too restrictive in exactly this way. Using it would silently turn
keywords into identifiers.

Answering the question from the tables alone would mean following default
reductions transitively, and performing a reduction needs the parser's state
STACK to pop and compute the goto - yyss is local to yyparse() and the lexer
only ever sees the top state. bison's LALR lookahead sets already encode the
answer, and y.output is the only place bison publishes them.

So the y.output dependency is not a shortcut, it is the only route to the
information. Comment added at the top of mk_states_c.in so the next person
does not spend the afternoon rediscovering it, including the note that the
-r itemsets,lookaheads flags are load bearing: without the lookahead sets the
generated table is wrong rather than merely incomplete.

No functional change.

4glc: emit the last parser state's token group, and bounds check the lookup

mk_states_c builds the per-state "which keyword tokens can the parser accept
here" table that the lexer uses (via FGLPARSE_allow_token_state) to decide
whether a word is a reserved word or an identifier. It emitted a state's group
when it saw the NEXT "State" header, and the END block only wrote the
terminator - so the highest numbered state was never emitted.

With the current grammar that is 4732 states (0-4731) described by 4731
entries, which left token_groups[4731] pointing at the terminating 0 and a NULL
dereference for anything asking about that state. It has been harmless only
because state 4731 happens to be a $default reduce state, where bison reduces
without asking for a lookahead so the lexer never queries it. Any grammar
change that makes the top numbered state one that needs a lookahead turns it
into a crash.

The per-state flush is now a function called from both the state header and
END, so the last state is emitted like every other. The table goes from 4731 to
4732 entries; the group dedup is unchanged at 607 distinct groups, and the
entries for states 0-4730 keep their existing positions, so nothing else moves.

Also added a bounds check. An out of range state now returns 1 (treat the word
as a keyword) rather than indexing past the table: if the table and the parser
ever get out of step, the old behaviour was to walk off the end of the array,
and the next worst thing would be silently turning every keyword in that state
into an identifier.

The same two changes are made in mk_states_opt.c, which generates the bsearch
variant. Note that variant is currently generated but not linked - 4glc links
rules/generated/mk_states.o, the linear scan version - so the live path is the
one in mk_states_c.in.

No behaviour change, as expected for a state that was never consulted: the full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

P-code: fix the typedef rule reading its type through the wrong union member

The TYPEDEF rule passed $<define_variables>2 for a dtype, but dtype yields
$<define_var> - a variable_element, not a define_variables. named_structs holds
a define_variables (a member list), so every typedef registered a
variable_element through the wrong union member and the name was unusable
afterwards.

A typedef name is only ever used as a pointer target or a sizeof operand, and
the named-struct machinery is the only way to record a name here, so the
typedef'd type is now wrapped in a one-element member list. That gives the
right size for sizeof and the right behaviour for a pointer.

Two things fell out of testing it:

* dtype had no pointer form for a typedef name, so "_dynelem_aa *aa=0;" - which
is how the generator declares every dynamic array - was a syntax error. Added
alongside the STRUCT and built-in pointer forms. The grammar conflict count is
unchanged (64 shift/reduce, 1 reduce/reduce).

* add_default_struct_list() was static; it is now exported so the rule can build
that one-element list.

Also removed three leftover debug printfs that went to stdout on every typedef
or typedef-name use ("TD", "Adding : x", "v=%p" and friends). c2pcode's output
is read by callers, so these were noise in it.

The p-code corpus run is now 28 passing with a single compile failure left - a
22,000 line module that exhausts the parser. It was 21 passing with 15 compile
failures before this round of work.

Grammar and lexer only; the C code generator is untouched. Full build clean,
OO suite passes, multi-module p-code still links and runs.

P-code: clear the remaining C-generation syntax errors

Nine tests in the aubit4gltest corpus failed to get through c2pcode with a
syntax error, and four more were rejected for "excess elements". All but two
of the compile failures are now gone: 15 down to 2, and 27 of the comparable
tests pass (was 21).

Most of this is on the code generation side rather than the grammar, since we
control what is emitted:

* _dtype_hint was declared just before its use in the PROMPT block, after
statements had already been emitted. The p-code grammar is C89, where a
declaration cannot follow a statement, so it is now declared in the block
prologue with the other locals.

* "a4gl_sqlca.sqlcode = a4gl_status = _fetcherr" is a chained assignment, and
the grammar's assign rule is "variable '=' expr". Split into two statements -
exactly equivalent, and clearer C.

* The generated event list declared aclfgl_event_list _sio_evt[n+1] where n
counts EVENTS, but the loop emits one entry per key code, so an event listing
several keys overflowed the array. C only warns about that; p-code rejected
it. Both backends now emit the unsized form and let the initialiser fix the
extent, so the count cannot drift.

* The ERR_CHK_* names are macros, and p-code skips every '#' line, so it saw
them as undefined variables. The generator emits the expansion when doing
p-code. There is no way to reach the preprocessor from the generator, so
expand_err_chk() in err_hand.c has to be kept in step with
incl/a4gl_incl_4gldef.h by hand.

Two additions to the grammar itself, both of which leave the conflict count
exactly where it was (64 shift/reduce, 1 reduce/reduce):

* "struct BINDING *reread" - dtype folds pointers into the type for CHAR, INT,
LONG, SHORT and VOID but had no form for a pointer to a named struct.

* sizeof of a TYPE rather than an expression, as in
"sizeof(_ordbind)/sizeof(struct BINDING)". The size is known at that point,
so it folds to a constant.

* A trailing comma in a brace initialiser list, which C allows and every
generated bind array uses.

Still failing to compile: one module using a typedef (the grammar's TYPEDEF
rule reads its dtype through the wrong union member, so it needs fixing before
it can help), and one 22,000 line module that exhausts the parser.

Verified: full aubit4gltest run unchanged against the r13015 baseline - the
generator changes are all behind A4GL_doing_pcode() except the chained
assignment split, which is equivalent C. Multi-module p-code still links and
runs, and the OO suite passes.

P-code: register struct s_field_name and s_field_name_list

Same gap as sDependantTable - these are emitted into generated code for field
list handling but were not among the p-code compiler's predefined structs, so
any module using them stopped with "Structure s_field_name not found or
defined". Definitions taken from incl/a4gl_incl_4gldef.h.

Found by running the aubit4gltest corpus through the p-code pipeline using each
test's own run_* script for execution and comparison (the scripts invoke the
program as "$DBG ./prog.4ae" and diff the .out files the program writes against
their .expected). 42 of those tests are comparable that way and 21 now pass,
with multi-module tests linked by link_fgl - which did not fail once.