Aubit 4GL SVN-Code

Bump version to 1.8.1.

configure is regenerated along with it. configure.ac reads the version
with m4_esyscmd_s at autoreconf time, so a bump needs autoreconf before
configure - otherwise the build keeps writing plug-ins to the old
plugins-<version> directory while the runtime, which gets its version
from lib/generated/project.c at make time, looks in the new one.

Optional, consent-based build reporting in configure.

configure can now send one line to the project saying that a build
happened and which back ends were detected, so we have some idea how
many people build Aubit and which drivers are still worth maintaining.

It only asks on a terminal, defaults to no, and never runs in a scripted
or CI build. --enable-usage-report sends without asking,
--disable-usage-report never asks, and DO_NOT_TRACK is honoured. The
prompt shows exactly what would be sent, and nothing is sent when
building or running programs afterwards - only here, and only on a yes.

Note the prompt reads /dev/tty rather than stdin. Autoconf detaches
stdin near the top of every generated configure

test -n "$DJDIR" || exec 7<&0 </dev/null

so that configure can never block on input, which means a prompt reading
stdin silently does nothing at all.

Test 217: put the test's own directory on the library path.

217 builds libprog2.so with "4glpc --as-dll" and links prog.4gl against
it, but nothing added that directory to LD_LIBRARY_PATH and the program
carries no rpath, so it died with "cannot open shared object file". That
is what the README.fail has meant since 2004 by "don't know why yet".

Also drops 240 from EXPECT_TO_FAIL_TESTS - the USING fix in r13027 makes
it pass.

Fix USING floating signs and the window message line.

USING: a floating sign was landing in the wrong column. Informix puts it
on the rightmost of its own mask positions that the digits and currency
symbol have not taken, and a single occurrence does not float at all.

---,###.## -12.34 " - 12.34" its positions are 1-3 only
----,--$.&& -123.45 " -$123.45" 6-7 hold digits, so it stops at 4
(($$,$$$.&&) -1.23 " ( $1.23)" the paren stops at its own pair

A suppressed thousands separator counts as one of those positions only
where the run of signs carries on past it. Test 240 now passes and comes
off the expected-to-fail list, verified with -esqli -tui.

Message line: a message line past the bottom of a window was clamped to
the last line; Informix uses the first line inside the window. The
correct behaviour was already there behind MSGLINEFIXUP but was not the
default. Test 230 matches all 28 of its captured Informix screens now.
Set MSGLINEFIXUP to keep the old clamp.

Suite: 1098 run, 29 failed, against 32 before.

Honour DBMONEY for MONEY formatting.

The runtime never read DBMONEY, so the currency symbol and decimal
separator were fixed at "$" and ".". It is now parsed the way Informix
defines it, [front][. or ,][back], into a money format alongside the
existing per context separators.

DBMONEY=DM, -> DM1234,56
DBMONEY=£. -> £1234.56
DBMONEY=,DM -> 1234,56DM

resource.c carries an internal default of "#." for DBMONEY, so only a
value differing from that counts as user set - MONEY keeps following
A4GL_NUMERIC otherwise, and default output is unchanged.

A symbol longer than one byte (DM, or £ in UTF-8) takes the extra from
the blanks left of the placeholder, leaving the digits where they are.
Where the field has no room the symbol is left off rather than pushing
digits out of it.

Suite: 1099 run, 32 failed, unchanged from r13025. Test 240 still fails
on its existing USING alignment bug, with byte identical output.

Add test 1906: NCHAR/NVCHAR semantics on multi byte data.

Covers character based length, subscripts and truncation, locale aware
case folding, and the NVARCHAR cases fixed in r13024. Skips itself if
the machine has no UTF-8 locale.

Fix NCHAR/NVCHAR handling of multi byte (UTF-8) data.

Bytes, characters and display columns were all being treated as one
number, which only holds for ASCII. nchar.c now has one function per
quantity and each caller uses the one it needs.

- subscripts and padding now work in characters; truncation no longer
cuts a UTF-8 sequence in half
- A4GL_wcswidth counted characters despite its name, so screen and
report layout now ask for columns instead
- upshift/downshift fold non-ASCII via towupper/towlower
- NCHAR was invisible to the ODBC driver, so those columns came back
empty; fgl_sizes[] also ran off the end for NCHAR/NVCHAR/INT8/SERIAL8
- NVARCHAR was compiled as DTYPE_CHAR, so no character aware path ever
applied to it. Now a real DTYPE_NVCHAR, along with the operators,
conversions and pop_var2/alloc entries that were missing behind it
- a locale named in the environment but not installed no longer aborts
the program at startup

Adds A4GL_COLLATE (off by default) to order comparisons with strcoll.

ASCII behaviour is unchanged. Suite: 1098 run, 32 failed, against 33
before the change.

4glc: two parsing hot spots - 4glc is now about a third faster

Profiled 4glc compiling a 7737 line module (callgrind, since perf is not
permitted here). Two things dominated, and neither was what I expected.

1. A4GL_get_current_comments() declared

char buff[200000]="";

An initialiser on an array that size makes the compiler zero all 200KB on
entry, and the function is called once per command - 53512 times in that
module, about 10GB of memset, 41% of total run time. Everything below
treats buff as a C string (strlen/strcat/strdup), so terminating the first
byte is all that is needed. The redundant strcpy(buff,"") that immediately
followed it goes too.

2. FGLPARSE_allow_token_state() was the linear scan version. The token groups
average about 120 entries, and the lexer calls this for every word that
spells a reserved word, so it was 23% of what remained. states_optimised.c
already generates the same table sorted with a bsearch lookup - it just was
not linked, because WANTKW_C pointed at mk_states.c. It now points at
states_optimised.c, using the absolute path the existing build rule for
that file defines.

7737 lines: 0.58s -> 0.39s
instructions: 21.6G -> 12.9G after (1); (2) took the remainder down
further

Both verified. The comment change was A/B'd on the path that actually uses
comments - A4GL_LEXTYPE=WRITE, which stores them in the .dat for fgllint -
and the output is byte identical apart from the embedded compile timestamp
(one byte, at offset 361, 28 seconds apart between the two runs). The full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

What is left is the semantic value stack. The %union carries char str[12288],
so sizeof(YYSTYPE) is 12KB, and bison copies that on every shift
("*++yyvsp = yylval") and every reduction ("yyval = yyvsp[1-yylen]"). That is
now 54% of the remaining time, and YYINITDEPTH 200 also puts a 2.4MB array on
the stack. Making str a pointer would remove most of it, but it touches every
$$.str in the .rule files, so it is not a change to make casually.

4glc: build the keyword state table from bison --xml, keep y.output as fallback

The per-state "which tokens can the parser accept here" table was scraped out
of bison's y.output, which is a human-readable report rather than an interface
- hence the %empty stripping, the dot-position scanning and the per-version
format workarounds, and hence a failure mode where a format change yields a
well-formed but wrong table and keywords silently become identifiers.

bison's --xml report carries exactly the same information, including the
<lookaheads> sets that are the whole reason y.output was needed in the first
place (the parser's own yypact/yycheck/yytable cannot supply them - see the
comment at the top of mk_states_c.in). It is one element per line, so it is
read with anchored patterns instead of prose parsing.

--xml is not in every bison and passing -x to one without it is fatal, so
configure now checks and sets BISON_XML_FLAG, which reaches the bison command
line as YACC_XML. mk_states_c uses the XML only if bison actually produced
one, and otherwise takes the original y.output route, which is preserved
unchanged. The two routes were run against each other on this grammar and
produce byte identical output, so the fallback is a real equivalent rather
than a degraded mode.

The script is now two small extractors - one for XML, one for y.output -
feeding a shared emitter over a flat "S <state> / T <token>" stream, so the
table building logic exists once.

Also made the group dedup deterministic. The key was built by "for (a in arr)",
whose order is unspecified, so the same set of tokens could produce different
key strings between runs: states that should have shared a group got separate
ones, and the generated file was not reproducible. Sorting the key first drops
the group count from 607 to 578 and makes repeated runs byte identical.

Verified: both routes produce the same mk_states.c, full build clean, and the
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run with
test 1413 still fixed. Test 706 shows up in this run and not the last, but it
passes on its own on both builds - it is flaky in batch, and the generated
table here is byte identical so parser behaviour cannot have changed.

4glc: record why the keyword/identifier table is scraped from y.output

Reading bison's y.output to build the per-state token table looks like it
should be unnecessary - yypact/yycheck/yytable are sitting right there in
y.tab.c, and bison uses exactly that scan in yysyntax_error() to list expected
tokens. I tried it, and it does not work.

Those tables only carry a state's EXPLICIT actions. A token that is acceptable
through a default reduction has no entry at all. State 29 is the clearest case:

1025 module_globals_section: actual_globals_section . [$end, KW_CSTART, ...
MAIN, FORMHANDLER]
GLOBALS shift, and go to state 12
$default reduce using rule 1025

Only GLOBALS has a yypact entry, but MAIN, FUNCTION, DEFINE and the rest are
perfectly legal there - the parser reduces rule 1025 first and shifts them in
the next state. A yypact based implementation was compiled alongside the
existing one and compared over all 4732 x 2324 (state, token) pairs: they
disagree on 146338 of them, and nearly all of those are the yypact version
being too restrictive in exactly this way. Using it would silently turn
keywords into identifiers.

Answering the question from the tables alone would mean following default
reductions transitively, and performing a reduction needs the parser's state
STACK to pop and compute the goto - yyss is local to yyparse() and the lexer
only ever sees the top state. bison's LALR lookahead sets already encode the
answer, and y.output is the only place bison publishes them.

So the y.output dependency is not a shortcut, it is the only route to the
information. Comment added at the top of mk_states_c.in so the next person
does not spend the afternoon rediscovering it, including the note that the
-r itemsets,lookaheads flags are load bearing: without the lookahead sets the
generated table is wrong rather than merely incomplete.

No functional change.