Aubit 4GL SVN-Code

4glc: two parsing hot spots - 4glc is now about a third faster

Profiled 4glc compiling a 7737 line module (callgrind, since perf is not
permitted here). Two things dominated, and neither was what I expected.

1. A4GL_get_current_comments() declared

char buff[200000]="";

An initialiser on an array that size makes the compiler zero all 200KB on
entry, and the function is called once per command - 53512 times in that
module, about 10GB of memset, 41% of total run time. Everything below
treats buff as a C string (strlen/strcat/strdup), so terminating the first
byte is all that is needed. The redundant strcpy(buff,"") that immediately
followed it goes too.

2. FGLPARSE_allow_token_state() was the linear scan version. The token groups
average about 120 entries, and the lexer calls this for every word that
spells a reserved word, so it was 23% of what remained. states_optimised.c
already generates the same table sorted with a bsearch lookup - it just was
not linked, because WANTKW_C pointed at mk_states.c. It now points at
states_optimised.c, using the absolute path the existing build rule for
that file defines.

7737 lines: 0.58s -> 0.39s
instructions: 21.6G -> 12.9G after (1); (2) took the remainder down
further

Both verified. The comment change was A/B'd on the path that actually uses
comments - A4GL_LEXTYPE=WRITE, which stores them in the .dat for fgllint -
and the output is byte identical apart from the embedded compile timestamp
(one byte, at offset 361, 28 seconds apart between the two runs). The full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

What is left is the semantic value stack. The %union carries char str[12288],
so sizeof(YYSTYPE) is 12KB, and bison copies that on every shift
("*++yyvsp = yylval") and every reduction ("yyval = yyvsp[1-yylen]"). That is
now 54% of the remaining time, and YYINITDEPTH 200 also puts a 2.4MB array on
the stack. Making str a pointer would remove most of it, but it touches every
$$.str in the .rule files, so it is not a change to make casually.

4glc: build the keyword state table from bison --xml, keep y.output as fallback

The per-state "which tokens can the parser accept here" table was scraped out
of bison's y.output, which is a human-readable report rather than an interface
- hence the %empty stripping, the dot-position scanning and the per-version
format workarounds, and hence a failure mode where a format change yields a
well-formed but wrong table and keywords silently become identifiers.

bison's --xml report carries exactly the same information, including the
<lookaheads> sets that are the whole reason y.output was needed in the first
place (the parser's own yypact/yycheck/yytable cannot supply them - see the
comment at the top of mk_states_c.in). It is one element per line, so it is
read with anchored patterns instead of prose parsing.

--xml is not in every bison and passing -x to one without it is fatal, so
configure now checks and sets BISON_XML_FLAG, which reaches the bison command
line as YACC_XML. mk_states_c uses the XML only if bison actually produced
one, and otherwise takes the original y.output route, which is preserved
unchanged. The two routes were run against each other on this grammar and
produce byte identical output, so the fallback is a real equivalent rather
than a degraded mode.

The script is now two small extractors - one for XML, one for y.output -
feeding a shared emitter over a flat "S <state> / T <token>" stream, so the
table building logic exists once.

Also made the group dedup deterministic. The key was built by "for (a in arr)",
whose order is unspecified, so the same set of tokens could produce different
key strings between runs: states that should have shared a group got separate
ones, and the generated file was not reproducible. Sorting the key first drops
the group count from 607 to 578 and makes repeated runs byte identical.

Verified: both routes produce the same mk_states.c, full build clean, and the
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run with
test 1413 still fixed. Test 706 shows up in this run and not the last, but it
passes on its own on both builds - it is flaky in batch, and the generated
table here is byte identical so parser behaviour cannot have changed.

4glc: record why the keyword/identifier table is scraped from y.output

Reading bison's y.output to build the per-state token table looks like it
should be unnecessary - yypact/yycheck/yytable are sitting right there in
y.tab.c, and bison uses exactly that scan in yysyntax_error() to list expected
tokens. I tried it, and it does not work.

Those tables only carry a state's EXPLICIT actions. A token that is acceptable
through a default reduction has no entry at all. State 29 is the clearest case:

1025 module_globals_section: actual_globals_section . [$end, KW_CSTART, ...
MAIN, FORMHANDLER]
GLOBALS shift, and go to state 12
$default reduce using rule 1025

Only GLOBALS has a yypact entry, but MAIN, FUNCTION, DEFINE and the rest are
perfectly legal there - the parser reduces rule 1025 first and shifts them in
the next state. A yypact based implementation was compiled alongside the
existing one and compared over all 4732 x 2324 (state, token) pairs: they
disagree on 146338 of them, and nearly all of those are the yypact version
being too restrictive in exactly this way. Using it would silently turn
keywords into identifiers.

Answering the question from the tables alone would mean following default
reductions transitively, and performing a reduction needs the parser's state
STACK to pop and compute the goto - yyss is local to yyparse() and the lexer
only ever sees the top state. bison's LALR lookahead sets already encode the
answer, and y.output is the only place bison publishes them.

So the y.output dependency is not a shortcut, it is the only route to the
information. Comment added at the top of mk_states_c.in so the next person
does not spend the afternoon rediscovering it, including the note that the
-r itemsets,lookaheads flags are load bearing: without the lookahead sets the
generated table is wrong rather than merely incomplete.

No functional change.

4glc: emit the last parser state's token group, and bounds check the lookup

mk_states_c builds the per-state "which keyword tokens can the parser accept
here" table that the lexer uses (via FGLPARSE_allow_token_state) to decide
whether a word is a reserved word or an identifier. It emitted a state's group
when it saw the NEXT "State" header, and the END block only wrote the
terminator - so the highest numbered state was never emitted.

With the current grammar that is 4732 states (0-4731) described by 4731
entries, which left token_groups[4731] pointing at the terminating 0 and a NULL
dereference for anything asking about that state. It has been harmless only
because state 4731 happens to be a $default reduce state, where bison reduces
without asking for a lookahead so the lexer never queries it. Any grammar
change that makes the top numbered state one that needs a lookahead turns it
into a crash.

The per-state flush is now a function called from both the state header and
END, so the last state is emitted like every other. The table goes from 4731 to
4732 entries; the group dedup is unchanged at 607 distinct groups, and the
entries for states 0-4730 keep their existing positions, so nothing else moves.

Also added a bounds check. An out of range state now returns 1 (treat the word
as a keyword) rather than indexing past the table: if the table and the parser
ever get out of step, the old behaviour was to walk off the end of the array,
and the next worst thing would be silently turning every keyword in that state
into an identifier.

The same two changes are made in mk_states_opt.c, which generates the bsearch
variant. Note that variant is currently generated but not linked - 4glc links
rules/generated/mk_states.o, the linear scan version - so the live path is the
one in mk_states_c.in.

No behaviour change, as expected for a state that was never consulted: the full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

P-code: fix the typedef rule reading its type through the wrong union member

The TYPEDEF rule passed $<define_variables>2 for a dtype, but dtype yields
$<define_var> - a variable_element, not a define_variables. named_structs holds
a define_variables (a member list), so every typedef registered a
variable_element through the wrong union member and the name was unusable
afterwards.

A typedef name is only ever used as a pointer target or a sizeof operand, and
the named-struct machinery is the only way to record a name here, so the
typedef'd type is now wrapped in a one-element member list. That gives the
right size for sizeof and the right behaviour for a pointer.

Two things fell out of testing it:

* dtype had no pointer form for a typedef name, so "_dynelem_aa *aa=0;" - which
is how the generator declares every dynamic array - was a syntax error. Added
alongside the STRUCT and built-in pointer forms. The grammar conflict count is
unchanged (64 shift/reduce, 1 reduce/reduce).

* add_default_struct_list() was static; it is now exported so the rule can build
that one-element list.

Also removed three leftover debug printfs that went to stdout on every typedef
or typedef-name use ("TD", "Adding : x", "v=%p" and friends). c2pcode's output
is read by callers, so these were noise in it.

The p-code corpus run is now 28 passing with a single compile failure left - a
22,000 line module that exhausts the parser. It was 21 passing with 15 compile
failures before this round of work.

Grammar and lexer only; the C code generator is untouched. Full build clean,
OO suite passes, multi-module p-code still links and runs.

P-code: clear the remaining C-generation syntax errors

Nine tests in the aubit4gltest corpus failed to get through c2pcode with a
syntax error, and four more were rejected for "excess elements". All but two
of the compile failures are now gone: 15 down to 2, and 27 of the comparable
tests pass (was 21).

Most of this is on the code generation side rather than the grammar, since we
control what is emitted:

* _dtype_hint was declared just before its use in the PROMPT block, after
statements had already been emitted. The p-code grammar is C89, where a
declaration cannot follow a statement, so it is now declared in the block
prologue with the other locals.

* "a4gl_sqlca.sqlcode = a4gl_status = _fetcherr" is a chained assignment, and
the grammar's assign rule is "variable '=' expr". Split into two statements -
exactly equivalent, and clearer C.

* The generated event list declared aclfgl_event_list _sio_evt[n+1] where n
counts EVENTS, but the loop emits one entry per key code, so an event listing
several keys overflowed the array. C only warns about that; p-code rejected
it. Both backends now emit the unsized form and let the initialiser fix the
extent, so the count cannot drift.

* The ERR_CHK_* names are macros, and p-code skips every '#' line, so it saw
them as undefined variables. The generator emits the expansion when doing
p-code. There is no way to reach the preprocessor from the generator, so
expand_err_chk() in err_hand.c has to be kept in step with
incl/a4gl_incl_4gldef.h by hand.

Two additions to the grammar itself, both of which leave the conflict count
exactly where it was (64 shift/reduce, 1 reduce/reduce):

* "struct BINDING *reread" - dtype folds pointers into the type for CHAR, INT,
LONG, SHORT and VOID but had no form for a pointer to a named struct.

* sizeof of a TYPE rather than an expression, as in
"sizeof(_ordbind)/sizeof(struct BINDING)". The size is known at that point,
so it folds to a constant.

* A trailing comma in a brace initialiser list, which C allows and every
generated bind array uses.

Still failing to compile: one module using a typedef (the grammar's TYPEDEF
rule reads its dtype through the wrong union member, so it needs fixing before
it can help), and one 22,000 line module that exhausts the parser.

Verified: full aubit4gltest run unchanged against the r13015 baseline - the
generator changes are all behind A4GL_doing_pcode() except the chained
assignment split, which is equivalent C. Multi-module p-code still links and
runs, and the OO suite passes.

P-code: register struct s_field_name and s_field_name_list

Same gap as sDependantTable - these are emitted into generated code for field
list handling but were not among the p-code compiler's predefined structs, so
any module using them stopped with "Structure s_field_name not found or
defined". Definitions taken from incl/a4gl_incl_4gldef.h.

Found by running the aubit4gltest corpus through the p-code pipeline using each
test's own run_* script for execution and comparison (the scripts invoke the
program as "$DBG ./prog.4ae" and diff the .out files the program writes against
their .expected). 42 of those tests are comparable that way and 21 now pass,
with multi-module tests linked by link_fgl - which did not fail once.

Make the P-code compiler and runner work; restore panel/form to the TUI link

P-code
------
P-code had never worked on a 64 bit build. bin/runner_fgl did not even get
built, and no 4GL module could be turned into p-code at all. It now compiles
and runs ordinary 4GL, and on the test programs used here its output is
byte-identical to the C backend.

The runner would not build: compilers/pcode/fgl_calls.h registered
&A4GLSTK_pushFunction, the v1 stack API, which was retired and is now a poison
macro in a4gl_4gl_callable.h. Updated to pushFunction_v2 / popFunction_nl - the
handlers for those shapes already existed. A4GL_swap_bind_stmt was also
declared char** in a4gl_sql.h and void** in a4gl_4gl_callable.h, so anything
including both headers could not compile; the implementation only saves and
restores an opaque struct BINDING *, so void** is correct.

Sizes and signs, all 64 bit issues:

* a4gl_htonl/a4gl_ntohl returned htonl()'s uint32_t as a long, which zero
extends on LP64, so every negative value written through the PACKED packer
came back positive - -1 read back as 4294967295. The x1element "no subscript"
marker was the first casualty. Fixed by sign extending through int32_t; the
bytes on disk are unchanged for values that fit in 32 bits, and readers only
look at the low 32 bits, so existing files still read correctly. The magic
numbers (0xa4fc2345 and friends) have their top bit set, so the two places
that compare them now do so at 32 bits.

* The p-code type model said a long and a pointer were 4 bytes while the runner
reads and writes both through a C long. Widening it alone was not enough:
get_var_ptr reported only a size, and the read/write dispatch keyed off it
with 4 meaning long and 8 meaning double - so an 8 byte slot looked like a
double and every variable read back as garbage. get_var_ptr now reports the
dtype as well, each width is read and written as itself, and DLONG/DDBL/DPTR
are sized from the real C types. This is what fixed accumulation across a
loop, char variables corrupting the heap, and the lost first element of an
array initialiser.

* A string literal's address was cast through (int) in common_eval.c, which
truncated it and crashed printf.

The p-code compiler skips every '#' line, so it never sees a #define. NULL,
LABEL_USED, fglvarchar, the 34 OP_* codes and the REPORT_* codes were all
unknown to it. The OP_* and REPORT_* values now come from the real macros via
compilers/pcode/op_lookup.c rather than being copied, so they cannot drift -
which is exactly what had happened to the predefined BINDING struct, six
members against the header's seven, and to REPORT_NOTHING.

A4GL_push_long and A4GL_push_int were still keyword-bound to constants, but
generated code pushes variables far more often. Retired the same way
A4GL_push_char and A4GL_push_variable already had been, so they resolve
through the runner's call table.

An unspecified array bound is held as -1, which was multiplied into a negative
total_size - and that is what the runner mallocs for a static, so loading any
module with a "char x[]=..." aborted. Sizes are now taken from the initialiser
where one is given.

On the generator side, several things emitted only for the C backend are not
parseable by p-code's cut down C grammar, and are now suppressed when
A4GL_doing_pcode(): forward declarations for functions, reports and MAIN;
the version helper (which lacks the A4GL_FUNCTION marker p-code requires);
and "(void) x;" between two declarations, which p-code's C89 grammar rejects.
_objData is emitted before the locals for the same reason. ERRCHK now passes
the module name as a string literal, which is what the grammar wants.
MARK_SCOPE_MODULE, which p-code switched on, prefixed declarations but not
uses, emitting "long L_total;" alongside "total=0;" in the same function.

Suppressing the MAIN prototype means an ordinary "MAIN ... END MAIN" program
now compiles to p-code, removing the constraint documented in
compilers/pcode/README.

TUI link
--------
m4/aubit_ui.m4 substituted CURSES_LIB_NAME="-lncurses". The old configure.in
set "-lform -lpanel -lncurses", and ui_curses/Makefile.in still expands
@CURSES_LIB_NAME@, so the rewrite alone dropped panel and form and left
libUI_TUI.so with an undefined update_panels - the TUI driver could not load,
which is why so much had to be run in CONSOLE mode. configure now detects
panel and form and lists them dependents-first.

Testing
-------
aubit4gltest, all 21 ranges, -esqli -tui, against a build of r12988 (the last
revision before the configure rewrite):

r12988 1098 run, 59 skipped, 35 failed
this 1098 run, 59 skipped, 33 failed

The two differences are tests 706 and 1413. Run individually, 706 passes on
both (it is flaky in batch), and 1413 - logical report layout - fails on
r12988 and passes here. No regressions.

For reference, with the TUI driver broken these runs managed 283 tests with
449 skipped and five ranges hanging on menu tests looping on EOF.

Build warning cleanup, object system fixes, embedded Python

Three strands of work.

Build warnings
--------------
Top-level build warnings are down from roughly 2700 to roughly 230, and
nothing fatal to GCC 14/15 remains -- those promote
-Wimplicit-function-declaration, -Wincompatible-pointer-types,
-Wint-conversion and -Wimplicit-int from warnings to errors, which is what
the Ubuntu 26.04 build reports were hitting.

Mostly noise (sign-compare, unused-but-set, missing prototypes, dead
declarations), but some real bugs came out of it, including several places
passing a long* where an int* was expected. On LP64 that leaves the top
four bytes of an XDR *_len uninitialised; 4GL 'integer' maps to C 'long'
while the XDR members are u_int, so it turned up five separate times.

Also added svn:keywords=Id to the modules that had lost it.

4GL object system
-----------------
Inheritance was broken in a way that made most of it unusable, plus two
older bugs found while testing the fix:

* objData->base.objectid is not a 4GL variable, so the root-scanning
collector could not see it and disposed of a parent while a live child
still pointed at it. The first inherited method call after that failed.
Objects now hold a counted reference on their parent: refCnt counts
references held by other objects, the generated constructor/destructor
inc/dec it, and the collector only disposes when no 4GL variable refers
to an object AND its refCnt is 0. Fixing this fixed inheritance, castTo
and polymorphism together.

* Destructors are registered as "<type>.~" but A4GL_destroy_object looked
for "<type>.-", so it had always been a silent no-op.

* Class methods pushed a call frame on entry but only popped it on an
explicit RETURN -- printPopFunction was commented out in the class
epilogue. Every call to a method that fell off the end leaked a frame,
giving "Function calls too deep" after ~2000 calls. Plain functions were
always correct; this was specific to classes.

* castTo could not be reached from 4GL at all. The lookup is a plain
strcmp and the compiler lowercases method names, but only ":X.castTo"
was registered, so "let r = s.castTo(...)" fell through to
RouteToParent and walked off the top of the chain. There are now two
registrations with different calling conventions: ".castTo" returns the
object id (what getObject needs for an implicit upcast like "let f = m")
and ".castto" pushes it (the normal 4GL convention).

* Polymorphic overloads resolved backwards. getSigForTopOfStack built the
signature from the top of the stack, which is the LAST argument, while
the compiler registers signatures in declaration order -- so
f(integer,char) dispatched to f(char,integer). Only visible with mixed
parameter types, which is why the existing sample never showed it.

New samples in tools/test/OO exercising a three-level chain (shape <-
rect <- square): method resolution, castTo, refcounts and object lifetime
under churn. tools/test/OO/README updated.

Embedded Python
---------------
Optional object(python), implemented in C but registered like a 4GL class,
so it is used the same way:

define p object(python)
let p = python.new()
call p.import("math")
display p.call("math.pow", 2, 10)

Methods: new, import, addpath, available, run, eval, call, set, get,
lasterror, version. Each object gets its own namespace. import/addpath
exist so module names and paths are passed as values rather than pasted
into Python source; available() reports whether a package could be
imported without importing it and without setting the error status, so a
Python package can be an optional dependency of a 4GL program.

configure detects an embeddable Python (pkg-config python3-embed, then
python3-config --embed) and link-tests it before enabling anything:

./configure --with-python / --without-python / (auto by default)

With no python3-dev the implementation compiles away, nothing links
against libpython, and object(python) reports that support was not built
in. Python's headers go on the include path for python.c alone, since it
ships object.h, token.h and compile.h which would otherwise shadow ours.

Documented in docs/README-Python.txt, example in
tools/test/OO/test_python.4gl.

Verified: full build clean in both configurations, tools/test builds, and
the OO samples pass.

Fix the real bugs behind the remaining compiler warnings

These are the ones set aside earlier as "the interesting shortlist" - every
warning here was pointing at code that does not do what it was written to do.
A full clean build stays at exit 0 and drops from 754 to 721 warnings.

Memory:

- compilers/4glc/parsehelp.c: malloc(sizeof(expr_str)) followed by memcpy of
sizeof(l->list.list_val[a]) - the size of the *pointer*. It copied 8 bytes
and left the rest of the new expr_str uninitialised.
- lib/libaubit4gl/sql_common.c: A4GL_free_prepare() called
blank_any_cursors_using(sid) after free(sid). That function only compares the
pointer, so it now runs before the free - otherwise any cursor still holding
the sid keeps a dangling statement pointer.
- lib/libsql/odbc/sqlex.c: A4GL_debug("-%p", ptr) after free(ptr). Reading the
pointer value after the free is undefined even just to print it; logs first.
- lib/liblogical/processor/process_report.c: fclose() on a stream from popen().
Now pclose(), so the child is reaped and its status collected. The fwrite in
the copy loop was also inside an assert(), so a build with NDEBUG would have
dropped the copy entirely - it is now a separate statement.
- compilers/4glc/compile.c: two sprintf calls passing incl_path as both the
destination and the first argument. Overlapping source and destination is
undefined; they now format the suffix into a temporary and strcat it.

Logic that never ran, or always ran:

- compilers/4glc/lint.c: "whencode & 0x15 == WHEN_CALL" - 0x15 masks off bit 1
and WHEN_CALL is 2, so the WHENEVER ERROR CALL branch could never be taken and
lint never checked the named function existed. The action is the low nibble
(php.c uses & 0xf for the action, >> 4 for the condition), so it is now 0xf.
- compilers/4glc/lint.c: system_function_dtype() fell off the end without
returning. Callers test for -2, and the two other copies of this function
(calltree.c, prototypes.c) both end with return -2 - this one now does too.
- compilers/fcompile/dump_scr.c: get_attr_from_field() fell off the end when it
found nothing, returning whatever was in the register. Returns 0, which is
what its own "not applicable" path returns and what callers test for.
- lib/libaubit4gl/function_call_stack.c: "moduleName == '\0'" compared the
pointer against NULL, but its unset value is "" - so the no-module branch
never fired. Tests the string now.
- lib/libui/ui_xml/uilib/uilib.c and ui_json/uilib/uilib.c: "rval < 0" where
rval is the size_t returned by iconv(), which reports failure as (size_t)-1.
Conversion errors were silently ignored in both copies.
- compilers/4glc/variables_new.c: "if (idtype!=idtype)" - a self comparison, so
dead. What it meant is not recoverable; removed with a note.
- 16 -Waddress cases across sqlexpr.c, report.c, stack.c, mod.c, json.c, xml.c
and two generic_ui.c copies: tests on the address of a struct's char array,
which is never NULL. Each had a real test beside it (strlen, [0], or a
function call), so the redundant half is gone and the meaning is unchanged.

Format strings:

- compilers/4glc/prototypes.c: three fprintf calls passing arguments to format
strings with no conversions at all.
- tools/asql/parse.l: "%d" given strlen() and sizeof() results, ie size_t.

Left alone deliberately: compile_c.c's "arr_subscripts_len >= 0" was always true
on an unsigned member, but changing it to "> 0" would alter what happens for an
empty subscript list, so only the dead half was dropped.