Aubit 4GL SVN-Code

4glc: emit the last parser state's token group, and bounds check the lookup

mk_states_c builds the per-state "which keyword tokens can the parser accept
here" table that the lexer uses (via FGLPARSE_allow_token_state) to decide
whether a word is a reserved word or an identifier. It emitted a state's group
when it saw the NEXT "State" header, and the END block only wrote the
terminator - so the highest numbered state was never emitted.

With the current grammar that is 4732 states (0-4731) described by 4731
entries, which left token_groups[4731] pointing at the terminating 0 and a NULL
dereference for anything asking about that state. It has been harmless only
because state 4731 happens to be a $default reduce state, where bison reduces
without asking for a lookahead so the lexer never queries it. Any grammar
change that makes the top numbered state one that needs a lookahead turns it
into a crash.

The per-state flush is now a function called from both the state header and
END, so the last state is emitted like every other. The table goes from 4731 to
4732 entries; the group dedup is unchanged at 607 distinct groups, and the
entries for states 0-4730 keep their existing positions, so nothing else moves.

Also added a bounds check. An out of range state now returns 1 (treat the word
as a keyword) rather than indexing past the table: if the table and the parser
ever get out of step, the old behaviour was to walk off the end of the array,
and the next worst thing would be silently turning every keyword in that state
into an identifier.

The same two changes are made in mk_states_opt.c, which generates the bsearch
variant. Note that variant is currently generated but not linked - 4glc links
rules/generated/mk_states.o, the linear scan version - so the live path is the
one in mk_states_c.in.

No behaviour change, as expected for a state that was never consulted: the full
aubit4gltest run (-esqli -tui, all 21 ranges) is unchanged at 1098 run, 33
failed, 1065 passed.

P-code: fix the typedef rule reading its type through the wrong union member

The TYPEDEF rule passed $<define_variables>2 for a dtype, but dtype yields
$<define_var> - a variable_element, not a define_variables. named_structs holds
a define_variables (a member list), so every typedef registered a
variable_element through the wrong union member and the name was unusable
afterwards.

A typedef name is only ever used as a pointer target or a sizeof operand, and
the named-struct machinery is the only way to record a name here, so the
typedef'd type is now wrapped in a one-element member list. That gives the
right size for sizeof and the right behaviour for a pointer.

Two things fell out of testing it:

* dtype had no pointer form for a typedef name, so "_dynelem_aa *aa=0;" - which
is how the generator declares every dynamic array - was a syntax error. Added
alongside the STRUCT and built-in pointer forms. The grammar conflict count is
unchanged (64 shift/reduce, 1 reduce/reduce).

* add_default_struct_list() was static; it is now exported so the rule can build
that one-element list.

Also removed three leftover debug printfs that went to stdout on every typedef
or typedef-name use ("TD", "Adding : x", "v=%p" and friends). c2pcode's output
is read by callers, so these were noise in it.

The p-code corpus run is now 28 passing with a single compile failure left - a
22,000 line module that exhausts the parser. It was 21 passing with 15 compile
failures before this round of work.

Grammar and lexer only; the C code generator is untouched. Full build clean,
OO suite passes, multi-module p-code still links and runs.

P-code: clear the remaining C-generation syntax errors

Nine tests in the aubit4gltest corpus failed to get through c2pcode with a
syntax error, and four more were rejected for "excess elements". All but two
of the compile failures are now gone: 15 down to 2, and 27 of the comparable
tests pass (was 21).

Most of this is on the code generation side rather than the grammar, since we
control what is emitted:

* _dtype_hint was declared just before its use in the PROMPT block, after
statements had already been emitted. The p-code grammar is C89, where a
declaration cannot follow a statement, so it is now declared in the block
prologue with the other locals.

* "a4gl_sqlca.sqlcode = a4gl_status = _fetcherr" is a chained assignment, and
the grammar's assign rule is "variable '=' expr". Split into two statements -
exactly equivalent, and clearer C.

* The generated event list declared aclfgl_event_list _sio_evt[n+1] where n
counts EVENTS, but the loop emits one entry per key code, so an event listing
several keys overflowed the array. C only warns about that; p-code rejected
it. Both backends now emit the unsized form and let the initialiser fix the
extent, so the count cannot drift.

* The ERR_CHK_* names are macros, and p-code skips every '#' line, so it saw
them as undefined variables. The generator emits the expansion when doing
p-code. There is no way to reach the preprocessor from the generator, so
expand_err_chk() in err_hand.c has to be kept in step with
incl/a4gl_incl_4gldef.h by hand.

Two additions to the grammar itself, both of which leave the conflict count
exactly where it was (64 shift/reduce, 1 reduce/reduce):

* "struct BINDING *reread" - dtype folds pointers into the type for CHAR, INT,
LONG, SHORT and VOID but had no form for a pointer to a named struct.

* sizeof of a TYPE rather than an expression, as in
"sizeof(_ordbind)/sizeof(struct BINDING)". The size is known at that point,
so it folds to a constant.

* A trailing comma in a brace initialiser list, which C allows and every
generated bind array uses.

Still failing to compile: one module using a typedef (the grammar's TYPEDEF
rule reads its dtype through the wrong union member, so it needs fixing before
it can help), and one 22,000 line module that exhausts the parser.

Verified: full aubit4gltest run unchanged against the r13015 baseline - the
generator changes are all behind A4GL_doing_pcode() except the chained
assignment split, which is equivalent C. Multi-module p-code still links and
runs, and the OO suite passes.

P-code: register struct s_field_name and s_field_name_list

Same gap as sDependantTable - these are emitted into generated code for field
list handling but were not among the p-code compiler's predefined structs, so
any module using them stopped with "Structure s_field_name not found or
defined". Definitions taken from incl/a4gl_incl_4gldef.h.

Found by running the aubit4gltest corpus through the p-code pipeline using each
test's own run_* script for execution and comparison (the scripts invoke the
program as "$DBG ./prog.4ae" and diff the .out files the program writes against
their .expected). 42 of those tests are comparable that way and 21 now pass,
with multi-module tests linked by link_fgl - which did not fail once.

Make the P-code compiler and runner work; restore panel/form to the TUI link

P-code
------
P-code had never worked on a 64 bit build. bin/runner_fgl did not even get
built, and no 4GL module could be turned into p-code at all. It now compiles
and runs ordinary 4GL, and on the test programs used here its output is
byte-identical to the C backend.

The runner would not build: compilers/pcode/fgl_calls.h registered
&A4GLSTK_pushFunction, the v1 stack API, which was retired and is now a poison
macro in a4gl_4gl_callable.h. Updated to pushFunction_v2 / popFunction_nl - the
handlers for those shapes already existed. A4GL_swap_bind_stmt was also
declared char** in a4gl_sql.h and void** in a4gl_4gl_callable.h, so anything
including both headers could not compile; the implementation only saves and
restores an opaque struct BINDING *, so void** is correct.

Sizes and signs, all 64 bit issues:

* a4gl_htonl/a4gl_ntohl returned htonl()'s uint32_t as a long, which zero
extends on LP64, so every negative value written through the PACKED packer
came back positive - -1 read back as 4294967295. The x1element "no subscript"
marker was the first casualty. Fixed by sign extending through int32_t; the
bytes on disk are unchanged for values that fit in 32 bits, and readers only
look at the low 32 bits, so existing files still read correctly. The magic
numbers (0xa4fc2345 and friends) have their top bit set, so the two places
that compare them now do so at 32 bits.

* The p-code type model said a long and a pointer were 4 bytes while the runner
reads and writes both through a C long. Widening it alone was not enough:
get_var_ptr reported only a size, and the read/write dispatch keyed off it
with 4 meaning long and 8 meaning double - so an 8 byte slot looked like a
double and every variable read back as garbage. get_var_ptr now reports the
dtype as well, each width is read and written as itself, and DLONG/DDBL/DPTR
are sized from the real C types. This is what fixed accumulation across a
loop, char variables corrupting the heap, and the lost first element of an
array initialiser.

* A string literal's address was cast through (int) in common_eval.c, which
truncated it and crashed printf.

The p-code compiler skips every '#' line, so it never sees a #define. NULL,
LABEL_USED, fglvarchar, the 34 OP_* codes and the REPORT_* codes were all
unknown to it. The OP_* and REPORT_* values now come from the real macros via
compilers/pcode/op_lookup.c rather than being copied, so they cannot drift -
which is exactly what had happened to the predefined BINDING struct, six
members against the header's seven, and to REPORT_NOTHING.

A4GL_push_long and A4GL_push_int were still keyword-bound to constants, but
generated code pushes variables far more often. Retired the same way
A4GL_push_char and A4GL_push_variable already had been, so they resolve
through the runner's call table.

An unspecified array bound is held as -1, which was multiplied into a negative
total_size - and that is what the runner mallocs for a static, so loading any
module with a "char x[]=..." aborted. Sizes are now taken from the initialiser
where one is given.

On the generator side, several things emitted only for the C backend are not
parseable by p-code's cut down C grammar, and are now suppressed when
A4GL_doing_pcode(): forward declarations for functions, reports and MAIN;
the version helper (which lacks the A4GL_FUNCTION marker p-code requires);
and "(void) x;" between two declarations, which p-code's C89 grammar rejects.
_objData is emitted before the locals for the same reason. ERRCHK now passes
the module name as a string literal, which is what the grammar wants.
MARK_SCOPE_MODULE, which p-code switched on, prefixed declarations but not
uses, emitting "long L_total;" alongside "total=0;" in the same function.

Suppressing the MAIN prototype means an ordinary "MAIN ... END MAIN" program
now compiles to p-code, removing the constraint documented in
compilers/pcode/README.

TUI link
--------
m4/aubit_ui.m4 substituted CURSES_LIB_NAME="-lncurses". The old configure.in
set "-lform -lpanel -lncurses", and ui_curses/Makefile.in still expands
@CURSES_LIB_NAME@, so the rewrite alone dropped panel and form and left
libUI_TUI.so with an undefined update_panels - the TUI driver could not load,
which is why so much had to be run in CONSOLE mode. configure now detects
panel and form and lists them dependents-first.

Testing
-------
aubit4gltest, all 21 ranges, -esqli -tui, against a build of r12988 (the last
revision before the configure rewrite):

r12988 1098 run, 59 skipped, 35 failed
this 1098 run, 59 skipped, 33 failed

The two differences are tests 706 and 1413. Run individually, 706 passes on
both (it is flaky in batch), and 1413 - logical report layout - fails on
r12988 and passes here. No regressions.

For reference, with the TUI driver broken these runs managed 283 tests with
449 skipped and five ranges hanging on menu tests looping on EOF.

Build warning cleanup, object system fixes, embedded Python

Three strands of work.

Build warnings
--------------
Top-level build warnings are down from roughly 2700 to roughly 230, and
nothing fatal to GCC 14/15 remains -- those promote
-Wimplicit-function-declaration, -Wincompatible-pointer-types,
-Wint-conversion and -Wimplicit-int from warnings to errors, which is what
the Ubuntu 26.04 build reports were hitting.

Mostly noise (sign-compare, unused-but-set, missing prototypes, dead
declarations), but some real bugs came out of it, including several places
passing a long* where an int* was expected. On LP64 that leaves the top
four bytes of an XDR *_len uninitialised; 4GL 'integer' maps to C 'long'
while the XDR members are u_int, so it turned up five separate times.

Also added svn:keywords=Id to the modules that had lost it.

4GL object system
-----------------
Inheritance was broken in a way that made most of it unusable, plus two
older bugs found while testing the fix:

* objData->base.objectid is not a 4GL variable, so the root-scanning
collector could not see it and disposed of a parent while a live child
still pointed at it. The first inherited method call after that failed.
Objects now hold a counted reference on their parent: refCnt counts
references held by other objects, the generated constructor/destructor
inc/dec it, and the collector only disposes when no 4GL variable refers
to an object AND its refCnt is 0. Fixing this fixed inheritance, castTo
and polymorphism together.

* Destructors are registered as "<type>.~" but A4GL_destroy_object looked
for "<type>.-", so it had always been a silent no-op.

* Class methods pushed a call frame on entry but only popped it on an
explicit RETURN -- printPopFunction was commented out in the class
epilogue. Every call to a method that fell off the end leaked a frame,
giving "Function calls too deep" after ~2000 calls. Plain functions were
always correct; this was specific to classes.

* castTo could not be reached from 4GL at all. The lookup is a plain
strcmp and the compiler lowercases method names, but only ":X.castTo"
was registered, so "let r = s.castTo(...)" fell through to
RouteToParent and walked off the top of the chain. There are now two
registrations with different calling conventions: ".castTo" returns the
object id (what getObject needs for an implicit upcast like "let f = m")
and ".castto" pushes it (the normal 4GL convention).

* Polymorphic overloads resolved backwards. getSigForTopOfStack built the
signature from the top of the stack, which is the LAST argument, while
the compiler registers signatures in declaration order -- so
f(integer,char) dispatched to f(char,integer). Only visible with mixed
parameter types, which is why the existing sample never showed it.

New samples in tools/test/OO exercising a three-level chain (shape <-
rect <- square): method resolution, castTo, refcounts and object lifetime
under churn. tools/test/OO/README updated.

Embedded Python
---------------
Optional object(python), implemented in C but registered like a 4GL class,
so it is used the same way:

define p object(python)
let p = python.new()
call p.import("math")
display p.call("math.pow", 2, 10)

Methods: new, import, addpath, available, run, eval, call, set, get,
lasterror, version. Each object gets its own namespace. import/addpath
exist so module names and paths are passed as values rather than pasted
into Python source; available() reports whether a package could be
imported without importing it and without setting the error status, so a
Python package can be an optional dependency of a 4GL program.

configure detects an embeddable Python (pkg-config python3-embed, then
python3-config --embed) and link-tests it before enabling anything:

./configure --with-python / --without-python / (auto by default)

With no python3-dev the implementation compiles away, nothing links
against libpython, and object(python) reports that support was not built
in. Python's headers go on the include path for python.c alone, since it
ships object.h, token.h and compile.h which would otherwise shadow ours.

Documented in docs/README-Python.txt, example in
tools/test/OO/test_python.4gl.

Verified: full build clean in both configurations, tools/test builds, and
the OO samples pass.

Fix the real bugs behind the remaining compiler warnings

These are the ones set aside earlier as "the interesting shortlist" - every
warning here was pointing at code that does not do what it was written to do.
A full clean build stays at exit 0 and drops from 754 to 721 warnings.

Memory:

- compilers/4glc/parsehelp.c: malloc(sizeof(expr_str)) followed by memcpy of
sizeof(l->list.list_val[a]) - the size of the *pointer*. It copied 8 bytes
and left the rest of the new expr_str uninitialised.
- lib/libaubit4gl/sql_common.c: A4GL_free_prepare() called
blank_any_cursors_using(sid) after free(sid). That function only compares the
pointer, so it now runs before the free - otherwise any cursor still holding
the sid keeps a dangling statement pointer.
- lib/libsql/odbc/sqlex.c: A4GL_debug("-%p", ptr) after free(ptr). Reading the
pointer value after the free is undefined even just to print it; logs first.
- lib/liblogical/processor/process_report.c: fclose() on a stream from popen().
Now pclose(), so the child is reaped and its status collected. The fwrite in
the copy loop was also inside an assert(), so a build with NDEBUG would have
dropped the copy entirely - it is now a separate statement.
- compilers/4glc/compile.c: two sprintf calls passing incl_path as both the
destination and the first argument. Overlapping source and destination is
undefined; they now format the suffix into a temporary and strcat it.

Logic that never ran, or always ran:

- compilers/4glc/lint.c: "whencode & 0x15 == WHEN_CALL" - 0x15 masks off bit 1
and WHEN_CALL is 2, so the WHENEVER ERROR CALL branch could never be taken and
lint never checked the named function existed. The action is the low nibble
(php.c uses & 0xf for the action, >> 4 for the condition), so it is now 0xf.
- compilers/4glc/lint.c: system_function_dtype() fell off the end without
returning. Callers test for -2, and the two other copies of this function
(calltree.c, prototypes.c) both end with return -2 - this one now does too.
- compilers/fcompile/dump_scr.c: get_attr_from_field() fell off the end when it
found nothing, returning whatever was in the register. Returns 0, which is
what its own "not applicable" path returns and what callers test for.
- lib/libaubit4gl/function_call_stack.c: "moduleName == '\0'" compared the
pointer against NULL, but its unset value is "" - so the no-module branch
never fired. Tests the string now.
- lib/libui/ui_xml/uilib/uilib.c and ui_json/uilib/uilib.c: "rval < 0" where
rval is the size_t returned by iconv(), which reports failure as (size_t)-1.
Conversion errors were silently ignored in both copies.
- compilers/4glc/variables_new.c: "if (idtype!=idtype)" - a self comparison, so
dead. What it meant is not recoverable; removed with a note.
- 16 -Waddress cases across sqlexpr.c, report.c, stack.c, mod.c, json.c, xml.c
and two generic_ui.c copies: tests on the address of a struct's char array,
which is never NULL. Each had a real test beside it (strlen, [0], or a
function call), so the redundant half is gone and the meaning is unchanged.

Format strings:

- compilers/4glc/prototypes.c: three fprintf calls passing arguments to format
strings with no conversions at all.
- tools/asql/parse.l: "%d" given strlen() and sizeof() results, ie size_t.

Left alone deliberately: compile_c.c's "arr_subscripts_len >= 0" was always true
on an unsigned member, but changing it to "> 0" would alter what happens for an
empty subscript list, so only the dead half was dropped.

Drop the Glade IDE from the default build

glade/ide and glade/ide/sqleditor are no longer built by "make". They have not
been used in years, and keeping them in the default target means everyone pays
for them: the sqleditor sources are a 2017 fork of the tools/asql ones that has
since drifted, and their generated code calls aclfgl_* functions in sibling
modules with no prototypes in scope - 32 diagnostics that GCC 14 and later
treat as errors rather than warnings. Fixing those properly means working out
why tools/asql does not have the same problem, which is not worth doing for a
component nobody runs.

Only the ALL list changes. The "glade" target itself is untouched, so "make
glade" still builds it for anyone who wants it, and "make clean" still cleans
it. Nothing else references it.

With this, a full clean build of the default target is free of every diagnostic
GCC 14+ turns into an error - in the compiler output and in the .warn files
4glpc writes, which is where the last few rounds of these were hiding. 49 .warn
files, none with a fatal diagnostic; build exits 0 at 754 warnings; asql,
adbload2 and the compiled 4gl test programs all still run.

Clear the GCC 14+ errors from the rest of corecompile: adbload2 and runforms

Continues r13011. The .warn files that 4glpc writes - which never reach the
build log - had the same class of problem in the two directories corecompile
reaches after tools/asql. Both are now free of diagnostics that GCC 14+ treats
as errors; a full clean build stays at exit 0.

tools/adbload2 (22 down to 0):

- process.ec and jtypes.c call runtime entry points with no prototype in scope.
Declared in a4gl_dbload_int.h rather than by including a4gl_libaubit4gl.h,
because that header defines A4GL_assertion as a two argument macro and
jtypes.c calls it with parenthesised arguments the preprocessor miscounts.
The block is guarded on _AUBIT_LIB_INCL_EXT_ so the translation units that do
include the header keep using its versions and do not get redundant
declarations.
- Forward declarations for the functions defined further down process.ec, and
for those in jtypes.c, which is #included near the bottom of it - so a
declaration in jtypes.c itself comes too late for callers above.
- Declaring A4GL_assertion turned up three calls that never matched it: one
passing no message at all, and two passing a printf-style format plus an
argument to a function that takes (int, char *) and does no formatting. They
now build the message with snprintf first.
- Three A4GL_assertion calls passed dbload->f->delim, a char *, as the int
condition; they mean "is it set", so they now say != NULL.

tools/asql/runforms (3 down to 0):

- A4GL_isnull, A4GL_read_columns and aclfgl_a4gl_get_info were declared only in
a4gl_libaubit4gl.h, which generated 4gl code does not include - it takes the
a4gl_incl_4glhdr.h path. Moved to a4gl_incl_4gldef.h, which is where CLAUDE.md
says shared declarations belong. Signatures checked against stack.c,
sql_common.c and helper.c rather than copied from the header.
- dbi.4gl passes &lv_dtype and &lv_size to A4GL_read_columns, which writes ints,
but 4gl integers are longs: the top four bytes were left uninitialised on
LP64. Same bug and same fix as the sqgetdbs calls in r13011.

Also in lib/libaubit4gl/ui.c: A4GL_open_gui_form declared its last parameter as
"void (*handler_c (int a, int b))", ie a function returning void *, while every
caller passes int (*)(void *, void *) and the function it forwards to,
A4GL_open_gui_form_internal, takes a plain void * - as does handler_e beside it.
Made it void *. That declaration moved to a4gl_incl_4gldef.h too, since the
callers are generated 4gl code.

Not done: glade/ide and glade/ide/sqleditor still have 32, mostly generated 4gl
code calling aclfgl_* functions from sibling modules without prototypes. glade
is a separate target from corecompile, built only when enable_minimal is not
set, so it does not block the core build.

One thing tried and reverted: declaring compat.c's bare trim() in
a4gl_libaubit4gl.h. The name is too common - lib/liblogical/txt/process has its
own static trim() - and it broke the build. Callers that want the library one
declare it themselves; there is now a comment in the header saying so.

Fix the GCC 14+ errors in tools/asql, and a real sqgetdbs/long bug with them

Reported building r13010: "Error compiling infx.ec" in tools/asql, with no
detail. The detail is in tools/asql/infx.warn, which 4glpc writes instead of
sending to stdout - that is why none of this showed up in the build log.

Three things in there are warnings under GCC 13 and errors from GCC 14 on:

- jtypes.c calls isxdigit, isspace, isalnum, isdigit, isalpha, ispunct and
tolower with only <unistd.h> included. Added <ctype.h>. There are two copies
of this file, tools/asql and tools/adbload2, and both had it.
- infx.4gl defines execute_select_free() and asql_explain() with no return
type. Both are used as int by their callers (execute.c) and declared int in
generic.c, so they are now written that way. asql_explain also had no return
statement at all while execute.c does "raffected=asql_explain(...)", so it
now returns 0 rather than whatever was in the register.
- sqgetdbs() is exported by the ESQL/C library but declared in none of the
shipped headers, so it was an implicit declaration. Added a prototype, the
same one added to lib/libsql/esqlc/esql.ec in r13002.

Declaring sqgetdbs then exposed a genuine bug at three of its four call sites.
It writes an int through its first argument, but they pass &ndbs where ndbs is
a 4gl integer, ie a long: on LP64 the top four bytes are left uninitialised and
the database count comes back wrong. The fourth call site already worked around
this with an int temporary; the other three now do the same.

Verified by rebuilding tools/asql: infx.warn goes from 10 GCC-14-fatal
diagnostics to 0, and a full clean build still passes.

Note for whoever picks this up next: a sweep of the .warn files this build
writes shows the same class of problem still present in tools/adbload2 (22),
glade/ide (about 26 across several files) and tools/asql/runforms (3) - all of
them functions called without a visible prototype. They are next in line to
stop a GCC 15 build.