Aubit 4GL SVN-Code

lex_cs/runtime: the 4GL builtins the generator emits unqualified

Running a large real 4GL application through the C# generator turned up
33 names the generated code calls that nothing defined. This adds them,
in a new lib/liblex/lex_cs/runtime - beside the generator that emits
them rather than off in the runtime tree, since the two have to agree.

They compile into Aubit4GL.Runtime rather than forming an assembly of
their own, and FglModule is now partial to allow it. That is forced
rather than chosen: generated code calls these unqualified, so they
have to be members of the class it inherits, and a separate assembly
could not provide them.

FglStrings GetSubstring, WriteSubstring, SwitchTrim
FglEnvironment FGL_GETENV, NUM_ARGS, ARG_VAL, RUN, ERRORLOG, and the
file helpers
FglProcess getuser, get_pid, get_envvar, err_get, lastkeypress

The substring pair carries the semantics that are easy to get wrong:
4GL substrings are 1-based and inclusive at both ends, a range running
off the end returns what there is rather than failing, and a write pads
the target with blanks and pads or truncates the value to the width of
the range, so the string's length never changes. Those rules came from
the older C# runtime in cs/, which had them right; they are not
invented here.

The client/server call layer - create_named_socket, connect_to_server,
push_server_call_arg and the rest - is deliberately NOT implemented.
It is the client half of a socket protocol whose server side is not in
this tree, so it cannot be written blind, and a stub that returned
success would give a program that appears to run and silently never
talks to its server. They throw with an explanation instead.

Also fixes ASqlReportError, which was mine: r13063 retargeted
IfxReportError to ASqlSetError everywhere except the report path, which
kept emitting a name nothing defines. 93 of the errors were that.

And the end-to-end probe now builds the runtime once before its loop.
Without that the first program in it absorbs a cold restore and can be
reported as "the generated C# did not build" - a failure that has
nothing to do with the program and disappears on the next run. It cost
me a while to trust it was not a real regression, which is exactly the
kind of doubt a test suite must not create.

Fidelity: 27 semantics, 9 UI messages, 30 USING, the 66-line report and
6 end-to-end programs all still agree with C, from a cold build cache.
203 runtime tests pass.

dotnet: fglproto takes module names, and it is already documented

I had the README telling people to pass .dat files and to filter the
.glb out of the glob. That works, but it is not the documented form and
it is more awkward than it needs to be: fglproto takes the module NAME,
appends .dat itself, and reads the module definition the WRITE pass
left. Naming modules rather than files also makes the .glb problem
disappear - a globals file is not a module, so it never appears in the
list in the first place.

Verified both forms produce the same prototypes.unl, on a two-module
toy and on the eight-module d4 program (45 prototypes either way).

Also says where this comes from, since it is reasonable to wonder. The
sequence is not something the C# back end invented: docs/web_services.txt
has documented fglproto since long before there was one - it is how
exported function stubs are generated for the SOAP support - and it
gives the same two steps, 4glpc -t WRITE per module then fglproto over
them all. The C# back end just consumes the same prototypes.unl.

The end-to-end probe was already using the documented form; only the
README prose was off.

dotnet: answer the fglproto question in the README

Records what the prototype pass is actually for, since it is the first
thing anyone hits: for a single module it is redundant - the generator
infers return types from the bodies it can see, and the prototypes only
save a defensive As* conversion - but for anything multi-module it is
required, and a function returning several values is the case that
cannot be guessed at all.

Also adds the two things the new arithmetic probe turned up to the list
of mistakes the probes caught, and records DISPLAY of an unassigned
expression as a known gap rather than leaving it implied by what the
probe does not test.

dotnet: 4GL division does not truncate

Found while answering whether the C# backend still needs fglproto: a
cross-module test came out with c=3.00 where the C runtime said 3.50.

4GL division never truncates. 7/2 is 3.5 whatever the operands are
declared as, and it is the assignment afterwards that rounds or
truncates - LET i = 7/2 gives 3 because i is an INTEGER, not because
the division did anything. Checked against the C runtime:

dec 7/2 = 3.5000
flt 7/2 = 3.50
int 7/2 = 3
lit 7/2 = 3.5000
neg -7/2 = -3.5000

The generator emitted a bare C# "/", which truncates at the division
itself when both sides are integers, so a DECIMAL target held 3.00.
Division now goes through Fgl.Divide, which promotes first. Division by
zero returns NULL rather than throwing, as the C runtime does - a 4GL
program tests the result, it does not catch anything.

Two things fell out of fixing it:

assign_wrapper had no case for FLOAT or SMALLFLOAT, so assigning any
decimal-valued expression to one would not compile. It has one now,
the same shape as the CHAR, DECIMAL and SMALLINT cases.

DISPLAY of a MONEY lost its currency symbol. MONEY and DECIMAL are
both decimal? at runtime - that is deliberate, the difference is
presentational - so the type has to be named where it is still known,
and the generator now emits Fgl.DispMoney for it. The symbol comes
from DBMONEY, which can also place it after the number and change the
decimal separator, and it is extra to the declared width: a
MONEY(10,2) shows in 13 columns where a DECIMAL(10,2) shows in 12.

New end-to-end probe arith.4gl covers all of it: division into DECIMAL,
FLOAT, INTEGER and SMALLINT, negative and literal division, MONEY, and
the other three operators plus MOD.

Every result in it is assigned to a declared variable before being
displayed, deliberately. DISPLAYing a raw expression is a separate
question - an unassigned division has no declared scale, and 4GL prints
it at DECIMAL's maximum width - and that gap is recorded in the README
rather than quietly folded into this probe.

Fidelity: 27 semantics, 9 UI messages, 30 USING, the 66-line report and
6 end-to-end programs all agree with C. 203 runtime tests pass, d4 still
builds clean, and the standalone corpus is unchanged at 30 of 33.

etc/containers: check the installed tree, not just the build

Every install bug of the last few days - the missing aubitrc, the
generated headers that were never installed, libSQL_pg8 built and then
left behind - had the same shape: make install reported success and
produced a tree nothing could be built against. None of it is visible
to anyone working in the source tree, which is where we all work.

So the container check now has two phases:

in-tree compile and run a 4GL program from the source tree
installed make install to a prefix, verify what actually landed,
then compile and run again with ONLY the installed tree
on PATH and LD_LIBRARY_PATH

The second phase does not trust the exit status. It checks that 4glc,
aubit-config, libaubit4gl, the public headers, json.h, at least eight
generated a4gl_API_*.h and an aubitrc are all present, and that every
libSQL_* plug-in that was built was also installed. Then it runs
aubit-config, which is the first thing to notice a config it cannot
read, and finally compiles and runs a program with a REPORT and checks
the output is right rather than merely that the compiler exited 0.

The container-side half moved into its own checks.sh, since it had
outgrown being a string inside the run command.

Both negative-tested by reintroducing a bug in a scratch export and
confirming the harness fails, by name and before the compile:

dropping json.h from the header list
MISSING: incl/json.h (exit 1)

dropping SQL_pg8 from the install list
BUILT BUT NOT INSTALLED: libSQL_pg8.so (exit 1)

In both, the in-tree phase still passed - which is the whole point.

Clean tree passes on both images.

configure: make an installed tree usable

Compiling a 4GL program against a --prefix install failed three times
over. Each fault hid the next, so they only came out one at a time.

1. The runtime could not find its own aubitrc.

resource.c has had an #ifdef AUBITETC for this all along, but nothing
ever defined it, so it fell back to a hardcoded /etc/opt/aubit4gl -
correct only when the prefix is the default. configure computes
AUBITETC as $sysconfdir/aubit4gl and installs there, so with
--prefix=/opt/aubit4gl the file went to
/opt/aubit4gl/etc/aubit4gl/aubitrc while every tool looked in
/etc/opt/aubit4gl/aubitrc and reported

*********** ERROR IN CONFIGURATION ***********
No aubitrc file has been found on your system

which surfaced as "ERROR: no TARGET_OS STOP." from bin/aubit, because
TARGET_OS comes from aubit-config, which could not read its config.
AC_DEFINE_UNQUOTED now tells the C code what configure chose.

2. None of the a4gl_API_*.h headers were installed.

The list filters each header on existence, so that one since removed
cannot break the install. That is right for the hand-written ones and
wrong for these: make generates them from the .spec files, long after
configure has run, so the test always failed and quietly dropped all
eight. Since a4gl_4gl_callable.h - which IS installed - includes
a4gl_API_exreport.h, the first compile against the install died with
"a4gl_API_exreport.h: No such file or directory". They are now listed
unconditionally; the filter stays on the headers it makes sense for.

3. json.h was not installed either, though a4gl_4gl_callable.h includes
it. It is as public as the rest of them, whatever its name suggests.
Found by copying only the listed headers into an empty directory and
compiling generated output against it until it stopped complaining -
json.h was the only one left, the rest of the include closure being
behind ifdefs that are not taken.

Verified on AlmaLinux 8: configure, make and make install with
--prefix=/opt/aubit4gl, then from the installed tree alone -
aubit-config answers TARGET_OS and A4GL_SQLTYPE, 18 headers are in
place including all eight generated ones, both PostgreSQL drivers are
present, and a program with a REPORT compiles and runs and prints what
it should. The source-tree path is unchanged and the fidelity probes
still pass.

configure: install libSQL_pg8, and make it the PostgreSQL default

Reported: a compile against an installed tree dies with

Error: can't open DLL /opt/aubit4gl/plugins-1.8.1/libSQL_pg8.so - STOP
Error msg: ... cannot open shared object file: No such file or directory

Reproduced in a container on AlmaLinux 8 with postgresql-devel: the
plug-in builds, and then is not installed. lib/libsql/postgresql builds
both libSQL_pg and libSQL_pg8, but only SQL_pg was in ALL_PLUGINS,
which is what drives the install. So libSQL_pg8.so sat in the source
tree and never reached the install, and a program that asked for it got
the error above from an otherwise complete installation.

The default was wrong as well, and more seriously.

libSQL_pg and libSQL_pg8 are not two versions of the same driver.
libSQL_pg is built from simple.c - "This simple PG plug-in with limited
functionality", as its own Makefile comment says - and it has no
cursor, scroll or transaction support at all: 22 entry points against
pg8.c's 78, and not one mention of CURSOR, SCROLL or COMMIT in 1074
lines. A 4GL program that does a FOREACH or a BEGIN WORK cannot run on
it. libSQL_pg8 is the real driver, at 4616 lines.

configure was defaulting A4GL_SQLTYPE to pg, so a PostgreSQL user got
the limited driver by default and could not select the other one,
because it was not installed.

Now: both are installed, both are listed in A4GL_SQLTYPE_DETECTED, and
the default is pg8. Both still sort below Informix, so a box with both
still picks esql as it did.

Verified on Alma 8: configure reports "SQL backend: pg8", make install
puts libSQL_pg.so and libSQL_pg8.so in the plug-in directory, and the
installed aubitrc carries A4GL_SQLTYPE=pg8 with
A4GL_SQLTYPE_DETECTED="nosql pg pg8".

common/dataio: drop the dead menu_x rpcgen rules

You were right, and it goes further than the rpcgen path: nothing in
the default build consumes common/dataio/menu_x.x at all.

- menu_x_xdr.o was a prerequisite of nothing. The four XDRPACKER
plug-ins are struct_form, report, module and module_definition.
- lib/libmenu is not wired into lib/Makefile, so no libMENU_* is
built - configure already says so in a comment, and aubitrc reports
A4GL_MENUTYPE_DETECTED="NOMENU GENERIC" with no XDR in the list.
- The one place that could still have wanted the rpcgen header,
compilers/menus/a4gl_mcompile_int.h, takes the xgen branch:
_USE_RPCGEN_HEADERS_ is not defined anywhere in the tree.
- compilers/menus/Makefile had already commented its own rpcgen rules
out, with "Should be obsolete - using xgen instead of rpcgen".
- Even the xgen side is unused here: libDATA_any is the only thing
that wants menu_x.xio.o, and it is commented out of ALL. An Alma
build produces libDATA_struct_form, _report, _module and
_module_definition, and no menu_x object of any kind.

So the RPCGEN_MENU definitions and the menu_x_xdr.c and menu_x_xdr.o
rules are gone, including the fallback line I added to them in r13066 -
which never fired, since nothing asked for the file.

One distinction worth keeping straight, because the two files have the
same name: compilers/menus/menu_x.x IS live. xgen turns it into
menu_x.xs.h, which mcompile includes, and bin/mcompile is built. Only
the common/dataio copy is dead. Nothing has been removed from
compilers/menus.

tools/cygwin/menu_x.h and menu_x_xdr.c are now referenced by nothing.
They are still the odd ones out there - hand-adapted rather than
rpcgen output - and deleting versioned files is your call, so they are
left alone; refresh.norpcgen does not touch them either way.

Verified: common/dataio and compilers/menus both still build, a clean
Alma 8 container build with no rpcgen still configures, makes and runs
a compiled 4GL program, and the fidelity probes still pass.

Makefile: make refresh.norpcgen work again, and refresh what had drifted

The target was there, as you said. It had not copied anything in a very
long time, for two reasons:

if test "$x" = "1"

"$x" is expanded by make, not by the shell, so the shell saw
test "" = "1" and the branch never ran. It wanted "$$x" - which
refresh.noyacc, twenty lines below, gets right.

The sense was inverted anyway. diff -q exits 0 when two files are the
SAME, so

if diff -q common/dataio/form_x.h tools/cygwin/form_x.h ; then x=1; fi

set "needs updating" precisely when they already matched, and left it
clear when they had diverged. Both bugs had to be fixed for either to
show.

Rewritten as a loop over the file list, so a new .x file is one word to
add rather than three lines, and it now says which files it copied.

Then run, which refreshed three files that had drifted from the .x
sources:

form_x.h FA_S_ALLOWEDCHARS, an enum used by lib/libui/ui_curses
fgl.h the s_expr_json type and a struct's config field
fgl_xdr.c the matching xdr_s_expr_json serialiser

All additions; nothing was removed. report.h, report_xdr.c, npcode.h
and npcode_xdr.c were already current.

menu_x stays out of the list, as it always was. Unlike the other four,
tools/cygwin/menu_x.h is not rpcgen output - it is hand-adapted to need
no RPC headers, declaring bool_t itself rather than including
<rpc/rpc.h>. Copying rpcgen's version over it would throw that away.
That is now written down in the target rather than being implicit in
the omission.

Verified: a second run reports everything up to date; the fidelity
probes still pass here; and the no-rpcgen Alma 8 container build still
configures, makes and runs a compiled 4GL program against the refreshed
files - which is the case these copies exist for.

etc/containers: build on Alma 8 without needing an Alma 8

We have been fixing RHEL-family build failures blind, one report at a
time, because they do not happen on a current Debian. Three in a row now
- --with-pg-all being silently ignored, configure dying with no message
where there is no controlling terminal, and the missing -ldl - were all
invisible here and obvious there.

./etc/containers/build-in-container.sh # alma8, no rpcgen
./etc/containers/build-in-container.sh alma8-rpcgen # with rpcgen
./etc/containers/build-in-container.sh alma8 --with-mysql

It exports the tree rather than mounting the working copy, so a
generated file or an object file from a host build cannot mask a
missing dependency - which is the entire point of building elsewhere.
Then it configures, makes, and compiles and runs a 4GL program, so a
green result means the toolchain works rather than merely that it
linked. Uses podman when it is there and docker otherwise; podman needs
no daemon and no group membership, so it works without setup.

Alma 8 is worth the trouble because of how it differs:

glibc 2.28 libdl.so.2 is still a separate object
gcc 8.5 stricter in places than what we develop against
autoconf 2.69 older than configure.ac wants, so the image installs
no autoconf at all and builds with the configure that
ships in the tree - what a release user actually does
rpcgen in PowerTools, off by default; the plain image leaves
it out so the pre-generated files get exercised, and
the -rpcgen image puts it back to cover the other path

Both images verified: configure, make and the smoke test pass on each.