#41 XML GENERATE

[Edward Hart]

I've spent the last week hacking at an implementation of IBM XML GENERATE. Attached is an initial, very simple patch for it. It has many, many problems:

• Field data. Currently, the content of the XML is just copied from a cob_field's data pointer - with no regard for its size. To get the correct output, we obviously need to provide the entire cob_field and output a string of the correct length. This will require moving initialisation of the cob_xml_tree's to after all the fields are allocated.
• Four stages of data wrangling. Currently, the flow of the XML to generate looks like (record to generate from + XML GENERATE clauses) -> (record structure annotated with XML details) -> (as before, but with the tree flattened into a list and XML elements/attributes/contents separated) -> (cob_xml_tree and cob_xml_attr structs in generated C code). I would like to reduce the two middle steps to one step.
• Scanner hacks. WHEN and SUPPRESS act like statements in the parser: if the parser is parsing a statement and sees one of them, it assumes it has reached a new statement. This is bad: XML GENERATE has WHEN and SUPPRESS as words in clauses. The current, ugly workaround is to add new tokens WHEN_XML and SUPPRESS_XML which don't behave like statements. They'll do.
• Reserved words. I haven't looked at what kind of reserved words XML GENERATE (and XML PARSE) need yet, so some words which should be context-sensitive aren't context-sensitive.
• Exception handling. I haven't tested, so almost certainly doesn't work.
• Code quality.* I have unashamedly hacked at things until they compiled and ran without segfaulting, so a lot of functions need refactoring.
• Syntax checks. Most are missing.
• Output field selection. I haven't implemented the rules on what records are output as XML and which ones aren't.
• SUPPRESS. Not yet implemented for content and attribute elements.
• libxml2 detection. I'm not sure if my changes to configure.ac are enough to reliably detect libxml2.
• Tests. None yet.
• ENCODING. Not yet handled.
• OCCURS fields. Currently codegen and libcob do not have any tools to iterate over an entire table. Ideally, we should be able to extend cob_field to indicate that it contains multiple data fields (corresponding to each subscripted entry). However this problem is solved, what will quickly follow will be a fix for [bugs:#35] and an implementation of the (ALL) subscript.

Example usage:

:::cobolfree
       >>SOURCE FREE
IDENTIFICATION DIVISION.
PROGRAM-ID. prog.

DATA DIVISION.
WORKING-STORAGE SECTION.
01  x PIC X(200).
01  y.
    03  z PIC X(15) VALUE "hello, world!".
    03  az PIC X(15) VALUE "goodbye, world!".
    03  ab.
        05  abc PIC x(3) value spaces.

PROCEDURE DIVISION.
    XML GENERATE x
        FROM y
        WITH XML-DECLARATION
        NAME OF abc IS "ABCDEF", z IS "zeta"
        TYPE OF z IS ATTRIBUTE
        SUPPRESS WHEN SPACES

    DISPLAY FUNCTION TRIM(x)
    .

Output:

:::xml
<?xml version="1.0"?>
<y zeta="hello, world!  goodbye, world!   "><az>goodbye, world!   </az><ab/></y>

What it should output:

:::xml
<?xml version="1.0"?>
<y zeta="hello, world!"><az>goodbye, world!</az></y>

[Edward Hart]

This update has field output of the right length and resets SUPPRESS conditions between XML GENERATE's. This patch also includes xml.c, which I forgot from the first.

Example:

:::cobolfree
    XML GENERATE x
        FROM y
        WITH XML-DECLARATION
        NAME OF abc IS "ABCDEF", z IS "zeta"
        TYPE OF z IS ATTRIBUTE
        SUPPRESS WHEN SPACES

    DISPLAY "[" FUNCTION TRIM(x) "]"

    XML GENERATE x FROM abc, NAME OF abc IS "sfdasd"
    DISPLAY "[" FUNCTION TRIM(x) "]"

    XML GENERATE x FROM ab, WITH ATTRIBUTES.
    DISPLAY "[" FUNCTION TRIM(x) "]"

Improved, but still incorrect, output:

[<?xml version="1.0"?>
<y zeta="hello, world!  "><az>goodbye, world!</az><ab/></y>
]
[<sfdasd>   </sfdasd>
]
[<ab abc="   "/>
]

[Edward Hart]

Version 3 has a fully functional SUPPRESS clause, correct trimming of alphanumeric data, reserved word fixes and tests.

[Edward Hart]

Version 4 adds the XML-CODE register, detection of invalid XML characters in COBOL names or COBOL records, and the ignoring of subrecords whch are FILLER, REDEFINES or RENAMES.

[Edward Hart]

The patch is nearly complete. Version 5 adds syntax checks, config options and completes the exception handling test.

[Simon Sobisch]

Hi Edward,

thank you for the patch. Some first notes:

• the "very simple" part in the description seems to not apply any more :-)
• the scanner hack is no issue but the correct thing to do
• the new file testsuite.src/run_xml.at is missing in the patch
• Does ACUCOBOL really support the IBM extension but not the "xml-generate-extra-phrases"?
• Changelog / NEWS entry is missing in the patch
• configure.ac:
  • wouldn't it be better to name the new option --with-xml2?
  • if libxml2 is not used we should include a warning at the end
  • it would be good to allow libxml2 to be used even if xml2-config is not available, first a fall-back to pkg-config --cflags libxml-2.0 and in general testing if we can compile a minimal sample and use the library (if neither xml2-config nor pkg-config provide something we leave it to the user to specify necessary C[PP]FLAGS) - on at least one of my testing machines xml2-config is not available and there is no pkg-config entry for libxml (no version) but library and headers are available
  • the ideal test case and nice in general: output xmlversion.h info and compare it with the one provided in libxml2
• if possible: please add libxml2 use/version output in both print_info() functions
• please commit the formatting-only issues (the changes are fine) directly to trunk with a seperate commit, removing the parts from the patch

Simon

[Edward Hart]

the new file testsuite.src/run_xml.at is missing in the patch

Whoops. It's included in the attached updated patch.

Does ACUCOBOL really support the IBM extension but not the "xml-generate-extra-phrases"?

Yes.

wouldn't it be better to name the new option --with-xml2?

Done.

it would be good to allow libxml2 to be used even if xml2-config is not available, first a fall-back to pkg-config --cflags libxml-2.0 and in general testing if we can compile a minimal sample and use the library ...

I've added a check for pkg-config and a default case where we check for the headers and -lxml2.

the ideal test case and nice in general: output xmlversion.h info and compare it with the one provided in libxml2

libxml2 comes with a LIBXML_TEST_VERSION macro which I think does something like that.

[Simon Sobisch]

Thank you. configure.ac contains a copy&paste issue "xml library is required as -ldb". For AC_MSG_NOTICE([ Use libxml2 for XML I/O yes]) we should add an else (or directly output $COB_HAS_XML2). Similar copy&paste issue var_print (_("ISAM handler"), "libxml2",.

syn_misc.at contains AT_SKIP_IF([test "$COB_HAS_XML2" = "no"]), shouldn't this be necessary only for run_xml.at?

As XML PARSE doesn't have much clauses, can you please add it to parser.y with a PENDING notice?

I'd like to compile and run the tests next week on some machines to verify that this will work on more envrionments, afterwards I'd say it is time to commit it. What do you think about this?

[Edward Hart]

Version 7 fixes the mistakes in copying-and-pasting, fixes memory leaks and includes a refactoring of all the cobc changes.

All that remains (for me, at least, to do):

• add errors when using currently unimplemented OCCURS and floating-point items.
• test the changes with Visual Studio 2017 (and then update build_windows.h.in)
• add XML PARSE to parser.y.
• add a NEWS entry.

[Edward Hart]

Version 8 includes changes for Visual C++: replaced free() in xml.c with xmlFree() (fixing a heap exception which did not occur on Ubuntu) and updated build_windows/config.h.in and build_windows/README.txt. NB: This patch includes some unintentional changes which will be removed in version 9 (e.g. changing VBISAM to BDB in config.h.in and changes in the .sln files); they are only included because I can't be bothered to remove them tonight.

[Edward Hart]

Committed in [r2688].

[Simon Sobisch]

Thank you very much for this marvelous patch.

It inspired me to go a little bit deeper on some details (mainly build envrionment), so I took the time to do some minor code adjustments (minimal URI check without lib2xml, compiler warnings), build_windows adjustments and tweaked configure.

The last one actually brings in some changes:

• XML runtime support is not "default" like ISAM -> it will be enabled if possible but otherwise auto-disabled (--with-xml will enforce it and abort if not possible to compile+link it) ; the reasons for this:
  • it is (currently) an extension, not a "standard" feature like ISAM
  • the necessary header and library are not available on a lot of machines where the build worked before
• I've included pkg-config via m4 macro, this brings the benefits that it is more compatible for cross-compilation this way, we get two nice variables to override the x_CFLAGS and x_LIBS (both specified = pkg-config not called at all)
• pkg-config gets priority over xml2-config as it is available on more machines and will be used for other libraries in the future (it doesn't work fine for curses, bdb or vbisam :-( )
• I've manually added the x_CFLAGS/ x_LIBS to the xml2-config part
• in each case there is a check if compile+link works (not only in the "no tool to use" case)

I just used "XML2" as internal prefix for pkg-config, but we may change it to XML. Thoughts?

Do you have any plans on working on the following related parts?

• XML PARSE
• XML file encoding (likely comes "free" with the used libraries)
• OCCURS entries as you've outlined above
• the other xml-related registers

[Edward Hart]

Thank you for improving my configure changes.

the necessary header and library are not available on a lot of machines where the build worked before

Which ones? I quick Google search suggests it's available on Solaris, HP-UX and z/OS, which I thought were the most "exotic" systems we support.

I just used "XML2" as internal prefix for pkg-config, but we may change it to XML. Thoughts?

I've only tested my changes with libxml2, so I'd be surprised if it works with libxml1, which, in any case, is no longer widely available. There's no standard XML API, so my changes won't work with other XML libraries either. So I think the prefix XML2 is best.

Do you have any plans on working on the following related parts?

• XML PARSE
• XML file encoding (likely comes "free" with the used libraries)
• OCCURS entries as you've outlined above
• the other xml-related registers

I have no plans currently. I will note that:

• XML PARSE will require me to get familiar with libxml2 parsing's APIs. I'll probably have to get advice from the libxml mailing list before I know where to start.
• I do not want to do file encoding until libcob is Unicode safe, which is a project in itself.
• OCCURS is the most important thing to fix.
• JSON GENERATE will be very easy to implement now.

[Simon Sobisch]

Which ones? I quick Google search suggests it's available on Solaris, HP-UX and z/OS, which I thought were the most "exotic" systems we support.

The headers and libraries are available but may not be pre-installed (it isn't hard to install them, but an additional thing that may also break automated build systems as long as the additional dependency is not in).
And there are cases like the machine I currently ssh to (SLES9):

$ configure --with-xml2

...
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libxml-2.0... no
checking for xml2-config... no
../configure: line 14898: xml2-config: command not found
checking libxml/uri.h usability... no
checking libxml/uri.h presence... no
checking for libxml/uri.h... no
configure: error: Headers for libxml2 missing

... note: I just recognized that ther are some parts to adjust as the changed configure parts don't work on this machine... Will try to fix it this evening... EDIT: sometime this week.

$ configure --with-xml2 CPPFLAGS="-I$HOME/gettext-0.18.3.1/gettext-tools/gnulib-lib" LIBS="$HOME/gettext-0.18.3.1/gettext-tools/gnulib-lib/.libs/libxml_rpl.a" # these are the only paths that contain the headers/library

...
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libxml-2.0... no
checking for xml2-config... no
../configure: line 14898: xml2-config: command not found
checking libxml/uri.h usability... yes
checking libxml/uri.h presence... yes
checking for libxml/uri.h... yes
checking libxml/xmlwriter.h usability... yes
checking libxml/xmlwriter.h presence... yes
checking for libxml/xmlwriter.h... yes
checking for LIBXML_WRITER_ENABLED defined in libxml/xmlwriter.h... yes
checking if linking against libxml2 with "" works... yes
...
configure:  Use libxml2 for XML I/O                      yes

$ make
common.c no definition for access # much more compiler warnings like this follow
*** Warning: Linking the shared library libcob.la against the
*** static library /home/me/gettext-0.18.3.1/gettext-tools/gnulib-lib/.libs/libxml_rpl.a is not portable!

... failure on the final link ...

:-) Don't do stuff like this in production, it may magically link :-)

Solaris:

checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libxml-2.0... yes
checking libxml/uri.h usability... yes
checking libxml/uri.h presence... yes
checking for libxml/uri.h... yes
checking libxml/xmlwriter.h usability... yes
checking libxml/xmlwriter.h presence... yes
checking for libxml/xmlwriter.h... yes
checking for LIBXML_WRITER_ENABLED defined in libxml/xmlwriter.h... yes
checking if linking against libxml2 with "-lxml2  " works... yes

Note: The "XML GENERATE syntax checks" test fails when tree-debug is active, can you inspect this, please?

< prog.cob:147: error: SUPPRESS WHEN SPACE item must be USAGE DISPLAY or NATIONAL
< prog.cob:153: error: SUPPRESS WHEN HIGH-VALUE item must be USAGE DISPLAY or NATIONAL
< prog.cob:153: error: SUPPRESS WHEN HIGH-VALUE item must be an integer
< prog.cob:152: error: SUPPRESS WHEN LOW-VALUE item must be USAGE DISPLAY or NATIONAL
---
> 
> cobc: tree.c: 1046: invalid cast from 'NULL' type None to type XML OUTPUT TREE
> 
> cobc: aborting compile of prog.cob at line 146 (PROGRAM-ID: prog)
> cobc: Please report this!

[Simon Sobisch]

JSON GENERATE will be very easy to implement now.

Hm...
PLEASE add JSON GENERATE and (parsing only) JSON PARSE :-)

As this will get in more "registers" I actually think we also should adjust the register handling within libcob.

What do you think about something like the following?

enum cob_module_register_type {
    COB_REGISTER_CRT_STATUS = 0,
    COB_REGISTER_CURSOR_POS = 1,
    COB_REGISTER_XML_CODE = 2,
    COB_REGISTER_XML_EVENT = 3,
    ...
};

typedef struct __cob_module_register {
    cob_field       *f;         /* Field for the register */
    cob_module_register_type    type;   /* Register type */
} cob_module_register;

And then genereate the field reference in a local table and its pointer in the cob_module structure, instead of the current cob_module->xml_code (similar to what is done with const char **module_sources).
The references would be set only once during cob_module_global_enter() .

Opinions?

[Edward Hart]

Ah, sorry for not replying to this sooner.

I started looking into JSON GENERATE, but immediately ran into the problem of deciding which JSON library to use. It's a choice between

• cJSON
• json-c
• jansson
• JSON-GLib

Of these, JSON-GLib is the only one which is part of a wider project and not just a GitHub repo. But I'm not sure how many dependencies JSON-GLib has. We need something portable, something powerful enough to support JSON PARSE (and maybe even ORGANIZATION JSON) and something which will be maintained for years to come.

Regarding register changes: they'd be fine to implement, but I'm not sure why we need them.

[Brian Tiffin]

I've looked a few times, and cJSON seems like a sane choice. Knows a little bit about UTF-16 encoding, which is a step in the right direction for future NATIONAL UCS-2/other support. cloc's in at half the lines of code as say jansson, and scores 99% on conformance test linked below. But it's not in Debian, that I can tell. Which may mean it's in less major distro channels than some of the other choices.

To help make the decision, this page has conformance and performance benchmarks on most of the C implementations, Edward.

https://github.com/miloyip/nativejson-benchmark