#2094 mingw-dist needs XML validation tools

[Keith Marshall]

It has been informally noted elsewhere; this ticket states it formally.

Although mingw-get itself has no XML validation capability, (beyond requiring that every catalogue file must comprise structurally well-formed XML), it is difficult for developers to ascertain that their catalogues will convey the intended meaning to mingw-get. For example, the current mingw-dist catalogue inventory exhibits the following anomalies:

$ make xml_datarootdir=../../data check
./xmlcheck ./pkgspec.xsd ../../data/*/*.xml
Problem Report:
Document: /home/keith/src/mingw/mingw-get-setup/data/contrib/mingw32-tcltk.xml
20:56: ERROR: no declaration found for element 'license'
20:56: ERROR: attribute 'tarname' is not declared for element 'license'
44:15: ERROR: element 'license' is not allowed for content model '((affiliate|description),(source|licence|component|action))'
55:55: ERROR: no declaration found for element 'license'
55:55: ERROR: attribute 'tarname' is not declared for element 'license'
77:15: ERROR: element 'license' is not allowed for content model '((affiliate|description),(source|licence|component|action))'

Problem Report:
Document: /home/keith/src/mingw/mingw-get-setup/data/mingw32/mingw32-zlib.xml
113:53: ERROR: no declaration found for element 'affilate'
113:53: ERROR: attribute 'group' is not declared for element 'affilate'
123:15: ERROR: element 'affilate' is not allowed for content model '((affiliate|description),(source|licence|component|action))'

Problem Report:
Document: /home/keith/src/mingw/mingw-get-setup/data/msys/msys-gcc.xml
29:54: ERROR: no declaration found for element 'license'
29:54: ERROR: attribute 'tarname' is not declared for element 'license'
47:15: ERROR: element 'license' is not allowed for content model '((affiliate|description),(source|licence|component|action))'

Problem Report:
Document: /home/keith/src/mingw/mingw-get-setup/data/msys/msys-perl.xml
11:7: ERROR: no character data is allowed by content model
make: *** [check] Error 1

It may be noted that these anomalies are predominantly misspellings: "license" where "licence" is correct; "affilate" where it should be "affiliate". The one remaining anomaly is due to extraneous data within the msys-perl.xml file.

The issue is that mingw-get doesn't catch these errors, and they are difficult for developers to spot without appropriate validation tools. The out-of-tree toolkit, which I've used to generate the above report, needs to be integrated into the mingw-dist build infrastructure.

[Keith Marshall]

I've now pushed a formal XML Schema Definition, and source code for a free-standing XML validation program, to the mingw-dist repository, and integrated it via a make check build goal, into the mingw-dist build machinery.

As I've currently integrated it, XML validation is now a mandatory prerequisite for compilation of any mingw-dist catalogue for publication; thus all mingw-dist contributors will now be obliged to install the Xerces-C++ SDK, which is required by the XML validation tool.

Right now, I have not fixed the existing XML anomalies listed on this ticket; they belong to others. However, it will be necessary to fix them before any further catalogues can now be compiled, due to the mandatory validation prerequisite which applies to the entire catalogue corpus. Consequently, I'm leaving this ticket status as "pending" consensus on this discussion topic.

[Earnie Boyd]

Keith I'm getting the below issue with xmlcheck on my system.

xmlcheck.log

Assertion failed: manager != 0, file ../../src/xercesc/util/XMemory.cpp, line 63

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

[Keith Marshall]

That seems to be coming from within the guts of Xerces-C++ itself. If you're just using my prebuild, then maybe there's a compiler version conflict; perhaps you need to build it from source, using the same compiler as used to build xmlcheck.exe

It WJFFM, but my xmlcheck is built for my LMDE host, against the Xerces-C++ SDK as delivered by the LinuxMint folks for that specific host configuration.

[Earnie Boyd]

That didn't help. Make check in the xerces-c++ build directory causes no such issues.

The assertion is happening in the new operator for MemoryManager.

[Keith Marshall]

Hmm. I'm also seeing a crash on Win7 VM, but not even getting as much in the way of diagnostics; just "xmlcheck.exe has stopped working".

Strangely, my Linux build works flawlessly. I can also get a working build, running on Win7, with an earlier transitional source, more closely resembling Boris Kolpackov's original. Something I've done, in an effort to tidy it up, seems to be interfering with proper operation on Win7, (but not on Linux). I'll need to investigate further, but it won't be tonight.

[Keith Marshall]

The perils of hidden non-standard typedefs, where a standard type would have been so much better! Microsoft's proliferation of them is bad enough; Xerces-C++ takes it to new dizzy heights.

Printing XML document line and column numbers needs PRIu64, not simple "%u" or "%d". My 64-bit Linux build will have worked, because an int is 64-bits anyway; not so in Windows.

It should be fixed in the repository, now.

[Earnie Boyd]

I still get the same issue as above.

I think the focus needs to be around MemoryManager; for what ever reason to assertion of the new operator is that the pointer is not zero.

[Earnie Boyd]

I just gave Boris' version a try and get the same assertion issue.

[Earnie Boyd]

Removing the libxerces-c.dll.a library causing the static version to be used resolves the issue.

[Keith Marshall]

I can't reproduce this. I've now been through the painful process of building Xerxes-C-3.1.1 in situ on Win7, (~2hrs vs. ~10mins to cross compile on Linux); I'm still using GCC-4.7.2 FWIW. xmlcheck.cpp now builds, to dynamically linked xmlcheck.exe, and runs just fine for me, on both Windows and Linux now.

[Earnie Boyd]

I don't remember it being 2 hours to build. It did take a while. So somewhere we have something with GCC that is breaking the DLL build?

I committed a change to tests/Makefile.in to use -static which can be reverted if we figure out what is happening with the DLL. I also committed the fixes needed to improve the catalogue.

Should we close this ticket and open another to track the xerces-c DLL issue?

[Keith Marshall]

I don't remember it being 2 hours to build. It did take a while.

Maybe 2 hrs was an exaggeration, but I think it was closer to 2 than 1; I didn't actually watch it, the whole time. BTW, did you patch the test suite? It doesn't compile OOTB for me.

So somewhere we have something with GCC that is breaking the DLL build?

Seems so. Out of interest, what configure options did you use? Mine were:

../configure --enable-shared --enable-static --disable-rpath LDFLAGS=-no-undefined

(using a separate build subdirectory).

I committed a change to tests/Makefile.in to use -static which can be reverted if we figure out what is happening with the DLL.

Unfortunately, that breaks it (badly) for me. I'm going to revert it, straight away; if you need to kludge it, you can add -static to the CFLAGS, at either configure or at make time.

I also committed the fixes needed to improve the catalogue.

Thanks.

Should we close this ticket and open another to track the xerces-c DLL issue?

I think so. We may need to seek assistance from the Apache/Xerces folks.

[Keith Marshall]

... you can add -static to the CFLAGS, at either configure or at make time.

Correction: it should be added to CXXFLAGS, or maybe better LDFLAGS.

[Cesar Strauss]

I'm also getting this error:

Assertion failed: manager != 0, file ../../src/xercesc/util/XMemory.cpp, line 63

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

However, -static didn't work:

$ export LDFLAGS=-static
$ ../src/configure
[...]
$ make
[...]
g++ -g -O2 -o xmlcheck.exe -static xmlcheck.o -lxerces-c 
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../..\libxerces-c.a(PlatformUtils.o):(.text+0x1ee): undefined reference to `__gxx_personality_sj0'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../..\libxerces-c.a(PlatformUtils.o):(.text+0x204): undefined reference to `_Unwind_SjLj_Register'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../..\libxerces-c.a(PlatformUtils.o):(.text+0x23a): undefined reference to `_Unwind_SjLj_Unregister'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../..\libxerces-c.a(PlatformUtils.o):(.text+0x279): undefined reference to `_Unwind_SjLj_Resume'
[...]

I'm on Windows 7 64-bit. I made a fresh install with:

mingw-developer-toolkit
mingw32-base
mingw32-g++
msys-base

plus:

mingw32-xerces-c-bin
mingw32-xerces-c-dev
mingw32-xerces-c-dll

The `_Unwind_SjLj_Resume' references are suspicious. I'll try building my own xerces lib next time.

Full build logs are attached.

Regards,
Cesar

[Earnie Boyd]

Cesar if you're in a hurry to get this resolved for you, remove the libxerces-c.dll.a file from your /mingw/lib directory.

[Cesar Strauss]

Cesar if you're in a hurry to get this resolved for you, remove the
libxerces-c.dll.a file from your /mingw/lib directory.

It still didn't work:

cstrauss@home /c/projetos/mingw-dist/bld
$ mv /mingw/lib/libxerces-c.dll.a /mingw/lib/libxerces-c.dll.a.bak

cstrauss@home /c/projetos/mingw-dist/bld
$ ls /mingw/lib/libxerces-c.dll.a
ls: /mingw/lib/libxerces-c.dll.a: No such file or directory

cstrauss@home /c/projetos/mingw-dist/bld
$ ../src/configure
[...]

cstrauss@home /c/projetos/mingw-dist/bld
$ make
[...]
g++ -g -O2 -o xmlcheck.exe  xmlcheck.o -lxerces-c 
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libxerces-c.a(PlatformUtils.o):(.text+0x1ee): undefined reference to `__gxx_personality_sj0'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libxerces-c.a(PlatformUtils.o):(.text+0x204): undefined reference to `_Unwind_SjLj_Register'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libxerces-c.a(PlatformUtils.o):(.text+0x23a): undefined reference to `_Unwind_SjLj_Unregister'
c:/programas/mingw/bin/../lib/gcc/mingw32/4.8.1/../../../libxerces-c.a(PlatformUtils.o):(.text+0x279): undefined reference to `_Unwind_SjLj_Resume'
[...]

Full log attached.

I do have the static library /mingw/lib/libxerces-c.a, but I suspect it is incompatible with our compiler, because of SJLJ x Dwarf-2 differences.

Regards,
Cesar

[Keith Marshall]

I do have the static library /mingw/lib/libxerces-c.a,

Is that my build, as posted in FRS?

but I suspect it is incompatible with our compiler, because of SJLJ x Dwarf-2 differences.

I feared that may be an issue; it is compatible with our GCC-3.4.5 compiler, (which I used to build it); I guess GCC-4.x needs a different build.

Building on Windows (under MSYS) takes an age, but I did build it a week or so ago, with GCC-4.7.2 on Win7 32-bit VM; that didn't create any issue for me. If it helps, I've attached my working build of xmlcheck.exe, and the associated libxerces-c-3-1.dll

[Cesar Strauss]

I do have the static library /mingw/lib/libxerces-c.a,

Is that my build, as posted in FRS?

Yes.

If it helps, I've attached my working build of xmlcheck.exe, and the
associated libxerces-c-3-1.dll

It worked, thanks.

[Earnie Boyd]

So maybe the solution is to deliver xmlcheck.exe with the xerces-c package instead of relying on the client process to build it?

[Keith Marshall]

I don't think so. It isn't part of xerces-c; it is part of mingw-dist.

By all means, kludge around your build issue by copying my prebuilt xmlcheck.exe, and my Win32 build of libxerces-c-3-1.dll, into your working mingw-dist tree, but the issue remains: I can successfully build it on 32-bit Win7, with GCC-4.7.2 under MSYS; what's broken in your GCC-4.8, that you can't?

[Keith Marshall]

FWIW, I've repackaged my original xerces-c-3.1.1 as two separate releases:--

1. xerces-c-3.1.1-1-mingw32-3.4.5
2. xerces-c-3.1.1-2-mingw32-4.7.2

The first of these is my original cross compiled GCC-3.4.5 build, renamed; the second is the GCC-4.7.2 build, from the 32-bit Win7 VM, with MSYS. Does my GCC-4.7.2 build make it possible for you to build xmlcheck.exe successfully?

[Cesar Strauss]

FWIW, I've repackaged my original xerces-c-3.1.1 as two separate releases:--

xerces-c-3.1.1-1-mingw32-3.4.5
xerces-c-3.1.1-2-mingw32-4.7.2

The first of these is my original cross compiled GCC-3.4.5 build, renamed; the
second is the GCC-4.7.2 build, from the 32-bit Win7 VM, with MSYS. Does my
GCC-4.7.2 build make it possible for you to build xmlcheck.exe successfully?

Indeed it does. I didn't even had to remove the static library:

$ ls /mingw/lib/libxerces-c.*
/mingw/lib/libxerces-c.a      /mingw/lib/libxerces-c.la
/mingw/lib/libxerces-c.dll.a

$ make
g++ -g -O2 -o xmlcheck.exe  xmlcheck.o -lxerces-c
./xmlcheck.exe ../../src/tests/pkgspec.xsd ../../src/tests/../*/*.xml \
          2> xmlcheck.log && rm -f xmlcheck.log \
             || { cat xmlcheck.log; exit 1; }

$ echo $?
0

Thanks,
Cesar