#160 OASIS XML Test Suite

open-accepted
None
5
2003-09-24
2002-06-15
Rolf Ade
No

I've tested expat-1.95.3 (with xmltok.c updated to rev.
1.17, becase of bug 566240, all other files are the
original 1.95.3) against the recently updated OASIS xml
test suite (XML 1.0 (Second Edition) errata 20020320,
W3C Conformance Test Suite 20020606), avaliable via
http://www.w3.org/XML/Test/ and found a few new
problems, that are not triggered by older versions of
this test suite.

As in previous reports, I checked all
not-wellformedness tests (should
all raise error) and all valid tests (should all pass)
of the
test-suites xmltest, ibm, sun and oasis with xmlwf -p.
Especially for the well-formedness tests, I have _not_
throughout checked if the error reason, reported by
expat is the expected error, but checked only
mechanical, if the test has raised an error, regardless
of the exact error reason. This method is clearly not
perfect, and this time we have an example, that
underlines this.

ibm/not-wf/P32/ibm32n09.xml

This is a new test, not included in previous versions.
Problem is, that the standalone document declaration
has the value "yes" and there is an external markup
declaration of an entity (other than amp, lt, gt, apos,
quot). xmlwf -p doesn't report an error. The not
well-formedness problem is, that standalone="yes"
means, that all informations needed to build the XML
infoset must be found in the document entity
(standalone="yes" doesn't mean, that the document must
not have an external subset or external PE's, only that
this external entities doesn't change - per attribute
defaults or as in this case, entity declarations -
change the info in the document entity. See the last
sentence of "Well-Formedness Constraint: Entity
Declared" (P68).

ibm/not-wf/P68/ibm68n06.xml

Same reason as the test befor. This test _was_ present
in previous versions of the test suite. But with the
previous version of the external subset of this test,
xmlwf claimed a "syntax error" error in the external
subset, which I plain can't understand (eventually an
other expat bug?), but is clearly not the expected
error. In the new version of the test suite, this
external subset now has an XML declaration with
explicite encoding (the older version had only an XML
declaration without encoding) and is accepted by expat.

xmltest/not-wf/not-sa/010.xml
xmltest/not-wf/not-sa/011.xml

This tests are new in this edition of the test suite.
Unfortunately, this both tests seems to be not
documented, either in the test files isself nor in the
documentation file xmlconf-20020606.htm. As far as I
see, this tests test "Validity Constraint: Proper
Declaration/PE Nesting" (P29).

xmltest/not-wf/not-sa/005.xml

This test raised error with previous expat versions,
but does not anymore due to the changes, discussed in
bug 548690. This is intentional, according to the
548690 discussion. This test is now listed under "XML
Documents with Optional Errors". The test suite
documentation says:

"Conforming XML 1.0 Processors are permitted to ignore
certain errors, or to report them at user option. In
this section of this test report are found descriptions
of test cases which fit into this category.

Processor behavior on such test cases does not affect
conformance to the XML 1.0 (Second Edition)
Recommendation, except as noted."

So, according to this, it's OK, that expat doesn't
report an error for this case. Since both reporting
error and not reporting error are OK, it may be
debatably, which behavior is more convenient for the
expat user. (Karl: ;-))

sun/not-wf/not-sa03.xml

This is a new test in this edition of the test suite.
Unfortunately, this test seems not to be documented. As
far as I see, it tests the same as
xmltest/not-wf/not-sa/005.xml

Tests, that still are wrong, as in previous versions are

ibm/not-wf/misc/432gewf.xml
sun/not-wf/uri01.xml

These are already discussed in the past.

Well, that's all.

rolf

Discussion

  • Karl Waclawek

    Karl Waclawek - 2002-06-17

    Logged In: YES
    user_id=290026

    Given an improved understanding of section 4.1
    in the XML spec, I will try to fix the following
    test cases in the next Expat release:

    ibm/not-wf/P32/ibm32n09.xml,
    ibm/not-wf/P68/ibm68n06.xml and
    sun/not-wf/not-sa03.xml

    In my opinion, the third one is not the same type as
    xmltest/not-wf/not-sa/005.xml, but the same type
    as the other two.

    About the test cases
    xmltest/not-wf/not-sa/010.xml and
    xmltest/not-wf/not-sa/011.xml:

    If they really check validity constraint P29, as Rolf
    has suggested, then it is OK that Expat does not report
    an error.

    So, If I am successful, we would be left with only:
    ibm/not-wf/misc/432gewf.xml and
    sun/not-wf/uri01.xml,
    conformance with which does not seem a 100%
    necessity, as previously discussed.

    Karl

     
  • Rolf Ade

    Rolf Ade - 2002-06-17

    Logged In: YES
    user_id=13222

    Agreed

     
  • Karl Waclawek

    Karl Waclawek - 2002-06-18
    • assigned_to: nobody --> kwaclaw
     
  • Karl Waclawek

    Karl Waclawek - 2002-06-18

    Logged In: YES
    user_id=290026

    Assigned to me, but only for the three test cases
    described in my last message.

    Karl

     
  • Karl Waclawek

    Karl Waclawek - 2002-07-07

    Logged In: YES
    user_id=290026

    Patch #587161 should fix some of the problems,
    but I specifically made no attempt to fix the problems
    Expat has with:
    - xmltest/not-wf/not-sa/010.xml and
    - xmltest/not-wf/not-sa/011.xml.

    Reason: It turns out, after consulting with the mailing
    list for the XML test suite, public-xml-testsuite@w3.org,
    that these two violate WFC: PE Between Declarations.
    There is no quick and easy fix for this in Expat,
    and I would have to spend some time thinking about it,
    which I don't have at the moment.

    Karl

     
  • Karl Waclawek

    Karl Waclawek - 2003-01-19

    Logged In: YES
    user_id=290026

    Just a comment:
    This bug report will likely stay open until Expat passes the
    OASIS test suite without any problem at all.
    Since no parser currently achieves this, there is a good
    chance this bug report will stay open for a long time to come..

     
  • Karl Waclawek

    Karl Waclawek - 2003-01-19
    • status: open --> open-accepted
     
  • Karl Waclawek

    Karl Waclawek - 2003-01-21

    Logged In: YES
    user_id=290026

    For comparison purposes, I ran the xmltest.sh test script
    against release 1.95.5. The results are attached as
    TestResults_1_95_5.txt.

    Discussion of results:

    There are many cases were "output differs"is reported,
    but these are due to xmlwf having a different definition
    of "canonical XML" than used in the test suite.

    Leaving these out, and those that were discussed
    already in this thread, we have the following errors
    reported:
    (Note: the two test cases
    ibm/not-wf/P32/ibm32n09.xml and
    ibm/not-wf/P68/ibm68n06.xml
    are not reported anymore by the script)

    * In ibm/invalid/P49/: ibm49i02.xml:7:1: error in
    processing external entity reference:
    The associated DTD file does not exist - an error
    in the test suite.

    The next three documents are not UTF-8 encoded,
    and do not have an XML declaration, so Expat rejects
    them, which is correct. An error in the test suite.

    * In xmltest/valid/sa/: 049.xml:2:0: not well-formed
    (invalid token)
    * In xmltest/valid/sa/: 050.xml:2:0: not well-formed
    (invalid token)
    * In xmltest/valid/sa/: 051.xml:2:0: not well-formed
    (invalid token)

    The next two documents are classified as invalid,
    but well-formed, but they contain faulty UTF-16
    encoding, so they should be classified as not
    well-formed. Expat seems correct here.

    * In sun/invalid/: utf16b.xml:2:0: not well-formed
    (invalid token)
    * In sun/invalid/: utf16l.xml:1:40: not well-formed
    (invalid token)

    The next three are not marked as why they should fail,
    so the script thinks they are not well-formed, but in
    fact they are:

    * Well formed: oasis/p06fail1.xml
    * Well formed: oasis/p08fail1.xml
    * Well formed: oasis/p08fail2.xml

    So, no new test case errors have really been added
    for release 1.95.5.

     
  • Karl Waclawek

    Karl Waclawek - 2003-01-29

    Logged In: YES
    user_id=290026

    This is just to report that the new release Expat 1.95.6
    passes the OASIS test suite (same version - 20020606)
    with the exact same results as Expat 1.95.5.

     
  • Karl Waclawek

    Karl Waclawek - 2003-09-24

    Logged In: YES
    user_id=290026

    Just changed the summary to be more generic, as this bug
    will probably stay open permanently, assuming we will never
    pass 100% of all test cases.

     
  • Karl Waclawek

    Karl Waclawek - 2003-09-24
    • summary: 1.95.3 and new OASIS xml test suite --> OASIS XML Test Suite
     
  • Nobody/Anonymous

    Logged In: NO

    can any one of u tell me why i am getting failed case as
    33...
    I will tell u all the errors i got...can anyone tell
    me how to rectify the same....and why these errors arised..
    ibm/valid/P02/ibm02v01.xml
    ibm/valid/p28/ibm28v02.xml
    /p29/ 02.xml
    /p29/ 01.xml
    similarly p54, p56 , p58, p57,p70, p82
    ibm/invalid/p49../p58/...
    xmltest/valid/sa/069.xml...76.xml...90.xml...91.xml....

     
  • Karl Waclawek

    Karl Waclawek - 2012-03-18

    The results for expat 2.1.0 are based on XML test suite version 20080827, but they are the same if tested against version 20020606 - see attached file. Essentially, the test cases that really can be considered as failed still are the same as for previous Expat versions:

    Expected not well-formed: ibm/not-wf/misc/432gewf.xml
    Expected not well-formed: sun/not-wf/uri01.xml

    Karl

     

Log in to post a comment.