Re: [aXSL-user] Testing suites?

[Tony G. <Ton...@Me...>]

On Thu, Oct 11 2007 15:51:56 +0100, vi...@ou... wrote:
> You wrote: 
>
>> The page at http://www.axsl.org/ includes:
>> 
>>    The following are possible areas where XSL-FO processing might
>>    benefit from standards:
>>    ...
>>    * Testing suites.
>> 
>> What do you mean?
>> 
>> The XSL FO SG has a workable DTD for defining tests and test 
>> results [1], and NIST has over 2,700 tests "covering most 
>> basic features of the XSL Formatting Objects." [2]
>
> You may be right. However, with regard to the DTD:

I'm not insisting that you use the DTD or test suite, just pointing out
that they exist.

I have stuck with it [2] partly through inertia and partly because it
has been the best prospect for a test interchange format, but to date
test interchange hasn't really happened outside of the CR phases of XSL
1.0 and XSL 1.1.

> 1. There is certainly a difference between a standardized format for a test
> and a set of standardized tests. In other words, one of the potential
> benefits of a standardized API is that JUnit tests can be written to that
> API and shared amongst the various implementations that use that API. The

Inasmuch as xmlroff (http://xmlroff.org) is written in C, I, at least,
have more use for a standard set of tests than I have for JUnit tests.

> aXSL Hyphenation contains one such test class right now, and I hope we can
> expand this concept in future releases. That test should perhaps be
> expressed in a more general way (e.g. an XML file) instead of as a JUnit
> test.

Since the largest body of existing tests uses the DTD, it seems to me
that either the existing DTD is the lowest common denominator for an XML
format for defining tests or a necessary step for any other XML format
is to define the transformation from the legacy format to the new
format.

> 2. aXSL has 13 modules, and I think the DTD would only be useful in testing
> one of them (the FO Tree module). However, I confess I am not aware of
> testing schemes that use the DTD you mentioned, and perhaps I am ignorant of
> its true scope.

xmlroff isn't from the same family tree as the open source Java
processors, so where we intersect is at the output.

> With regard to the NIST suite, I have spent quite a bit of time trying to
> figure out how to integrate it into product testing. In my experience with
> an aXSL implementation and another XSL-FO implementation, it has not been
> very useful, for two reasons:
> 1. It jumps directly to PDF output. To set up an automated testing scheme,
> one would need to parse the PDF files provided as output and logically (not
> literally) compare them to PDF output from the product being tested. If
> those tests included an abstraction of the area tree instead of (or in
> addition to) the end-result PDF, they would be EXTREMELY useful.

Except that there are allowable differences in laying out lines, not to
mention different fonts on different OSes, such that a test wouldn't
have to be very complex before it started producing different area trees
for either different implementations or different platforms.

Even the wiggle-room allowed for border styles [1] could produce area
trees with lots of differences.

> 2. I am under the impression that the PDFs in the NIST suite were generated
> by some XSL-FO implementation, but I could never find any certification that
> they were correct. Did someone at NIST measure fractions of points in the
> PDF itself or on the screen to ensure that all objects are positioned and
> sized correctly? Are we really testing for conformity with some unnamed
> XS-FO implementation by using those tests? I was never comfortable that
> those tests were actually useful. This may just be a documentation issue.

There's at least three uses for a test suite:

 - Checking conformance against the standard

 - Checking that the output is what you expect

 - Checking that the output hasn't regressed

Since there is no one, true output for any input, the PDFs in the NIST
suite could be looked on as a guide rather than a goal.  When it comes
to regression testing, you probably want to compare against your own
output rather than checking that you've maintained the same differences
w.r.t. the sample output.  And if you don't actually agree with the
spec, the provided reference isn't that much use to you while you're
waiting for an answer on xsl...@w3....

> On both of these issues, I may just be ill-informed, but, at any rate, that
> is my current thinking.

And hopefully I've explained mine.

Regards,


Tony Graham
======================================================================
Ton...@Me...   http://www.menteithconsulting.com

Menteith Consulting Ltd             Registered in Ireland - No. 428599
Registered Office: 13 Kelly's Bay Beach, Skerries, Co. Dublin, Ireland
----------------------------------------------------------------------
Menteith Consulting -- Understanding how markup works
======================================================================

[1] http://www.w3.org/TR/xsl11/#border-top-style
[2] http://xmlroff.org/wiki/TestingModule