Re: [aXSL-user] Testing suites?

[Tony G. <Ton...@Me...>]

Victor,

On Thu, Oct 11 2007 20:56:50 +0100, vi...@ou... wrote:
>> Inasmuch as xmlroff (http://xmlroff.org) is written in C, I, 
>> at least, have more use for a standard set of tests than I 
>> have for JUnit tests.
>
> I understand. But none of this diminishes the value of Java tests for those
> implementations that do use Java. The original idea behind the common
> testing was so that FOray and FOP could share code and tests. That specific
> use-case has disappeared, but the general idea is still valid, I think.

Though the aXSL page doesn't mention Java until near the end and, for
example, SAX and DOM are/have APIs that have been implemented for more
than just Java.

> ...
>
>> > 2. aXSL has 13 modules, and I think the DTD would only be useful in 
>> > testing one of them (the FO Tree module). However, I 
>> confess I am not 
>> > aware of testing schemes that use the DTD you mentioned, 
>> and perhaps I 
>> > am ignorant of its true scope.
>> 
>> xmlroff isn't from the same family tree as the open source 
>> Java processors, so where we intersect is at the output.
>
> While the C vs. Java difference is relevant, I don't follow this one at all.
> The heritage is not so much an issue as the philosophy. For example, FOP and
> FOray have the same heritage, but FOP prefers a monolithic design, FOray
> prefers a modular design. aXSL was designed for the modular approach, so
> won't probably ever be useful for FOP. Modules developed from different
> heritages that wanted to use aXSL interfaces could do so, and then take
> advantage of (theoretically possible) tests that were written to that API.

I'm sorry, I think I assumed more commonality between FOray and FOP than
exists.

(I have lurked on fop-dev on-and-off for several years and have seen you
post there (and arguing for modularity there as well).)

> So I agree that all of the implementation intersect at the output. But for
> those that use the same inter-module API, there is potential at least for
> testing at those points as well.

Though at present that's only FOray.

I'm not trying to impugn the worth of having an API for XSL processing,
but for the foreseeable future, aXSL is most useful to xmlroff if there
is something useful behind those two words "Testing suites" on the aXSL
home page.

...
>> Except that there are allowable differences in laying out 
>> lines, not to mention different fonts on different OSes, such 
>> that a test wouldn't have to be very complex before it 
>> started producing different area trees for either different 
>> implementations or different platforms.

...
> Is your point that testing is not possible at all? Or that these differences
> need to be allowed for? The point I was trying to make is that rendering to
> PDF makes comparison of results practically impossible.

Whereas I think that subjective comparisons of the PDF (or other) output
is the best that you can hope for unless you are comparing XSL
formatters that are practically identical.

> I suspect that I am not understanding something important about the way you
> are using the NIST tests and the DTD, and the links you have kindly provided
> don't enlighten me. What is the general approach to your xmlroff testing?
> Are you parsing the two PDFs and (logically) comparing the output? If so,
> that is very cool.

My approach is that eyeballs are better than 'diff' for recognising
significant versus insignificant differences.

I don't currently use the NIST tests because there's so many of them
that I don't have the bandwidth to verify all the results.  Years ago, I
used to run the NIST tests in a cron job every night and ignore the
results in my email every morning.  There is an xmlroff ticket to cut
down the NIST tests so we can start again with testing a subset of the
2,700+.

The tests that I do run are FO files that I wrote and the DocBook
testdocs collection, since DocBook formatting is an important goal for
xmlroff users.

The script that runs the tests is generated from XML that conforms to
the DTD.

The PDF or PostScript output of a test run is compared (using 'diff')
against the result from a previous run, and if there are differences,
the pages are rasterised, and each page is compared against the
corresponding page from the result of the previous run.  For pages that
have differences, a "stereo" PNG is made that has the red channel from
one version and the blue channel from the other so the differences are
more visible.

It then becomes a judgement call as to whether the differences are
significant or not, particularly when, if there are differences, the
differences may have been caused by changes to something unrelated to
the implementation of the FO or property that is the subject of the
test.

The summary of the test results is recorded in another XML file that
also conforms to the DTD.

Most of the time, the comparisons are between the results from two
builds of xmlroff, but I used the system several years ago to compare
xmlroff output against sample PDFs provided with the samples for the XSL
bake-of (I forget exactly what it was called) at XML 2003.

> ...
>
>> > On both of these issues, I may just be ill-informed, but, 
>> at any rate, 
>> > that is my current thinking.
>> 
>> And hopefully I've explained mine.
>
> Somewhat. I get it that the aXSL testing (mostly theoretical now) isn't
> useful for you, but that doesn't mean that it is not useful at all.

It could be useful in the general case, I agree.

> Nevertheless, I think your original point was suggesting that we needed to
> take full advantage of the standard tools that you mentioned. That
> might be

My original comment was really just to point out that the DTD and the
NIST tests already exist, though the best that I could say about the DTD
is that it is workable, since I'm not going to pretend that it's in any
way ideal.

> a very valid point -- I hope you can provide a brief overview of how you are
> using the NIST output to test your own work. I confess that I still don't
> understand.

I didn't start out with the intention of converting anybody to the
xmlroff way of testing, rather I was trying to find out what you meant
by "Testing suites", but hopefully I've done a better job this time of
explaining my testing approach.

Regards,


Tony Graham.
======================================================================
Ton...@Me...   http://www.menteithconsulting.com

Menteith Consulting Ltd             Registered in Ireland - No. 428599
Registered Office: 13 Kelly's Bay Beach, Skerries, Co. Dublin, Ireland
----------------------------------------------------------------------
Menteith Consulting -- Understanding how markup works
======================================================================