Re: [aXSL-user] Testing suites?

[Victor M. <vi...@ou...>]

Hi Tony: 

> Inasmuch as xmlroff (http://xmlroff.org) is written in C, I, 
> at least, have more use for a standard set of tests than I 
> have for JUnit tests.

I understand. But none of this diminishes the value of Java tests for those
implementations that do use Java. The original idea behind the common
testing was so that FOray and FOP could share code and tests. That specific
use-case has disappeared, but the general idea is still valid, I think.

...

> > 2. aXSL has 13 modules, and I think the DTD would only be useful in 
> > testing one of them (the FO Tree module). However, I 
> confess I am not 
> > aware of testing schemes that use the DTD you mentioned, 
> and perhaps I 
> > am ignorant of its true scope.
> 
> xmlroff isn't from the same family tree as the open source 
> Java processors, so where we intersect is at the output.

While the C vs. Java difference is relevant, I don't follow this one at all.
The heritage is not so much an issue as the philosophy. For example, FOP and
FOray have the same heritage, but FOP prefers a monolithic design, FOray
prefers a modular design. aXSL was designed for the modular approach, so
won't probably ever be useful for FOP. Modules developed from different
heritages that wanted to use aXSL interfaces could do so, and then take
advantage of (theoretically possible) tests that were written to that API.

So I agree that all of the implementation intersect at the output. But for
those that use the same inter-module API, there is potential at least for
testing at those points as well.

> > With regard to the NIST suite, I have spent quite a bit of 
> time trying 
> > to figure out how to integrate it into product testing. In my 
> > experience with an aXSL implementation and another XSL-FO 
> > implementation, it has not been very useful, for two reasons:
> > 1. It jumps directly to PDF output. To set up an automated testing 
> > scheme, one would need to parse the PDF files provided as 
> output and 
> > logically (not
> > literally) compare them to PDF output from the product 
> being tested. 
> > If those tests included an abstraction of the area tree 
> instead of (or 
> > in addition to) the end-result PDF, they would be EXTREMELY useful.
> 
> Except that there are allowable differences in laying out 
> lines, not to mention different fonts on different OSes, such 
> that a test wouldn't have to be very complex before it 
> started producing different area trees for either different 
> implementations or different platforms.
> 
> Even the wiggle-room allowed for border styles [1] could 
> produce area trees with lots of differences.

Is your point that testing is not possible at all? Or that these differences
need to be allowed for? The point I was trying to make is that rendering to
PDF makes comparison of results practically impossible.

I suspect that I am not understanding something important about the way you
are using the NIST tests and the DTD, and the links you have kindly provided
don't enlighten me. What is the general approach to your xmlroff testing?
Are you parsing the two PDFs and (logically) comparing the output? If so,
that is very cool.

...

> > On both of these issues, I may just be ill-informed, but, 
> at any rate, 
> > that is my current thinking.
> 
> And hopefully I've explained mine.

Somewhat. I get it that the aXSL testing (mostly theoretical now) isn't
useful for you, but that doesn't mean that it is not useful at all.
Nevertheless, I think your original point was suggesting that we needed to
take full advantage of the standard tools that you mentioned. That might be
a very valid point -- I hope you can provide a brief overview of how you are
using the NIST output to test your own work. I confess that I still don't
understand.

Victor Mote