xmlroff-list

[xmlroff] make testing more accessible to contributors

[Stefan S. <se...@sy...>]

When trying to confine a bug I tried to first see whether it was already
covered by an existing test. Unfortunately, I found the existing test
infrastructure a bit confusing / hard to use.

Is there anything that could be done to make that easier ? Ideally
I would only need to have to check out an additional module (not two !)
and then run 'make check' (or similar) in it.
I did glance over some READMEs both in 'testsuite' as well as 'testing',
but couldn't quite get it to work.

Thanks,
		Stefan

Re: [xmlroff] make testing more accessible to contributors

[Tony G. <Ton...@Su...>]

Stefan Seefeld <se...@sy...> writes:
> When trying to confine a bug I tried to first see whether it was already
> covered by an existing test. Unfortunately, I found the existing test
> infrastructure a bit confusing / hard to use.

How about "idiosyncratic"?

> Is there anything that could be done to make that easier ? Ideally
> I would only need to have to check out an additional module (not two !)
> and then run 'make check' (or similar) in it.

The test definition and the test results documents conform to the DTD
developed for XSL FO tests for the tests done for the XSL 1.0 Candidate
Recommendation stage.

Multiple vendors and NIST contributed test suites to the XSL 1.0 CR testing.
Multiple vendors each ran multiple test suites and produced documents listing
how well their implementation faired with each test.

The 'testsuite' and 'testing' modules are separate so, in principle, the
'testing' module could be used with any testsuite and the 'testsuite' module
could be used by anyone who is set up to use XSL 1.0 CR testsuites.

The scripts and XSLT stylesheets in the 'testing' package do three things:

 - Generates the shell script that can run all the tests

 - Creates summary and individual HTML reports for the tests

 - Updates results data from the form in each individual test result page

Updating the results is clunky and requires that you regenerate all the HTML
to see the results reflected in the HTML, but it's a lot more convenient than
updating results by hand.

And, yes, it could be improved.

> I did glance over some READMEs both in 'testsuite' as well as 'testing',
> but couldn't quite get it to work.

Try 'make-success-report.sh'.

There isn't a Makefile target because, with the testsuite in a separate
directory and the test FO files never referred to in any way sensible to
'make', there's no way that 'make' could manage the dependencies and reliably
run the tests only when necessary.

Regards,


Tony.

Re: [xmlroff] make testing more accessible to contributors

[Tony G. <Ton...@Su...>]

Stefan Seefeld <se...@sy...> writes:
> When trying to confine a bug I tried to first see whether it was already
> covered by an existing test. Unfortunately, I found the existing test
> infrastructure a bit confusing / hard to use.
>
> Is there anything that could be done to make that easier ? Ideally
> I would only need to have to check out an additional module (not two !)
> and then run 'make check' (or similar) in it.
> I did glance over some READMEs both in 'testsuite' as well as 'testing',
> but couldn't quite get it to work.

Are you still stuck?

Regards,


Tony.

Re: [xmlroff] make testing more accessible to contributors

[Stefan S. <se...@sy...>]

Tony Graham wrote:
> Stefan Seefeld <se...@sy...> writes:
> 
>>When trying to confine a bug I tried to first see whether it was already
>>covered by an existing test. Unfortunately, I found the existing test
>>infrastructure a bit confusing / hard to use.
>>
>>Is there anything that could be done to make that easier ? Ideally
>>I would only need to have to check out an additional module (not two !)
>>and then run 'make check' (or similar) in it.
>>I did glance over some READMEs both in 'testsuite' as well as 'testing',
>>but couldn't quite get it to work.
> 
> 
> Are you still stuck?

Sorry, haven't had time to look at it again after your explanations.
The reason I asked originally was because I was looking for some test
case I could use to find out whether some issues I observed with my own
document were known / expected.

Meanwhile I get my answer without looking at the test suite.

I still think that the test suite is important not only to developers,
but also to users who want to know what they can expect from xmlroff.
However, for the latter it would be best if features were expressed in
terms of their own language, e.g. docbook constructs that would fail,
instead of fo (which most users only care little about).

However, I can understand that it is out of the scope for xmlroff, and
you in particular, to set up a list of such issues.
Or may be it is some combination of the existing test suite results
with the filtering done by libfo-compat.xsl...

Regards,
		Stefan

Re: [xmlroff] make testing more accessible to contributors

[Tony G. <Ton...@Su...>]

Stefan Seefeld <se...@sy...> writes:
> Tony Graham wrote:
>> Stefan Seefeld <se...@sy...> writes:
>>
>>>When trying to confine a bug I tried to first see whether it was already
>>>covered by an existing test. Unfortunately, I found the existing test
>>>infrastructure a bit confusing / hard to use.
>>>
>>>Is there anything that could be done to make that easier ? Ideally
>>>I would only need to have to check out an additional module (not two !)
>>>and then run 'make check' (or similar) in it.
>>>I did glance over some READMEs both in 'testsuite' as well as 'testing',
>>>but couldn't quite get it to work.
>> Are you still stuck?
>
> Sorry, haven't had time to look at it again after your explanations.
> The reason I asked originally was because I was looking for some test
> case I could use to find out whether some issues I observed with my own
> document were known / expected.
>
> Meanwhile I get my answer without looking at the test suite.
>
> I still think that the test suite is important not only to developers,
> but also to users who want to know what they can expect from xmlroff.
> However, for the latter it would be best if features were expressed in
> terms of their own language, e.g. docbook constructs that would fail,
> instead of fo (which most users only care little about).
>
> However, I can understand that it is out of the scope for xmlroff, and
> you in particular, to set up a list of such issues.
> Or may be it is some combination of the existing test suite results
> with the filtering done by libfo-compat.xsl...

Something like the DocBook 'docbook-testdocs' package [1] run through the
DocBook stylesheets and libfo-compat.xsl?

If we are going to want to become the best little DocBook formatter on the
planet, that would be one place to start.

Regards,


Tony.

[1] http://sourceforge.net/project/shownotes.php?release_id=98501&group_id=21935

Re: [xmlroff] make testing more accessible to contributors

[Stefan S. <se...@sy...>]

Tony Graham wrote:

> Something like the DocBook 'docbook-testdocs' package [1] run through the
> DocBook stylesheets and libfo-compat.xsl?
> 
> If we are going to want to become the best little DocBook formatter on the
> planet, that would be one place to start.
> 
> Regards,
> 
> 
> Tony.
> 
> [1] http://sourceforge.net/project/shownotes.php?release_id=98501&group_id=21935

Excellent ! I'll have a look at it tonight...

Thanks,
		Stefan