Re: [Docutils-develop] Testing of writers

[Mark N. <Mar...@fr...>]

Felix Wiemann wrote:
> 
> Most parts of the writers are currently untested.
> 
> We already have a test file tools/test.txt, which could be used for
> testing the writers.
> 
> E.g., we could simply generate an expected output for test.txt and let
> the test module check if the real output matches the expected output.
> 
> However, this approach would have the drawback that anytime there is a
> change to a writer's output, the whole 'expected output' file must be
> regenerated and checked for validity.
> 
> Thus I rather propose splitting up test.txt into many single files (or
> strings) and then creating many tiny little expected-output files (or
> strings).
> 
> For example, there would be a file bullet_lists.txt:
> 
> ...
> 
> Expected outputs could be generated for HTML, LaTeX and pseudoxml.
> 
> When there is a new reST-element, we simply add an input file and three
> output files.  When there is a change to one of the writers, one output
> file has to be modified and checked for validity.
> 
> What do you think?

As I understand it, there are already regression tests for each of 
the individual features of the parser with expected pseudoxml output
(I adapted these for the regression suite for the Perl version).  What
is lacking is checking output for the other writers on each of the
input snippets.  There is some merit in what you say; however the
individual regression tests are aimed more at checking that the parser
works correctly (the pseudoxml writer being intentionally relatively
trivial) and test many things that are uninteresting from the standpoint
of writers, such as the generation of the correct error message for
incorrect syntax.

FWIW, when I went to write my latex writer two weeks ago, I first got
it working for the subset required by the paper I was submitting to
a conference (really short deadline; had to have it the day I started
working on the writer), and then extended it to handle the things in
test.txt.  The regression test I added for the latex writer was, in
fact, the generated output for test.txt.  As you say, there is a
certain fragility to such an approach.  If you change the way
that line-blocks are generated, for example, it will require updating
the expected result, but I don't think it's too burdensome.  The expects
for that feature would have to be updated anyway, and your proposal just
substitutes updating the expects for one or more small tests rather than
one large test.

	--Mark