Re: [Docutils-develop] [docutils:feature-requests] #81 0.17.1: pytest is failing

[Guenter M. <mi...@us...>]

Dear Adam, dear Docutils developers,

On 2022-12-02, Adam Turner wrote:

...

> Perhaps we should add ``maxDiff = None`` to all test cases.

To all tests where it applies, yes.
But this might also be done step by step whenever required.


>>     #> pytest-3 --quiet .
>>     [...]


>> * Only 369 out of 1739 tests reported by "alltests.py" are reported.
>>   Is this different counting or does pytest miss more than half of the
>>   tests?

> Different counting -- "sub tests" aren't counted individually by
> default, see ``NumbersTestResult`` in ``alltests.py``.

Ah. So this is because a large number of tests becam "downgraded" to
sub-tests.


...


> I am unsure if we should declare support for this method of running
> tests, whilst it works it may constrain us should we choose e.g. to
> move to ``pytest`` more properly.

We would need to announce this new requirement anyway.
For now, we may just add a caveat, e.g.

diff --git a/docutils/docs/dev/testing.txt b/docutils/docs/dev/testing.txt
index be934a7af..817f20733 100644
--- a/docutils/docs/dev/testing.txt
+++ b/docutils/docs/dev/testing.txt
@@ -40,15 +40,15 @@ In a pinch, the edge cases should cover most of it.
 .. note::
    The ``alltests.py`` test runner is based on the standard library's unittest_
    framework.
-   Since Docutils 0.19, running ``python -m unittest``, the pytest_, and the
-   nose_ frameworks no longer result in spurious failures (cf. `bug #270`_).
-   However, there are differences in the reported number of tests and in
-   test coverage.
+   Since Docutils 0.19, running ``python -m unittest`` and the pytest_
+   framework no longer result in spurious failures (cf. `bug #270`_).
+   However, there are differences in the reported number of tests
+   (``alltests.py`` also counts sub-tests).
+   In future, running the test suite may require pytest_.
 
 __ policies.html#check-ins
 .. _unittest: https://docs.python.org/3/library/unittest.html
 .. _pytest: https://pypi.org/project/pytest
-.. _nose: https://pypi.org/project/nose
 .. _`bug #270`: https://sourceforge.net/p/docutils/bugs/270/
 

>> The "nose" test framework fails, too::

> ``nose`` is unsupported and I don't intend to support it. The
> maintainers don't recommend using it
> (https://nose.readthedocs.io/en/latest/#note-to-users)

OK.

> -----

>> Running idividual test files::

> Must the executable be ``python`` here?  
> ``pytest .\test_parsers\test_rst\test_definition_lists.py`` works, as
> ``pytest`` discovers the root directory as the one containing
> ``setup.cfg`` (or a numer of other sentinel files) and prepends it to
> ``sys.path``. Similarly, ``pytest test_definition_lists.py`` works
> within ``test\test_parsers\test_rst``. 

With the fixes in r9263+, individual test files run fine again.
Switching to pytest is an option for the future 
(would require changing the "shebang line").

> -----

>> trying the test suite on an installation without recommonmark, I
>> stumbled about a new problem:

>>     Traceback (most recent call last):
>>     [...]
>>         raise unittest.SkipTest(f'Cannot test "{md_parser}". '
>>     unittest.case.SkipTest: Cannot test "recommonmark". Parser not found.

>> It seems raising SkipTest is only applicable for test run through the unittest framework (where I am not sure all test run).

> Ahh, we should use ``unittest.defaultTestLoader.discover`` to load
> tests, this respects ``SkipTest``.

Solved now.


Thank you for the work and for the explanations.

Günter