Re: [Docutils-develop] [docutils:feature-requests] #81 0.17.1: pytest is failing

[Adam T. <aat...@ou...>]

Dear Günter,

This note contains replies to all the recent emails in the "0.17.1: pytest is failing" thread.

> On 2022-11-10, Adam Turner via Docutils-develop wrote:

>> With [r9237] the test-suite refactoring project is complete -- using 
>> `pytest` and `python -m unittest` now work "out-of-the-box".

-----

> Run all test tests from the ``test`` directory::
>
>     #> cd REPO-ROOT/docutils/test
>
> Now testing with different methods::
>
>     # ./alltests.py
>     [...]
>
> * All tests run.
> * Test reporting (in case of failures) seems OK.
> * Some diffs only show up after adding ``maxDiff = None`` to the test class
>   definition.

Perhaps we should add ``maxDiff = None`` to all test cases.

-----

> ::
>
>     #> pytest-3 --quiet .
>     [...]
>
>
> * Only 369 out of 1739 tests reported by "alltests.py" are reported.
>   Is this different counting or does pytest miss more than half of the tests?

Different counting -- "sub tests" aren't counted individually by default, see ``NumbersTestResult`` in ``alltests.py``.


> * The warnings show up despite explicit silencing in the code.
>   (Except when we explicitely test for them.)
>   It seems `pytest` bypasses this and reports anyway??

``test_settings`` didn't have the warnings silenced in the ``setUp()`` block.

-----

> Next the standard Python way::
>
>     #> python3.9 -m unittest .
>
> :(

``python -m unittest`` has an odd calling convention. Had you typed ``python -m unittest``, it would've succeeded, as it is shorthand for the 'actual' call of ``python -m unittest discover .``. Adding any arguments means the shorthand isn't used, though.

I am unsure if we should declare support for this method of running tests, whilst it works it may constrain us should we choose e.g. to move to ``pytest`` more properly.

-----

> The "nose" test framework fails, too::

``nose`` is unsupported and I don't intend to support it. The maintainers don't recommend using it (https://nose.readthedocs.io/en/latest/#note-to-users)

-----

> Running idividual test files::
> [...]

> Its a "chicken or egg" problem: the only remaining code in ``DocutilsTestSupport.py`` is adding the "docutils root" and the "test root"
> to sys.path. However, it can only be imported if "test" is recognized as package.

Must the executable be ``python`` here? ``pytest .\test_parsers\test_rst\test_definition_lists.py`` works, as ``pytest`` discovers the root directory as the one containing ``setup.cfg`` (or a numer of other sentinel files) and prepends it to ``sys.path``. Similarly, ``pytest test_definition_lists.py`` works within ``test\test_parsers\test_rst``. 

-----

> trying the test suite on an installation without recommonmark, I stumbled about a new problem:
>     
>     Traceback (most recent call last):
>     [...]
>         raise unittest.SkipTest(f'Cannot test "{md_parser}". '
>     unittest.case.SkipTest: Cannot test "recommonmark". Parser not found.
>
> It seems raising SkipTest is only applicable for test run through the unittest framework (where I am not sure all test run).

Ahh, we should use ``unittest.defaultTestLoader.discover`` to load tests, this respects ``SkipTest``.

Thanks,
Adam