Re: [Docutils-develop] [docutils:feature-requests] #81 0.17.1: pytest is failing

[Guenter M. <mi...@us...>]

Dear Adam,

On 2022-11-10, Adam Turner via Docutils-develop wrote:

> With [r9237] the test-suite refactoring project is complete -- using
> `pytest` and `python -m unittest` now work "out-of-the-box".

Thank you for picking up the large project of modernising and
"disentangling" the Docutils test suite.

I did not manage to review all the patches line by line but skimmed them
and gave the tests a testing:

Run all test tests from the ``test`` directory::

    #> cd REPO-ROOT/docutils/test

Now testing with different methods::

    # ./alltests.py
    [...]
    Ran 1739 tests in 5.053s
    
    OK (skipped=2)
    Elapsed time: 5.210 seconds

* All tests run.
* Test reporting (in case of failures) seems OK.
* Some diffs only show up after adding ``maxDiff = None`` to the test class
  definition.

::

    #> pytest-3 --quiet .
    [...]
    =============================== warnings summary ===============================
    test/test_settings.py::HelperFunctionsTests::test_make_paths_absolute
    [...]
    test/test_settings.py::HelperFunctionsTests::test_validate_url_trailing_slash
      /usr/local/src/docutils-git-svn/docutils/test/test_settings.py:246: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later.
        self.option_parser = frontend.OptionParser(
    
    test/test_settings.py: 612 warnings
      /usr/lib/python3.9/optparse.py:1000: DeprecationWarning: The frontend.Option class will be removed in Docutils 0.21 or later.
        option = self.option_class(*args, **kwargs)
    
    -- Docs: https://docs.pytest.org/en/stable/warnings.html
    369 passed, 2 skipped, 621 warnings in 6.50s

* Only 369 out of 1739 tests reported by "alltests.py" are reported.
  Is this different counting or does pytest miss more than half of the tests?

* The warnings show up despite explicit silencing in the code.
  (Except when we explicitely test for them.)
  It seems `pytest` bypasses this and reports anyway??

* ``docs/dev/testing.txt`` suggests ``pytest --quiet ./test``
  which fails if the cwd is ``docutils/test`` as suggested by the preceding::

    From a shell do [#]_::

       cd docutils/test
       python -u alltests.py

  I suggest::

    -   For the pytest_ test framework, from a shell run::
    +   For the pytest_ test framework::

    -       pytest --quiet ./test
    +       pytest --quiet .


Next the standard Python way::

    #> python3.9 -m unittest .
    Traceback (most recent call last):
      File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/usr/lib/python3.9/unittest/__main__.py", line 18, in <module>
        main(module=None)
      File "/usr/lib/python3.9/unittest/main.py", line 100, in __init__
        self.parseArgs(argv)
      File "/usr/lib/python3.9/unittest/main.py", line 147, in parseArgs
        self.createTests()
      File "/usr/lib/python3.9/unittest/main.py", line 158, in createTests
        self.test = self.testLoader.loadTestsFromNames(self.testNames,
      File "/usr/lib/python3.9/unittest/loader.py", line 220, in loadTestsFromNames
        suites = [self.loadTestsFromName(name, module) for name in names]
      File "/usr/lib/python3.9/unittest/loader.py", line 220, in <listcomp>
        suites = [self.loadTestsFromName(name, module) for name in names]
      File "/usr/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
        module = __import__(module_name)
    ValueError: Empty module name

:(


The "nose" test framework fails, too::

    #> nosetests3
    ======================================================================
    FAIL: test_parser (test.test_parsers.test_rst.test_directives.test_admonitions_de.ParserTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/local/src/docutils-git-svn/docutils/test/test_parsers/test_rst/test_directives/test_admonitions_de.py", line 37, in test_parser
        self.assertEqual(output, case_expected)
    AssertionError: '<doc[27 chars]    <system_message level="1" line="1" source=[358 chars]n.\n' != '<doc[27 chars]    <admonition classes="admonition-admonition[105 chars]n.\n'
      <document source="test data">
    -     <system_message level="1" line="1" source="test data" type="INFO">
    -         <paragraph>
    -             No directive entry for "admonition" in module "docutils.parsers.rst.languages.de".
    -             Using English fallback for directive "admonition".
          <admonition classes="admonition-admonition">
              <title>
                  Admonition
              <paragraph>
                  This is a generic admonition.
    
    
    ======================================================================
    FAIL: test_parser (test.test_parsers.test_rst.test_directives.test_admonitions_dummy_lang.ParserTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/local/src/docutils-git-svn/docutils/test/test_parsers/test_rst/test_directives/test_admonitions_dummy_lang.py", line 32, in test_parser
        self.assertEqual(output, case_expected)
    AssertionError: '<doc[115 chars]    <system_message level="1" line="3" source=[274 chars]).\n' != '<doc[115 chars]    <attention>\n        <paragraph>\n        [40 chars]).\n'
      <document source="test data">
          <attention>
              <paragraph>
                  directive with silly localised name.
    -     <system_message level="1" line="3" source="test data" type="INFO">
    -         <paragraph>
    -             No directive entry for "Attention" in module "local_dummy_lang".
    -             Using English fallback for directive "Attention".
          <attention>
              <paragraph>
                  English fallback (an INFO is written).
    
    
    ----------------------------------------------------------------------
    Ran 79 tests in 0.642s
    
    FAILED (SKIP=1, failures=2)


Running idividual test files::

    #> python3 test_error_reporting.py
    Traceback (most recent call last):
      File "/usr/local/src/docutils-git-svn/docutils/test/test_error_reporting.py", line 32, in <module>
        from test import DocutilsTestSupport  # NoQA: F401
    ImportError: cannot import name 'DocutilsTestSupport' from 'test' (/usr/lib/python3.9/test/__init__.py)


Its a "chicken or egg" problem: the only remaining code in
``DocutilsTestSupport.py`` is adding the "docutils root" and the "test root"
to sys.path. However, it can only be imported if "test" is recognized as
package.

The solution is inlining the path-modification if scripts are directly
started, e.g.

~~~

diff --git a/docutils/test/test_writers/test_html4css1_template.py b/docutils/test/test_writers/test_html4css1_template.py
index 4113dd443..f771506a4 100755
--- a/docutils/test/test_writers/test_html4css1_template.py
+++ b/docutils/test/test_writers/test_html4css1_template.py
@@ -8,11 +8,16 @@
 Tests for the HTML writer.
 """

+from pathlib import Path
 import os
 import platform
+import sys
 import unittest

-from test import DocutilsTestSupport  # NoQA: F401
+if __name__ == '__main__':
+    # prepend the "docutils root" to the Python library path
+    # so we import the local `docutils` and `test` packages,
+    sys.path.insert(0, str(Path(__file__).parents[2]))

 import docutils
 from docutils.core import publish_string

~~~

For scripts in the "test root", replace ``parents[2]`` with ``parents[1]``
For deeper nested scripts like
docutils/test/test_parsers/test_rst/test_block_quotes.py
replace ``parents[2]`` with ``parents[3]``

Also, if importing "unittest" at the top of the file, there is no need to
repeat this at the end

~~~
diff --git a/docutils/test/test_parsers/test_rst/test_block_quotes.py b/docutils/test/test_parsers/test_rst/test_block_quotes.py
index 4c21e8112..521caf874 100755
--- a/docutils/test/test_parsers/test_rst/test_block_quotes.py
+++ b/docutils/test/test_parsers/test_rst/test_block_quotes.py
@@ -8,9 +8,14 @@
 Tests for states.py.
 """

+from pathlib import Path
+import sys
 import unittest

-from test import DocutilsTestSupport  # NoQA: F401
+if __name__ == '__main__':
+    # prepend the "docutils root" to the Python library path
+    # so we import the local `docutils` and `test` packages,
+    sys.path.insert(0, str(Path(__file__).parents[3]))

 from docutils.frontend import get_default_settings
 from docutils.parsers.rst import Parser
@@ -396,5 +401,4 @@ Paragraph.


 if __name__ == '__main__':
-    import unittest
     unittest.main()

~~~

When inlining the sys.path modification also in "alltests.py",
DocutilsTestSupport.py should become redundant.

I did not catch whether the adding of the "test root" is actually required
anywhere, though.


Have a nice weekend,

Günter