docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:feature-requests] #81 0.17.1: pytest is failing

[Adam T. <aa-...@us...>]

With [r9237] the test-suite refactoring project is complete -- using `pytest` and `python -m unittest` now work "out-of-the-box".

A


---

** [feature-requests:#81] 0.17.1: pytest is failing**

**Status:** open
**Group:** Default
**Created:** Sun Jun 27, 2021 03:07 AM UTC by Tomasz Kłoczko
**Last Updated:** Fri Jan 14, 2022 01:33 PM UTC
**Owner:** nobody


Just normal build, install and test cycle used on building package from non-root account:
- "setup.py build"
- "setup.py install --root </install/prefix>"
- "pytest with PYTHONPATH pointing to setearch and sitelib inside </install/prefix>

~~~
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Using --randomly-seed=2664516846
rootdir: /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1
plugins: forked-1.3.0, shutil-1.7.0, virtualenv-1.7.0, expect-1.1.0, httpbin-1.0.0, flake8-1.0.7, timeout-1.4.2, betamax-0.8.1, freezegun-0.4.2, case-1.5.3, isort-1.3.0, aspectlib-1.5.2, asyncio-0.15.1, toolbox-0.5, xprocess-0.17.1, aiohttp-0.3.0, checkdocs-2.7.0, mock-3.6.1, rerunfailures-9.1.1, requests-mock-1.9.3, cov-2.12.1, pyfakefs-4.5.0, cases-3.6.1, flaky-3.7.0, hypothesis-6.14.0, benchmark-3.4.1, xdist-2.3.0, pylama-7.7.1, randomly-3.8.0, Faker-8.8.2, datadir-1.3.1, regressions-2.2.0
collected 240 items

test/test_statemachine.py .................                                                                                                                          [  7%]
test/test_functional.py E                                                                                                                                            [  7%]
test/test_dependencies.py .....                                                                                                                                      [  9%]
test/test_parsers/test_get_parser_class.py ...                                                                                                                       [ 10%]
test/test_writers/test_latex2e_misc.py .                                                                                                                             [ 11%]
test/test_publisher.py ....                                                                                                                                          [ 12%]
test/test_writers/test_get_writer_class.py ...                                                                                                                       [ 14%]
test/test__init__.py ..                                                                                                                                              [ 15%]
. .                                                                                                                                                                  [ 15%]
test/test__init__.py ......                                                                                                                                          [ 17%]
test/test_settings.py ........................                                                                                                                       [ 28%]
test/test_writers/test_docutils_xml.py .....                                                                                                                         [ 30%]
test/test_parsers/test_parser.py .                                                                                                                                   [ 30%]
test/test_transforms/test___init__.py .                                                                                                                              [ 30%]
test/test_language.py EEEE                                                                                                                                           [ 32%]
test/test_traversals.py .                                                                                                                                            [ 33%]
test/test_writers/test_odt.py ..........                                                                                                                             [ 37%]
test/test_command_line.py F                                                                                                                                          [ 37%]
test/test_writers/test_html5_polyglot_parts.py EE                                                                                                                    [ 38%]
test/test_nodes.py ...............................                                                                                                                   [ 51%]
test/test_writers/test_html5_polyglot_misc.py ...............                                                                                                        [ 57%]
test/test_pickle.py .                                                                                                                                                [ 58%]
tools/test/test_buildhtml.py ..                                                                                                                                      [ 58%]
test/test_io.py .................                                                                                                                                    [ 66%]
test/test_readers/test_get_reader_class.py ...                                                                                                                       [ 67%]
test/test_error_reporting.py .............                                                                                                                           [ 72%]
test/test_writers/test_html4css1_misc.py ...............                                                                                                             [ 79%]
test/test_parsers/test_rst/test_directives/test_code_parsing.py ..                                                                                                   [ 79%]
test/test_utils.py ........................                                                                                                                          [ 89%]
test/test_parsers/test_recommonmark/test_misc.py ..Fs                                                                                                                [ 91%]
test/test_parsers/test_rst/test_directives/test__init__.py ....                                                                                                      [ 93%]
test/test_viewlist.py ................                                                                                                                               [100%]

================================================================================== ERRORS ==================================================================================
________________________________________________________________ ERROR at setup of FunctionalTestCase.test _________________________________________________________________

self = <[AttributeError("'FunctionalTestCase' object has no attribute '_testMethodName'") raised in repr()] FunctionalTestCase object at 0x7fbd75666df0>, args = ('test',)
kwargs = {}

    def __init__(self, *args, **kwargs):
        """Set self.configfile, pass arguments to parent __init__."""
>       self.configfile = kwargs['configfile']
E       KeyError: 'configfile'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_functional.py:95: KeyError
____________________________________________________________ ERROR at setup of LanguageTestCase.test_directives ____________________________________________________________

self = <[AttributeError("'LanguageTestCase' object has no attribute '_testMethodName'") raised in repr()] LanguageTestCase object at 0x7fbd754ce0d0>
args = ('test_directives',), kwargs = {}

    def __init__(self, *args, **kwargs):
        self.ref = docutils.languages.get_language(reference_language,
                                                   _reporter)
>       self.language = kwargs['language']
E       KeyError: 'language'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_language.py:81: KeyError
______________________________________________________________ ERROR at setup of LanguageTestCase.test_roles _______________________________________________________________

self = <[AttributeError("'LanguageTestCase' object has no attribute '_testMethodName'") raised in repr()] LanguageTestCase object at 0x7fbd753f98b0>, args = ('test_roles',)
kwargs = {}

    def __init__(self, *args, **kwargs):
        self.ref = docutils.languages.get_language(reference_language,
                                                   _reporter)
>       self.language = kwargs['language']
E       KeyError: 'language'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_language.py:81: KeyError
_______________________________________________________ ERROR at setup of LanguageTestCase.test_bibliographic_fields _______________________________________________________

self = <[AttributeError("'LanguageTestCase' object has no attribute '_testMethodName'") raised in repr()] LanguageTestCase object at 0x7fbd753c4850>
args = ('test_bibliographic_fields',), kwargs = {}

    def __init__(self, *args, **kwargs):
        self.ref = docutils.languages.get_language(reference_language,
                                                   _reporter)
>       self.language = kwargs['language']
E       KeyError: 'language'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_language.py:81: KeyError
______________________________________________________________ ERROR at setup of LanguageTestCase.test_labels ______________________________________________________________

self = <[AttributeError("'LanguageTestCase' object has no attribute '_testMethodName'") raised in repr()] LanguageTestCase object at 0x7fbd754181f0>
args = ('test_labels',), kwargs = {}

    def __init__(self, *args, **kwargs):
        self.ref = docutils.languages.get_language(reference_language,
                                                   _reporter)
>       self.language = kwargs['language']
E       KeyError: 'language'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_language.py:81: KeyError
______________________________________________________ ERROR at setup of HtmlWriterPublishPartsTestCase.test_publish _______________________________________________________

self = <[AttributeError("'HtmlWriterPublishPartsTestCase' object has no attribute '_testMethodName'") raised in repr()] HtmlWriterPublishPartsTestCase object at 0x7fbd54e7e550>
args = ('test_publish',), kwargs = {}

    def __init__(self, *args, **kwargs):
        if 'writer_name' in kwargs:
            self.writer_name = kwargs['writer_name']
            del kwargs['writer_name']
>       CustomTestCase.__init__(self, *args, **kwargs)
E       TypeError: __init__() missing 3 required positional arguments: 'input', 'expected', and 'id'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/DocutilsTestSupport.py:673: TypeError
______________________________________________________ ERROR at setup of Html5WriterPublishPartsTestCase.test_publish ______________________________________________________

self = <[AttributeError("'Html5WriterPublishPartsTestCase' object has no attribute '_testMethodName'") raised in repr()] Html5WriterPublishPartsTestCase object at 0x7fbd75348160>
args = ('test_publish',), kwargs = {}

    def __init__(self, *args, **kwargs):
        if 'writer_name' in kwargs:
            self.writer_name = kwargs['writer_name']
            del kwargs['writer_name']
>       CustomTestCase.__init__(self, *args, **kwargs)
E       TypeError: __init__() missing 3 required positional arguments: 'input', 'expected', and 'id'

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/DocutilsTestSupport.py:673: TypeError
================================================================================= FAILURES =================================================================================
_____________________________________________________________ CommandLineEncodingTests.test_sys_argv_decoding ______________________________________________________________

self = <docutils.frontend.OptionParser object at 0x7fbd54bba550>, args = ['-ra', '--source-url=test.txt', '--title=Dornröschen']
values = <Values at 0x7fbd54bba2e0: {'title': None, 'generator': True, 'datestamp': "%Y-%m-%d %H:%M UTC (If you see this in tes..._visitor': None, '_disable_config': None, '_source': None, '_destination': None, '_config_files': ['./docutils.conf']}>

    def parse_args(self, args=None, values=None):
        """
        parse_args(args : [string] = sys.argv[1:],
                   values : Values = None)
        -> (values : Values, args : [string])

        Parse the command-line options found in 'args' (default:
        sys.argv[1:]).  Any errors result in a call to 'error()', which
        by default prints the usage message to stderr and calls
        sys.exit() with an error message.  On success returns a pair
        (values, args) where 'values' is a Values instance (with all
        your option values) and 'args' is the list of arguments left
        over after parsing options.
        """
        rargs = self._get_args(args)
        if values is None:
            values = self.get_default_values()

        # Store the halves of the argument list as attributes for the
        # convenience of callbacks:
        #   rargs
        #     the rest of the command-line (the "r" stands for
        #     "remaining" or "right-hand")
        #   largs
        #     the leftover arguments -- ie. what's left after removing
        #     options and their arguments (the "l" stands for "leftover"
        #     or "left-hand")
        self.rargs = rargs
        self.largs = largs = []
        self.values = values

        try:
>           stop = self._process_args(largs, rargs, values)

/usr/lib64/python3.8/optparse.py:1387:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <docutils.frontend.OptionParser object at 0x7fbd54bba550>, largs = [], rargs = ['--source-url=test.txt', '--title=Dornröschen']
values = <Values at 0x7fbd54bba2e0: {'title': None, 'generator': True, 'datestamp': "%Y-%m-%d %H:%M UTC (If you see this in tes..._visitor': None, '_disable_config': None, '_source': None, '_destination': None, '_config_files': ['./docutils.conf']}>

    def _process_args(self, largs, rargs, values):
        """_process_args(largs : [string],
                         rargs : [string],
                         values : Values)

        Process command-line arguments and populate 'values', consuming
        options and arguments from 'rargs'.  If 'allow_interspersed_args' is
        false, stop at the first non-option argument.  If true, accumulate any
        interspersed non-option arguments in 'largs'.
        """
        while rargs:
            arg = rargs[0]
            # We handle bare "--" explicitly, and bare "-" is handled by the
            # standard arg handler since the short arg case ensures that the
            # len of the opt string is greater than 1.
            if arg == "--":
                del rargs[0]
                return
            elif arg[0:2] == "--":
                # process a single long option (possibly with value(s))
                self._process_long_opt(rargs, values)
            elif arg[:1] == "-" and len(arg) > 1:
                # process a cluster of short options (possibly with
                # value(s) for the last one only)
>               self._process_short_opts(rargs, values)

/usr/lib64/python3.8/optparse.py:1431:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <docutils.frontend.OptionParser object at 0x7fbd54bba550>, rargs = ['--source-url=test.txt', '--title=Dornröschen']
values = <Values at 0x7fbd54bba2e0: {'title': None, 'generator': True, 'datestamp': "%Y-%m-%d %H:%M UTC (If you see this in tes..._visitor': None, '_disable_config': None, '_source': None, '_destination': None, '_config_files': ['./docutils.conf']}>

    def _process_short_opts(self, rargs, values):
        arg = rargs.pop(0)
        stop = False
        i = 1
        for ch in arg[1:]:
            opt = "-" + ch
            option = self._short_opt.get(opt)
            i += 1                      # we have consumed a character

            if not option:
                raise BadOptionError(opt)
            if option.takes_value():
                # Any characters left in arg?  Pretend they're the
                # next arg, and stop consuming characters of arg.
                if i < len(arg):
                    rargs.insert(0, arg[i:])
                    stop = True

                nargs = option.nargs
                if len(rargs) < nargs:
                    self.error(ngettext(
                        "%(option)s option requires %(number)d argument",
                        "%(option)s option requires %(number)d arguments",
                        nargs) % {"option": opt, "number": nargs})
                elif nargs == 1:
                    value = rargs.pop(0)
                else:
                    value = tuple(rargs[0:nargs])
                    del rargs[0:nargs]

            else:                       # option doesn't take a value
                value = None

>           option.process(opt, value, values, self)

/usr/lib64/python3.8/optparse.py:1536:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Option at 0x7fbd54edfd90: -r/--report>, opt = '-r', value = 'a'
values = <Values at 0x7fbd54bba2e0: {'title': None, 'generator': True, 'datestamp': "%Y-%m-%d %H:%M UTC (If you see this in tes..._visitor': None, '_disable_config': None, '_source': None, '_destination': None, '_config_files': ['./docutils.conf']}>
parser = <docutils.frontend.OptionParser object at 0x7fbd54bba550>

    def process(self, opt, value, values, parser):
        """
        Call the validator function on applicable settings and
        evaluate the 'overrides' option.
        Extends `optparse.Option.process`.
        """
>       result = optparse.Option.process(self, opt, value, values, parser)

/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/frontend.py:354:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Option at 0x7fbd54edfd90: -r/--report>, opt = '-r', value = 'a'
values = <Values at 0x7fbd54bba2e0: {'title': None, 'generator': True, 'datestamp': "%Y-%m-%d %H:%M UTC (If you see this in tes..._visitor': None, '_disable_config': None, '_source': None, '_destination': None, '_config_files': ['./docutils.conf']}>
parser = <docutils.frontend.OptionParser object at 0x7fbd54bba550>

    def process(self, opt, value, values, parser):

        # First, convert the value(s) to the right type.  Howl if any
        # value(s) are bogus.
>       value = self.convert_value(opt, value)

/usr/lib64/python3.8/optparse.py:779:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Option at 0x7fbd54edfd90: -r/--report>, opt = '-r', value = 'a'

    def convert_value(self, opt, value):
        if value is not None:
            if self.nargs == 1:
>               return self.check_value(opt, value)

/usr/lib64/python3.8/optparse.py:771:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Option at 0x7fbd54edfd90: -r/--report>, opt = '-r', value = 'a'

    def check_value(self, opt, value):
        checker = self.TYPE_CHECKER.get(self.type)
        if checker is None:
            return value
        else:
>           return checker(self, opt, value)

/usr/lib64/python3.8/optparse.py:766:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

option = <Option at 0x7fbd54edfd90: -r/--report>, opt = '-r', value = 'a'

    def check_choice(option, opt, value):
        if value in option.choices:
            return value
        else:
            choices = ", ".join(map(repr, option.choices))
>           raise OptionValueError(
                _("option %s: invalid choice: %r (choose from %s)")
                % (opt, value, choices))
E           optparse.OptionValueError: option -r: invalid choice: 'a' (choose from 'info', '1', 'warning', '2', 'error', '3', 'severe', '4', 'none', '5')

/usr/lib64/python3.8/optparse.py:440: OptionValueError

During handling of the above exception, another exception occurred:

self = <test_command_line.CommandLineEncodingTests testMethod=test_sys_argv_decoding>

    def test_sys_argv_decoding(self):
        if argv_encoding == 'ascii': # cannot test
            return
        sys.argv.append('--source-url=test.txt') # pure ASCII argument
        if sys.version_info < (3, 0):
            sys.argv.append(u'--title=Dornröschen'.encode(argv_encoding))
        else:
            sys.argv.append(u'--title=Dornröschen')
        publisher = docutils.core.Publisher()
>       publisher.process_command_line()

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_command_line.py:41:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/core.py:162: in process_command_line
    self.settings = option_parser.parse_args(argv)
/usr/lib64/python3.8/optparse.py:1389: in parse_args
    self.error(str(err))
/usr/lib64/python3.8/optparse.py:1569: in error
    self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <docutils.frontend.OptionParser object at 0x7fbd54bba550>, status = 2
msg = "pytest: error: option -r: invalid choice: 'a' (choose from 'info', '1', 'warning', '2', 'error', '3', 'severe', '4', 'none', '5')\n"

    def exit(self, status=0, msg=None):
        if msg:
            sys.stderr.write(msg)
>       sys.exit(status)
E       SystemExit: 2

/usr/lib64/python3.8/optparse.py:1559: SystemExit
--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
Usage
=====
  pytest [options]

pytest: error: option -r: invalid choice: 'a' (choose from 'info', '1', 'warning', '2', 'error', '3', 'severe', '4', 'none', '5')
________________________________________________________________ reCommonMarkParserTests.test_parsing_error ________________________________________________________________

self = <test_parsers.test_recommonmark.test_misc.reCommonMarkParserTests testMethod=test_parsing_error>

    @unittest.skipUnless(recommonmark_wrapper.CommonMarkParser, skip_msg)
    def test_parsing_error(self):
>       output = publish_string(sample1, parser_name='recommonmark',
                                settings_overrides={'warning_stream': ''})

/home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_parsers/test_recommonmark/test_misc.py:55:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/core.py:407: in publish_string
    output, pub = publish_programmatically(
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/core.py:665: in publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/core.py:217: in publish
    self.document = self.reader.read(self.source, self.parser,
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/readers/__init__.py:72: in read
    self.parse()
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/readers/__init__.py:78: in parse
    self.parser.parse(self.input, document)
/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/parsers/recommonmark_wrapper.py:117: in parse
    if node['level'] != section_level:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <section "title": <title...><paragraph...>>, key = 'level'

    def __getitem__(self, key):
        if isinstance(key, basestring):
>           return self.attributes[key]
E           KeyError: 'level'

/home/tkloczko/rpmbuild/BUILDROOT/python-docutils-0.17.1-2.fc35.x86_64/usr/lib/python3.8/site-packages/docutils/nodes.py:652: KeyError
============================================================================= warnings summary =============================================================================
test/test_parsers/test_recommonmark/test_section_headers.py:37
  /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_parsers/test_recommonmark/test_section_headers.py:37: DeprecationWarning: invalid escape sequence \
    """\

test/test_parsers/test_recommonmark/test_section_headers.py:50
  /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_parsers/test_recommonmark/test_section_headers.py:50: DeprecationWarning: invalid escape sequence \
    """\

test/test_parsers/test_recommonmark/test_section_headers.py:164
  /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_parsers/test_recommonmark/test_section_headers.py:164: DeprecationWarning: invalid escape sequence \
    """\

test/test_transforms/test___init__.py:20
  /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/test_transforms/test___init__.py:20: PytestCollectionWarning: cannot collect test class 'TestTransform' because it has a __init__ constructor (from: test/test_transforms/test___init__.py)
    class TestTransform(transforms.Transform):

test/test_settings.py::ConfigFileTests::test_old
test/test_settings.py::ConfigFileTests::test_old_and_new
test/test_settings.py::ConfigEnvVarFileTests::test_old_and_new
test/test_settings.py::ConfigEnvVarFileTests::test_old
  /home/tkloczko/rpmbuild/BUILD/docutils-0.17.1/test/data/config_old.txt:0: ConfigDeprecationWarning:
  The "[option]" section is deprecated.  Support for old-format configuration
  files may be removed in a future Docutils release.  Please revise your
  configuration files.  See <http://docutils.sf.net/docs/user/config.html>,
  section "Old-Format Configuration Files".

test/test_parsers/test_recommonmark/test_misc.py::reCommonMarkParserTests::test_raw_disabled_inline
test/test_parsers/test_recommonmark/test_misc.py::reCommonMarkParserTests::test_raw_disabled
test/test_parsers/test_recommonmark/test_misc.py::reCommonMarkParserTests::test_parsing_error
  /usr/lib/python3.8/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
    warn("Container node skipped: type={0}".format(mdnode.t))

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [1] test/test_parsers/test_recommonmark/test_misc.py:91: recommonmark_wrapper: parser found, fallback not used
ERROR test/test_functional.py::FunctionalTestCase::test - KeyError: 'configfile'
ERROR test/test_language.py::LanguageTestCase::test_directives - KeyError: 'language'
ERROR test/test_language.py::LanguageTestCase::test_roles - KeyError: 'language'
ERROR test/test_language.py::LanguageTestCase::test_bibliographic_fields - KeyError: 'language'
ERROR test/test_language.py::LanguageTestCase::test_labels - KeyError: 'language'
ERROR test/test_writers/test_html5_polyglot_parts.py::HtmlWriterPublishPartsTestCase::test_publish - TypeError: __init__() missing 3 required positional arguments: 'inpu...
ERROR test/test_writers/test_html5_polyglot_parts.py::Html5WriterPublishPartsTestCase::test_publish - TypeError: __init__() missing 3 required positional arguments: 'inp...
FAILED test/test_command_line.py::CommandLineEncodingTests::test_sys_argv_decoding - SystemExit: 2
FAILED test/test_parsers/test_recommonmark/test_misc.py::reCommonMarkParserTests::test_parsing_error - KeyError: 'level'
===================================================== 2 failed, 229 passed, 1 skipped, 11 warnings, 7 errors in 10.64s =====================================================
~~~


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [PATCH] Revert FutureWarnings.

[Guenter M. <mi...@us...>]

Only add warnigns to core functions, if users can avoid this
warnings following a recommended practice.

For core API functions and classes: don't change the behaviour
but replace with new function(s) showing new behaviour.
This way users can switch to the new behaviour at some stage during the
transition period.
Add a deprecation warning once the new functions are in place.

# Suggestion for the next step to solve the "string I/O" problem.
# OK to commit?
---
 docutils/docutils/core.py |  2 --
 docutils/docutils/io.py   | 27 +++++++++++++--------------
 2 files changed, 13 insertions(+), 16 deletions(-)

diff --git a/docutils/docutils/core.py b/docutils/docutils/core.py
index a54ed7a90..edbcfc3df 100644
--- a/docutils/docutils/core.py
+++ b/docutils/docutils/core.py
@@ -443,8 +443,6 @@ def publish_string(source, source_path=None, destination_path=None,
 
     Parameters: see `publish_programmatically`.
     """
-    warnings.warn('The return type of publish_string will change to '
-                  '"str" from Docutils 0.21.', FutureWarning, stacklevel=2)
     output, pub = publish_programmatically(
         source_class=io.StringInput, source=source, source_path=source_path,
         destination_class=io.StringOutput,
diff --git a/docutils/docutils/io.py b/docutils/docutils/io.py
index 53e93886a..12342025d 100644
--- a/docutils/docutils/io.py
+++ b/docutils/docutils/io.py
@@ -247,6 +247,11 @@ class Output(TransformSpec):
         raise NotImplementedError
 
     def encode(self, data):
+        """Encode `data` with `self.encoding` (unless it is already encoded).
+
+        If `self.encoding` is set to the pseudo encoding name "unicode",
+        `data` must be a `str` instance and is returned unchanged.
+        """
         if self.encoding and self.encoding.lower() == 'unicode':
             assert isinstance(data, str), (
                 'the encoding given is "unicode" but the output is not '
@@ -581,7 +586,7 @@ class StringInput(Input):
     default_source_path = '<string>'
 
     def read(self):
-        """Return the source as `str` instance.
+        """Return the source as `str` object.
 
         Decode, if required (see `Input.decode`).
         """
@@ -589,26 +594,20 @@ class StringInput(Input):
 
 
 class StringOutput(Output):
-
-    """
-    Direct string output.
-    """
+    """Output to a `bytes` or `str` instance."""
 
     default_destination_path = '<string>'
 
     def write(self, data):
-        """Encode `data`, store it in `self.destination`, and return it."""
+        """Encode `data`, store it in `self.destination`, and return it.
+
+        If `self.encoding` is set to the pseudo encoding name "unicode",
+        `data` must be a `str` instance and is returned unchanged.
+        (cf. `Output.encode`)
+        """
         self.destination = self.encode(data)
         return self.destination
 
-    def encode(self, data):
-        data = super().encode(data)
-        if not isinstance(data, str):
-            warnings.warn("StringOutput.encode()'s return type will change to "
-                          f'``str`` from Docutils 0.21, got type {type(data)}',
-                          FutureWarning, stacklevel=2)
-        return data
-
 
 class NullInput(Input):
 
-- 
2.30.2

Re: [Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

Dear Adam, dear Docutils developers,

On 2022-11-06, Adam Turner wrote:

>> r9208 ... r9211:
...
> I saw these as fairly unobtrusive changes that didn't touch the core
> ``io`` classes -- thank you for noting the incompatibilities with input
> encoding, though I think this reveals some deficiencies in our test
> suite -- I run all my changes against the test suite on Python 3.7,
> 3.8, 3.9, 3.10, and 3.11 before committing.

Yes, the test suite is incomplete: it does not test all fields of the
"syntax - settings - components" matrix.

This is why non-breaking tests are no guarantee for backwards compatibility
and one reason why a review of all commits is so important.

I added some more tests for input encoding handling in included files in
r9220.

> -----

>>> For the "include" and "csv-table" directives, the changes definitely change
>>> the behaviour for the "input_encoding" default value!
...
>> Checked r9208 ... r9211 and reverted the incompatible changes in r9217.

> I should like to revert the reversion, though I shall look to see what
> needs to be done to address the custom input encoding logic.

Handling of the character encoding should be kept synchronous for the
main source and included files.

As long as the custom classes from `docutils.io` are used for the main
document, there is no much sense in replicating their behaviour for
included files.

If using the "with" statement is viewed as an important bonus, adding a
context manager to the custom io classes could be considered
(after the test suite refactoring is finished).


> -----

>>>> The naming of the `core.publish_string()` API function
...

>> IMV, the name `publish_string` uses the generic notation of a "string"
>> as "sequence of alphanumeric text or other characters, either as a
>> literal constant or as some kind of variable",
>> cf. https://en.wikipedia.org/wiki/String_(computer_science).

> I see your point with respect to "string", and it is an overloaded
> term. My primary argument is to do the "most obvious" (or "least
> surprising") thing given the domain context -- I firmly believe that in
> the modern Python 3 ecosystem, when advertising a method with "string"
> in the name we should return strings as defined by Python, which are
> sequences of Unicode code points.

I see your point. OTOH:

* `output-encoding`__ is a *general* setting defined as
  "The text encoding for output". 
  This raises the expectation that all Docutils output "has" the
  specified encoding and makes the "most obvious" return value a bit less
  obvious.
    
  __  https://docutils.sourceforge.io/docs/user/config.html#output-encoding 

* The behaviour of `publish_string()` has been stable for many years.
  Long-time users are familiar with it and expect it to remain stable.

>>> Regarding the "core.publish_string()" function, I see three possibilities:

>> Alternatives forward:

>>> 1. Just keep as is.

>>>    This is core API behaviour, so we need a very strong indication
>>>    that the advantages outweigh the hassle.

>>> 2. Add a new boolean argument: "encode". 
...

>>> 3. Deprecate¹ "publish_string()" in favour of new, separate
>>>    "publish_unicode_str()" and "publish_bytes()" functions.

>>>    ¹PendingDeprecationWarning now, DeprecationWarning in Docutils 1.0.

>>   4. Keep "publish_string()" as-is.
>>      New function "publish_str_instance()", say.

>>      The "output_encoding" setting will be used for encoding declarations
>>      and to determine required character replacements in the LaTeX writer
>>      but the return value is guaranteed to be a `str` instance.


>> My current favourites are 1 or 4.

> By (1) do you mean keep it as is as of [r9217], where returning bytes
> from "publish_string" is deprecated? 

> If so, I would vote this option.
> My rationale is as above, I think for the ordinary Python programmer,
> one would expect a "string" return type. Anecdotally, I found this
> behaviour confusing when I first used Docutils.

By 1), I mean keeping the behaviour in 0.18, without any changes (except
for a more clear documentation). This would be the simplest alternative
for experienced Docutils users.

If the documentation is clear about possible return values (and even more
after adding type hints) users should be able to live with the
unfortunate naming.

OTOH, I see a use-case for a convenience function returning a `str`
instance also in cases where an "intended encoding" of the output is
given in the "output-encoding" setting. This way, a program using this
function can export a HTML, XML or LaTeX with an encoding declaration
as `str` instance, post-process and finally encode it before handing it
to storage or a non-Python processor.



If we are going to change the core API functionality regarding the
convenience function(s) to publish the output as `str` or `bytes`
instance, then we should:

* do not start this in the middle of a major refactoring of the test suite
  (where it is hard to spot the changes in expected output from 
  "cosmetic" changes in the test code).

* do it in a "quasi static" manner: both, old and the new behaviour must
  be accessible over a sequence of two or more stable releases.

Variant 4) has the disadvantage that the "wart" of the
ambiguous/misleading name is not adressed.

Changing the behaviour of `publish_string()` breaks existing code and the
expectations of long-term users.

This brings us back to variant 3):

  Define 2 new functions with non-ambiguous names, e.g.,
  `publish_str_instance()` and `publish_bytes_instance()` that
  return the output as a `str` or `bytes` instance respectively.
  
  Document, that the "output-encoding" setting is used for 
  encoding declarations in HTML, XML, and LaTeX also with
  `publish_str_instance()`. It is up to the user to ensure the declared
  encoding and the actual encoding of output stored in a file or passed to
  a non-Python program match. 
  
  Support "output-encoding" value ``''`` (and for backwards compatibility
  'unicode') with `publish_str_instance()`: In this case there is no
  encoding declaration in XML, HTML, and LaTeX.

  Deprecate "publish_string()":
  PendingDeprecationWarning now, DeprecationWarning in Docutils 1.0.

...

>> I don't think we need a new custom "BytesOutput" class when in the long run
>> we will use standard io classes.

> By this, do you mean "BytesIO" et al? I think using this would be
> sensible, introducing the new class is (as I see it) a bridging
> mechanism, as dropping in "BytesIO" does not fit the expected API at
> the moment.

I see. The custom `docutils.io` classes are deeply embedded in the
overall structure of Docutils components and their API. 
Any move to change/replace them touches Docutils Design Specification
(PEP 258) and should be well justified and well-considered, i.e. nothing
for the near future.

This means that if we want to introduce an explicit `publish_bytes()`
convenience function, a corresponding `BytesOutput` class is appropriate.
Implementation details would need further discussion.
Again, I'd prefer not to do this in the middle of a major refactoring of
the test suite.



>>>>>> r9178
...
> I will look to add back the quicktest method, I think it might be
> valuable to keep the Python method too as it shows how to achieve the
> same result only using the Python API.

Maybe as a footnote or side-box, so it is less distracting from the main
course of the discussion.


>> r9207
>> There is an established format in the HISTORY file:
...

> Thank you for making me aware of this. What should be done for changes
> that are cross-cutting, e.g. the test changes where the same change is
> being made to many test files, though perhaps not in the same
> directory?

The test suite refactoring could be listed under the "general" changes or
under "test/*" (or, if you prefer, "test/**").

> To explain my original approach, I am more used to an approach which
> enumerates history by patch or changeset, so for example describing a
> set of changes and the rationale / effect, rather than enumerating by
> filename.

I did not invent the HISTORY file format, but I learned to accept and
value it as complimentary to the more detailled and chronological SVN log.


>> On 2022-11-03, Guenter Milde via Docutils-develop wrote:

>> New discovered bug with ``output_encoding == "unicode"``.

>> r9202 contains a "quick and dirty fix" for a hithero unrecognized problem
>> with the pseudo-encoding name "unicode":

>> This non-standard name is used in encoding declarations in XML and HTML and
>> in a package call in LaTeX.

>> The problem is, that we don't know which encoding an application calling
>> `publish_string()` will finally use for storing/transferring the output
>> outside the realms of Python.
>> Given this, an encoding declaration with hard-coded "utf-8" is worse than no
>> encoding declaration.

> OK, your fix seems reasonable. I would note though that LaTeX has used
> UTF-8 as the default encoding since 2018, so no "inputenc" and a
> declaration of UTF-8 encoding have exactly the same effect.



>> r9202
>> ... bug with ``output_encoding == "unicode"``.

>> This non-standard name is used in encoding declarations in XML and HTML and
>> in a package call in LaTeX.

>> The problem is, that we don't know which encoding an application calling
>> `publish_string()` will finally use for storing/transferring the output
>> outside the realms of Python.
>> Given this, an encoding declaration with hard-coded "utf-8" is worse than no
>> encoding declaration.

> OK, your fix seems reasonable. 

Committed in r9219, together with the planned dropping of the "inputenc"
call with UTF-8 encoded output.

> I would note though that LaTeX has used UTF-8 as the default encoding
> since 2018, so no "inputenc" and a declaration of UTF-8 encoding have
> exactly the same effect.

The difference is, that the case of encoding the output to something
other than utf-8 is easier to handle, e.g., with 
``setting_overrides[latex-preamble] = '\usepackage[latin1]{inputenc}\n'``.
https://docutils.sourceforge.io/docs/user/config.html#latex-preamble 

I will follow up with a fix dropping the encoding declaration for HTML
and XML output if "output_encoding" is "unicode".


> -----

>> In the unit tests, I suggest decoding the `bytes` returned by
>> `publish_string()` over the "unicode" pseudo-encoding, e.g.

>> diff --git a/docutils/test/DocutilsTestSupport.py b/docutils/test/DocutilsTestSupport.py

> This is to reflect where "publish_string" will end up (returning Python
> strings), but I don't see a large difference between the two approaches
> so happy to switch if you would prefer.

As discusses above, I don't think it is a good idea to change the
behaviour of `publish_string()` (but eventually replace it by 2 new
functions in the long run).

Comitted in r9218.


> Your documentation patch looks OK.

Commited as part of r9221


Günter

Re: [Docutils-develop] Recent commit activity

[Adam T. <aat...@ou...>]

Dear Günter,

I am replying to all three notes in the below, apologies for the rather long note that follows.

-----

>>> (In the long run, I'd like to replace the "homegrown" docutils.io classes
>>> with standard objects, but this is a separate topic.)

> r9208 ... r9211:
> "In the long run" is not next day! Please ask before such changes.

I saw these as fairly unobtrusive changes that didn't touch the core ``io`` classes -- thank you for noting the incompatibilities with input encoding, though I think this reveals some deficiencies in our test suite -- I run all my changes against the test suite on Python 3.7, 3.8, 3.9, 3.10, and 3.11 before committing.

-----

>> Replacing the "FileInput/StringInput/FileOutput" classes can currently
>> only be done in cases, where we are sure not to introduce backwards
>> incompatibilities.

>> For the "include" and "csv-table" directives, the changes definitely change
>> the behaviour for the "input_encoding" default value!

>> For the other changes, I am not sure.

> Checked r9208 ... r9211 and reverted the incompatible changes in r9217.

I should like to revert the reversion, though I shall look to see what needs to be done to address the custom input encoding logic.

-----

>>> The naming of the `core.publish_string()` API function pre-dates Python3
>>> and uses the term "string" as a superordinate for "Unicode strings"
>>> (`str`/`unicode`) and "encoded strings" (`bytes`/`str`).
>>> The documentation is explicit about the return value
>>> (defaulting to 'utf-8' encoded `bytes` instances).

>>> The function is part of the "first level API": intended for use by
>>> 3rd-party code. Changing the return value has the potential of many
>>> breaks. 
>>> Please revert the changes to core.py and io.py.

>> I have reverted the change (thank you) but introduced a deprecation for
>> returning non-(unicode) strings from "publish_string". I feel this is a
>> reasonable step towards having a more intuitive API.

> IMV, the name `publish_string` uses the generic notation of a "string"
> as "sequence of alphanumeric text or other characters, either as a
> literal constant or as some kind of variable",
> cf. https://en.wikipedia.org/wiki/String_(computer_science).

I see your point with respect to "string", and it is an overloaded term. My primary argument is to do the "most obvious" (or "least surprising") thing given the domain context -- I firmly believe that in the modern Python 3 ecosystem, when advertising a method with "string" in the name we should return strings as defined by Python, which are sequences of Unicode code points.

-----

>> Naming of "publish_bytes" and "BytesOutput" I am not that set on, we
>> can change to "BinaryXXX" if preferred.

> I don't think we need a new custom "BytesOutput" class when in the long run
> we will use standard io classes.

By this, do you mean "BytesIO" et al? I think using this would be sensible, introducing the new class is (as I see it) a bridging mechanism, as dropping in "BytesIO" does not fit the expected API at the moment.

-----

>> Regarding the "core.publish_string()" function, I see three possibilities:

> Alternatives forward:

>> 1. Just keep as is.

>>    This is core API behaviour, so we need a very strong indication
>>    that the advantages outweigh the hassle.

>> 2. Add a new boolean argument: "encode". 

>>    With ``publish_string(encode=False)``, the "output_encoding" setting
>>    would be ignored and the return value is a `bytes` object.

>>    With ``publish_string(encode=True)``, the behaviour is as currently,
>>    encoding with "output_encoding" or returning `bytes` or `str` object
>>    (the latter for ``output_encoding == "unicode"``).

>>    The default value could change from True to False in Docutils 2.0.

>> 3. Deprecate¹ "publish_string()" in favour of new, separate
>>    "publish_unicode_str()" and "publish_bytes()" functions.

>>    ¹PendingDeprecationWarning now, DeprecationWarning in Docutils 1.0.

>   4. Keep "publish_string()" as-is.
>      New function "publish_str_instance()", say.

>      The "output_encoding" setting will be used for encoding declarations
>      and to determine required character replacements in the LaTeX writer
>      but the return value is guaranteed to be a `str` instance.
>     
>     
> My current favourites are 1 or 4.

By (1) do you mean keep it as is as of [r9217], where returning bytes from "publish_string" is deprecated? If so, I would vote this option. My rationale is as above, I think for the ordinary Python programmer, one would expect a "string" return type. Anecdotally, I found this behaviour confusing when I first used Docutils.

-----

>>>>> r9178
>>> 8 lines of doctest code seems overly verbose in the context of an
>>> "overview of the Docutils architecture".

>> I tend to agree, "core.publish_string" would be nicer, but there is no
>> way in Docutils to disable applying the default transforms.

>>> It can still be useful for development: testing the reST parser (including
>>> the option to generate input/output pairs for parser test scripts).
>>> I'd move it to tools/dev and remove the EasyDialogs part.

>> Done in [r9201]

> So now the "hacking" guide can again refer to "quicktest.py".

> BTW: the fact that it "has not been substantively modified since 2006" is
>      no argument for the move (the `cat` shell command, say, has not been
>      substantially modified for ages, too but moving it to a developers only
>      path would be bad.)

>      Rather, "quicktest.py" has never been installed into the "PATH"
>      in setup.py and is a tool for developers rather than end-users.

I will look to add back the quicktest method, I think it might be valuable to keep the Python method too as it shows how to achieve the same result only using the Python API.

-----

> r9207

> There is an established format in the HISTORY file:

> General changes, followed by changes listed in alphabetical order of
> affected file (or directory for related changes in one directory)

> Name patch authors only for "external" patches (where the name differs
> from the committer in the SVN log).

Thank you for making me aware of this. What should be done for changes that are cross-cutting, e.g. the test changes where the same change is being made to many test files, though perhaps not in the same directory?

To explain my original approach, I am more used to an approach which enumerates history by patch or changeset, so for example describing a set of changes and the rationale / effect, rather than enumerating by filename.

-----

> On 2022-11-03, Guenter Milde via Docutils-develop wrote:

> New discovered bug with ``output_encoding == "unicode"``.

> r9202 contains a "quick and dirty fix" for a hithero unrecognized problem
> with the pseudo-encoding name "unicode":

> This non-standard name is used in encoding declarations in XML and HTML and
> in a package call in LaTeX.

> The problem is, that we don't know which encoding an application calling
> `publish_string()` will finally use for storing/transferring the output
> outside the realms of Python.
> Given this, an encoding declaration with hard-coded "utf-8" is worse than no
> encoding declaration.

OK, your fix seems reasonable. I would note though that LaTeX has used UTF-8 as the default encoding since 2018, so no "inputenc" and a declaration of UTF-8 encoding have exactly the same effect.

-----

> In the unit tests, I suggest decoding the `bytes` returned by
> `publish_string()` over the "unicode" pseudo-encoding, e.g.

> diff --git a/docutils/test/DocutilsTestSupport.py b/docutils/test/DocutilsTestSupport.py
> [snip]

This is to reflect where "publish_string" will end up (returning Python strings), but I don't see a large difference between the two approaches so happy to switch if you would prefer.

-----

> As an aside:

> The `docutils.io.StringInput` class can transparently handle input from
> `str` and `bytes` instances.
> The following patch makes this more explicit:

For similar reasons, I would prefer to make this more explicit and separate ``StringInput`` from a hypothetical ``BytesInput``. This can be discussed later though, when (if) we move away from all of the ``io`` classes.

Your documentation patch looks OK.

-----

> The only effect of setting ``input_encoding = 'unicode'`` is a warning if
> the input object is not a `str` instance. Given that 'unicode' is a
> non-standard encoding name that may be deprecated later, it is, IMV, not
> worth recommending this in the docstring to `publish_string`:

> diff --git a/docutils/docutils/core.py b/docutils/docutils/core.py
> [snip]

I agree with this.

-----

Thanks,
Adam

Re: [Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

On 2022-11-03, Guenter Milde via Docutils-develop wrote:
> On 2022-11-02, Adam Turner wrote:

> Replacing the "FileInput/StringInput/FileOutput" classes can currently
> only be done in cases, where we are sure not to introduce backwards
> incompatibilities.

> For the "include" and "csv-table" directives, the changes definitely change
> the behaviour for the "input_encoding" default value!

> For the other changes, I am not sure.

Checked r9208 ... r9211 and reverted the incompatible changes in r9217.

Günter

Re: [Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

Dear Adam,

a follow up with some more discoveries and thoughts...

On 2022-11-03, Guenter Milde via Docutils-develop wrote:
> On 2022-11-02, Adam Turner wrote:

>>>>> r9167
>> Partially reverted in [r9202]
...

> Regarding the "core.publish_string()" function ...

Alternatives forward:

> 1. Just keep as is.

>    This is core API behaviour, so we need a very strong indication
>    that the advantages outweigh the hassle.

> 2. Add a new boolean argument: "encode". 

>    With ``publish_string(encode=False)``, the "output_encoding" setting
>    would be ignored and the return value is a `bytes` object.

>    With ``publish_string(encode=True)``, the behaviour is as currently,
>    encoding with "output_encoding" or returning `bytes` or `str` object
>    (the latter for ``output_encoding == "unicode"``).

>    The default value could change from True to False in Docutils 2.0.

> 3. Deprecate¹ "publish_string()" in favour of new, separate
>    "publish_unicode_str()" and "publish_bytes()" functions.

>    ¹PendingDeprecationWarning now, DeprecationWarning in Docutils 1.0.

  4. Keep "publish_string()" as-is.
     New function "publish_str_instance()", say.

     The "output_encoding" setting will be used for encoding declarations
     and to determine required character replacements in the LaTeX writer
     but the return value is guaranteed to be a `str` instance.
     
     
My current favourites are 1 or 4.


New discovered bug with ``output_encoding == "unicode"``.

r9202 contains a "quick and dirty fix" for a hithero unrecognized problem
with the pseudo-encoding name "unicode": 

This non-standard name is used in encoding declarations in XML and HTML and
in a package call in LaTeX.

The problem is, that we don't know which encoding an application calling
`publish_string()` will finally use for storing/transferring the output
outside the realms of Python.
Given this, an encoding declaration with hard-coded "utf-8" is worse than no
encoding declaration.

A fix for LaTeX would be

diff --git a/docutils/docutils/writers/latex2e/__init__.py b/docutils/docutils/writers/latex2e/__init__.py
index 45086ea09..a306fef9a 100644
--- a/docutils/docutils/writers/latex2e/__init__.py
+++ b/docutils/docutils/writers/latex2e/__init__.py
@@ -1304,7 +1304,8 @@ class LaTeXTranslator(nodes.NodeVisitor):
         # ~~~~~~~~~~~~~~~~
         # Encodings:
         # Docutils' output-encoding => TeX input encoding
-        if self.latex_encoding != 'ascii':
+        if self.latex_encoding not in ('ascii', 'unicode'):
+            # TODO: also don't insert for 'utf8' (cf. RELEASE-NOTES)
             self.requirements['_inputenc'] = (r'\usepackage[%s]{inputenc}'
                                               % self.latex_encoding)
         # TeX font encoding
@@ -1459,8 +1460,7 @@
               # 'iso-8859-7': ''   # greek
               # 'iso-8859-8': ''   # hebrew
               # 'iso-8859-10': ''  # latin6, more complete iso-8859-4
-              'unicode': 'utf8',  # TEMPORARY, remove in Docutils 0.21
               }
         encoding = docutils_encoding.lower()


In the unit tests, I suggest decoding the `bytes` returned by
`publish_string()` over the "unicode" pseudo-encoding, e.g.

diff --git a/docutils/test/DocutilsTestSupport.py b/docutils/test/DocutilsTestSupport.py
index 496e68dd7..2df660d85 100644
--- a/docutils/test/DocutilsTestSupport.py
+++ b/docutils/test/DocutilsTestSupport.py
@@ -493,7 +493,7 @@ class WriterPublishTestCase(CustomTestCase, docutils.SettingsSpec):
 
     settings_default_overrides = {'_disable_config': True,
                                   'strict_visitor': True,
-                                  'output_encoding': 'unicode'}
+                                  }
     writer_name = ''  # set in subclasses or constructor
 
     def __init__(self, *args, writer_name='', **kwargs):
@@ -509,7 +509,11 @@ class WriterPublishTestCase(CustomTestCase, docutils.SettingsSpec):
               writer_name=self.writer_name,
               settings_spec=self,
               settings_overrides=self.suite_settings)
-        self.assertEqual(str(output), str(self.expected))
+        try:
+            output = output.decode()
+        except AttributeError:
+            pass
+        self.assertEqual(output, self.expected)
 
 
 class PublishTestSuite(CustomTestSuite):



As an aside:

The `docutils.io.StringInput` class can transparently handle input from
`str` and `bytes` instances. 
The following patch makes this more explicit:

diff --git a/docutils/docutils/io.py b/docutils/docutils/io.py
index 6714ca22b..53e93886a 100644
--- a/docutils/docutils/io.py
+++ b/docutils/docutils/io.py
@@ -576,15 +576,15 @@ class BytesOutput(Output):
 
 
 class StringInput(Input):
-
-    """
-    Direct string input.
-    """
+    """Input from a `str` or `bytes` instance."""
 
     default_source_path = '<string>'
 
     def read(self):
-        """Decode and return the source string."""
+        """Return the source as `str` instance.
+
+        Decode, if required (see `Input.decode`).
+        """
         return self.decode(self.source)
 

The only effect of setting ``input_encoding = 'unicode'`` is a warning if
the input object is not a `str` instance. Given that 'unicode' is a
non-standard encoding name that may be deprecated later, it is, IMV, not
worth recommending this in the docstring to `publish_string`:

diff --git a/docutils/docutils/core.py b/docutils/docutils/core.py
index 4d50e7fc7..798ffc5a1 100644
--- a/docutils/docutils/core.py
+++ b/docutils/docutils/core.py
@@ -437,10 +437,6 @@ def publish_string(source, source_path=None, destination_path=None,
 
         publish_string(..., settings_overrides={'output_encoding': 'unicode'})
 
-    Similarly for Unicode string input (`source`)::
-
-        publish_string(..., settings_overrides={'input_encoding': 'unicode'})
-
     Parameters: see `publish_programmatically`.
     """
     warnings.warn('The return type of publish_string will change to '


hope to hear from you soon,

Günter

[Docutils-develop] [docutils:patches] #197 MathML: support pandoc(1) as an external converter

[Günter M. <mi...@us...>]

- **status**: open --> open-accepted
- **Comment**:

The patch is committed in [r9216].
Thank you for your contribution.



---

** [patches:#197] MathML: support pandoc(1) as an external converter**

**Status:** open-accepted
**Group:** None
**Created:** Fri Oct 28, 2022 02:21 PM UTC by Ximin Luo
**Last Updated:** Tue Nov 01, 2022 11:44 AM UTC
**Owner:** nobody
**Attachments:**

- [0001-MathML-support-pandoc-1-as-an-external-converter.patch](https://sourceforge.net/p/docutils/patches/197/attachment/0001-MathML-support-pandoc-1-as-an-external-converter.patch) (3.5 kB; text/x-patch)


In my personal experience, `pandoc` is more reliable than `latexml`, e.g. the current support for the latter in docutils does not automatically include amsfonts which prevents things like mathbb from working. In `pandoc` it Just Works.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

is still too fast...

On 2022-11-02, Adam Turner wrote:

>> Generally, if a file is created or maintained by an active member of the
>> Docutils developer team, I'd prefer an "ask before change" (except for
>> emergencies and small typos).

> Sounds good!

...

>> (In the long run, I'd like to replace the "homegrown" docutils.io classes
>> with standard objects, but this is a separate topic.)

r9208 ... r9211:
"In the long run" is not next day! Please ask before such changes. 

Replacing the "FileInput/StringInput/FileOutput" classes can currently
only be done in cases, where we are sure not to introduce backwards
incompatibilities.

For the "include" and "csv-table" directives, the changes definitely change
the behaviour for the "input_encoding" default value!

For the other changes, I am not sure.


>>>> r9167
>> Introduces a backwards incompatible API change! (see below)

> Partially reverted in [r9202] (see below)

So we are half way. Please no new patches on top before this is sorted out.

...

>> The naming of the `core.publish_string()` API function pre-dates Python3
>> and uses the term "string" as a superordinate for "Unicode strings"
>> (`str`/`unicode`) and "encoded strings" (`bytes`/`str`).
>> The documentation is explicit about the return value
>> (defaulting to 'utf-8' encoded `bytes` instances).

>> The function is part of the "first level API": intended for use by
>> 3rd-party code. Changing the return value has the potential of many
>> breaks. 
>> Please revert the changes to core.py and io.py.

...

> I have reverted the change (thank you) but introduced a deprecation for
> returning non-(unicode) strings from "publish_string". I feel this is a
> reasonable step towards having a more intuitive API.

IMV, the name `publish_string` uses the generic notation of a "string" 
as "sequence of alphanumeric text or other characters, either as a
literal constant or as some kind of variable",
cf. https://en.wikipedia.org/wiki/String_(computer_science).


> Naming of "publish_bytes" and "BytesOutput" I am not that set on, we
> can change to "BinaryXXX" if preferred.

I don't think we need a new custom "BytesOutput" class when in the long run
we will use standard io classes.

Regarding the "core.publish_string()" function, I see three possibilities:

1. Just keep as is.

   This is core API behaviour, so we need a very strong indication
   that the advantages outweigh the hassle.

2. Add a new boolean argument: "encode". 

   With ``publish_string(encode=False)``, the "output_encoding" setting
   would be ignored and the return value is a `bytes` object.

   With ``publish_string(encode=True)``, the behaviour is as currently,
   encoding with "output_encoding" or returning `bytes` or `str` object
   (the latter for ``output_encoding == "unicode"``).
   
   The default value could change from True to False in Docutils 2.0.

3. Deprecate¹ "publish_string()" in favour of new, separate
   "publish_unicode_str()" and "publish_bytes()" functions.

   ¹PendingDeprecationWarning now, DeprecationWarning in Docutils 1.0.



>>>> r9178
>> 8 lines of doctest code seems overly verbose in the context of an
>> "overview of the Docutils architecture".

> I tend to agree, "core.publish_string" would be nicer, but there is no
> way in Docutils to disable applying the default transforms.

>> It can still be useful for development: testing the reST parser (including
>> the option to generate input/output pairs for parser test scripts).
>> I'd move it to tools/dev and remove the EasyDialogs part.

> Done in [r9201]

So now the "hacking" guide can again refer to "quicktest.py".

BTW: the fact that it "has not been substantively modified since 2006" is
     no argument for the move (the `cat` shell command, say, has not been
     substantially modified for ages, too but moving it to a developers only
     path would be bad.)

     Rather, "quicktest.py" has never been installed into the "PATH"
     in setup.py and is a tool for developers rather than end-users.

r9207

There is an established format in the HISTORY file:

General changes, followed by changes listed in alphabetical order of
affected file (or directory for related changes in one directory)

Name patch authors only for "external" patches (where the name differs
from the committer in the SVN log).


This is what I found at first glance, still not convinced this is all.

Günter

[Docutils-develop] [docutils:bugs] Re: #458 Two issues regarding manpage writer and .TH

[engelbert g. <gr...@us...>]

sorry , fixed


---

** [bugs:#458] Two issues regarding manpage writer and .TH**

**Status:** open
**Labels:** manpage writer 
**Created:** Mon Sep 05, 2022 11:13 AM UTC by Dmitry Shachnev
**Last Updated:** Wed Nov 02, 2022 11:30 PM UTC
**Owner:** nobody


Dear docutils developers,

I received two bug reports in Debian regarding the content of .TH macro in man pages:

First bug — https://bugs.debian.org/1016527 — asks that the value of `--date` argument be used for the third argument of `.TH`, instead of putting it on a separate line (Generated on: YYYY-MM-DD).

Second bug — https://bugs.debian.org/1018737 — asks that docutils not set fifth argument of `.TH` to empty string, but omit it completely and rely on the default value.

Please see those bugs for more detailed descriptions (and some discussion, in the first bug).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Recent commit activity

[Adam T. <aat...@ou...>]

Dear Günter,

>> The overall aim with the test changes is to be able to use the standard
>> library's ``unittest`` runner, which will alleviate some of the points
>> you raise.

> I understand and support the overall aim to use standard tools for the tests.
> I would have preferred a "feature branch" (as mandated by the "policies")
> for this change set, though. (I agree that with git this would have been
> easier and have to apologise for the delay of my respective reviews.)

I'll bear in mind for future projects, thank you.

> Generally, if a file is created or maintained by an active member of the
> Docutils developer team, I'd prefer an "ask before change" (except for
> emergencies and small typos).

Sounds good!

>> r9140, r9141: Import ``DocutilsTestSupport`` from ``test``

> When editing a test script, I regularly run it from my editor (it is
> just one click). This is a feature I don't want to miss, the actual
> implementation is less important ...

OK, I will look to ensure this is still possible.

>>> r9167
> Introduces a backwards incompatible API change! (see below)

Partially reverted in [r9202] (see below)

>>>   IMV, an encoded string ("bytes" object) is no "binary" output.

>> Python 3 bytes objects are sequences of 8-bit bytes, so I would say
>> that it is binary output, but perhaps we could improve here.

> The Python 3.10 documentation for `bytes` calls it a "binary sequence
> type", so I may have to adapt.

> The naming of the `core.publish_string()` API function pre-dates Python3
> and uses the term "string" as a superordinate for "Unicode strings"
> (`str`/`unicode`) and "encoded strings" (`bytes`/`str`).
> The documentation is explicit about the return value
> (defaulting to 'utf-8' encoded `bytes` instances).

> The function is part of the "first level API": intended for use by
> 3rd-party code. Changing the return value has the potential of many
> breaks. 
> Please revert the changes to core.py and io.py.

> Unicode `str` output can be achieved passing
> ``settings_overwrites={'output_encoding': 'unicode'}`` to
> publish_string().

> (In the long run, I'd like to replace the "homegrown" docutils.io classes
> with standard objects, but this is a separate topic.)

I have reverted the change (thank you) but introduced a deprecation for returning non-(unicode) strings from "publish_string". I feel this is a reasonable step towards having a more intuitive API.

Naming of "publish_bytes" and "BytesOutput" I am not that set on, we can change to "BinaryXXX" if preferred.

>>> r9168 Use ``.rstrip()`` instead

> This depends on the context: over-specified tests contribute to noise due to
> false positives. Test results should not depend on the "native" line endings
> of the system running the tests.

Python converts all line-endings to "\n" when opening a file in text-mode (or in string literals), so this should not be an issue. [1] [2]

[1]: https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files
[2]: https://peps.python.org/pep-0278/

>>> r9178
> 8 lines of doctest code seems overly verbose in the context of an
> "overview of the Docutils architecture".

I tend to agree, "core.publish_string" would be nicer, but there is no way in Docutils to disable applying the default transforms.

> It can still be useful for development: testing the reST parser (including
> the option to generate input/output pairs for parser test scripts).
> I'd move it to tools/dev and remove the EasyDialogs part.

Done in [r9201]

Thanks,
Adam

[Docutils-develop] [docutils:bugs] #458 Two issues regarding manpage writer and .TH

[Adam T. <aa-...@us...>]

@grubert 9 tests are now failing in `test_manpage.py` -- I think the test expectations need to be updated.

A


---

** [bugs:#458] Two issues regarding manpage writer and .TH**

**Status:** open
**Labels:** manpage writer 
**Created:** Mon Sep 05, 2022 11:13 AM UTC by Dmitry Shachnev
**Last Updated:** Wed Nov 02, 2022 05:11 PM UTC
**Owner:** nobody


Dear docutils developers,

I received two bug reports in Debian regarding the content of .TH macro in man pages:

First bug — https://bugs.debian.org/1016527 — asks that the value of `--date` argument be used for the third argument of `.TH`, instead of putting it on a separate line (Generated on: YYYY-MM-DD).

Second bug — https://bugs.debian.org/1018737 — asks that docutils not set fifth argument of `.TH` to empty string, but omit it completely and rely on the default value.

Please see those bugs for more detailed descriptions (and some discussion, in the first bug).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #458 Two issues regarding manpage writer and .TH

[engelbert g. <gr...@us...>]

Revision 9200 Does not output empty "manual" in ``.TH`` or the docinfo[Manual group]



---

** [bugs:#458] Two issues regarding manpage writer and .TH**

**Status:** open
**Labels:** manpage writer 
**Created:** Mon Sep 05, 2022 11:13 AM UTC by Dmitry Shachnev
**Last Updated:** Wed Nov 02, 2022 04:29 PM UTC
**Owner:** nobody


Dear docutils developers,

I received two bug reports in Debian regarding the content of .TH macro in man pages:

First bug — https://bugs.debian.org/1016527 — asks that the value of `--date` argument be used for the third argument of `.TH`, instead of putting it on a separate line (Generated on: YYYY-MM-DD).

Second bug — https://bugs.debian.org/1018737 — asks that docutils not set fifth argument of `.TH` to empty string, but omit it completely and rely on the default value.

Please see those bugs for more detailed descriptions (and some discussion, in the first bug).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #196 Revert header/footer borders in minimal.css

[Günter M. <mi...@us...>]

- **status**: open --> closed-wont-fix
- **Comment**:

Thank you for the patch and the explanations.




---

** [patches:#196]  Revert header/footer borders in minimal.css**

**Status:** closed-wont-fix
**Group:** None
**Created:** Wed Oct 26, 2022 10:57 PM UTC by Ximin Luo
**Last Updated:** Fri Oct 28, 2022 02:19 PM UTC
**Owner:** nobody
**Attachments:**

- [4.patch](https://sourceforge.net/p/docutils/patches/196/attachment/4.patch) (654 Bytes; text/x-patch)


These were uncommented in r8472 but that commit only talks about switching to html5 semantic tags, not changing the actual meaning of minimal.css

I assume this is a mistake; it interferes with the purpose of this stylesheet as a minimal stylesheet, and plain.css overrides this later anyway with border:none.



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #458 Two issues regarding manpage writer and .TH

[engelbert g. <gr...@us...>]

the values in the roff header .TH 
~~~
.TH title section date source manual
~~~
are taken from title and the docinfo
~~~
:Date:   2009-06-22
:Version: 0.0.1
:Manual section: 1
:Manual group: text processing
~~~
the Version is put into "source" ... don't know why I did this ?
No one ever complained.

So there should be a "Manual source" instead.
"Manual manual" sounds ... silly doesn't it so it is "Manual group"




---

** [bugs:#458] Two issues regarding manpage writer and .TH**

**Status:** open
**Labels:** manpage writer 
**Created:** Mon Sep 05, 2022 11:13 AM UTC by Dmitry Shachnev
**Last Updated:** Mon Oct 10, 2022 03:14 PM UTC
**Owner:** nobody


Dear docutils developers,

I received two bug reports in Debian regarding the content of .TH macro in man pages:

First bug — https://bugs.debian.org/1016527 — asks that the value of `--date` argument be used for the third argument of `.TH`, instead of putting it on a separate line (Generated on: YYYY-MM-DD).

Second bug — https://bugs.debian.org/1018737 — asks that docutils not set fifth argument of `.TH` to empty string, but omit it completely and rely on the default value.

Please see those bugs for more detailed descriptions (and some discussion, in the first bug).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #198 Add support for Python 3.11

[engelbert g. <gr...@us...>]

- **status**: open --> closed-accepted



---

** [patches:#198] Add support for Python 3.11**

**Status:** closed-accepted
**Group:** None
**Created:** Tue Nov 01, 2022 08:02 PM UTC by Hugo 
**Last Updated:** Tue Nov 01, 2022 08:02 PM UTC
**Owner:** nobody
**Attachments:**

- [add-3.11.patch](https://sourceforge.net/p/docutils/patches/198/attachment/add-3.11.patch) (1.1 kB; application/octet-stream)


Python 3.11 was [released on 2022-10-24](https://discuss.python.org/t/python-3-11-0-final-is-now-available/20291?u=hugovk) 🚀 

![image](https://user-images.githubusercontent.com/1324225/198233993-5b79523b-370f-4316-a4be-9403c8209068.png)


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #459 Invalid circular inclusion warning when including multiple documents from a directive

[Günter M. <mi...@us...>]

> I'm not sure the `lines = f.read()` quite works because I don't think that puts it into a nice array of lines for parsing ...

The `lines = f.read()` in my small example is a stub.
The point is, that if you insist on `statemachine.insert_input()` to inject generated lines to the document while parsing it, either use the currently active source or a "fancy description" as second argument, e.g.

    def run(self):
        name = self.content[0]
        self.state_machine.insert_input(
            ['.. include:: %s' % name], 'mytrickydirective')
        return []
This works in 0.18 but I can't tell for 0.17.


---

** [bugs:#459] Invalid circular inclusion warning when including multiple documents from a directive**

**Status:** pending-works-for-me
**Created:** Thu Oct 20, 2022 10:30 PM UTC by Ian Wienand
**Last Updated:** Wed Nov 02, 2022 10:46 AM UTC
**Owner:** nobody
**Attachments:**

- [circular-ref-bug.py](https://sourceforge.net/p/docutils/bugs/459/attachment/circular-ref-bug.py) (1.2 kB; text/x-python)


The recent sphinx release has found what I think is a problem in the circular-reference detection in docutils.

We have a situation where we build a page with a directive that automatically includes a file (readme from a sub-component).  In some cases, these readme's might then also include another file (e.g. similar components share a common file about their arguments, etc.).  So we have a include hierarchy like

~~~
 .. include:: ./component/README.rst
   .. include:: common.rst
.. include:: ./component2/README.rst
  .. include:: common.rst
~~~

Our directive works by reading the `README.rst` file and then using `self.state_machine.insert_input()`

Our build started failing due a `circular inclusion in "include" directive` which comes from the *second* inclusion of the `common.rst` above.  In this situation, I don't believe there is any circular inclusion -- we want the same content included twice; but it is not referencing itself.

I think it has something to do with `self.state.document.include_log` thinking that it has seen `common.rst` before.

I have isolated this down to the smallest example I can think of and attached that; it is also at https://paste.opendev.org/show/brTxHYllHebBIs1wHbfM/

When I run this I get

~~~
$ ./docutils-venv/bin/python3 ./recursive.py 
Write combined.rst
Write file1.rst
Write file2.rst
Write common.rst
file1.rst:1: (WARNING/2) circular inclusion in "include" directive:
file1.rst
> file1.rst
~~~



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #459 Invalid circular inclusion warning when including multiple documents from a directive

[Günter M. <mi...@us...>]

> Additionally, this only seems to occur with docutils 0.17 -- if I downgrade to 0.16 or upgrade to 0.18 I don't the same build error! 

This is very important information, because  [bugs:#366] was reported for 0.17 and fixed in 0.18.

The bug report described cases of "false positives" in the circular inclusion detection, reproducible in stock Docutils for specific input cases. Maybe your problem is triggered by the same fault to the 0.17 version of the detection algorithm.
The fix includes incompatible changes to the "manual management" of the `include_log` required around an `insert_input` call. This means any fix on your side should be tested across Docutils versions :(


---

** [bugs:#459] Invalid circular inclusion warning when including multiple documents from a directive**

**Status:** pending-works-for-me
**Created:** Thu Oct 20, 2022 10:30 PM UTC by Ian Wienand
**Last Updated:** Wed Nov 02, 2022 07:14 AM UTC
**Owner:** nobody
**Attachments:**

- [circular-ref-bug.py](https://sourceforge.net/p/docutils/bugs/459/attachment/circular-ref-bug.py) (1.2 kB; text/x-python)


The recent sphinx release has found what I think is a problem in the circular-reference detection in docutils.

We have a situation where we build a page with a directive that automatically includes a file (readme from a sub-component).  In some cases, these readme's might then also include another file (e.g. similar components share a common file about their arguments, etc.).  So we have a include hierarchy like

~~~
 .. include:: ./component/README.rst
   .. include:: common.rst
.. include:: ./component2/README.rst
  .. include:: common.rst
~~~

Our directive works by reading the `README.rst` file and then using `self.state_machine.insert_input()`

Our build started failing due a `circular inclusion in "include" directive` which comes from the *second* inclusion of the `common.rst` above.  In this situation, I don't believe there is any circular inclusion -- we want the same content included twice; but it is not referencing itself.

I think it has something to do with `self.state.document.include_log` thinking that it has seen `common.rst` before.

I have isolated this down to the smallest example I can think of and attached that; it is also at https://paste.opendev.org/show/brTxHYllHebBIs1wHbfM/

When I run this I get

~~~
$ ./docutils-venv/bin/python3 ./recursive.py 
Write combined.rst
Write file1.rst
Write file2.rst
Write common.rst
file1.rst:1: (WARNING/2) circular inclusion in "include" directive:
file1.rst
> file1.rst
~~~



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #198 Add support for Python 3.11

[Hugo <hu...@us...>]

---

** [patches:#198] Add support for Python 3.11**

**Status:** open
**Group:** None
**Created:** Tue Nov 01, 2022 08:02 PM UTC by Hugo 
**Last Updated:** Tue Nov 01, 2022 08:02 PM UTC
**Owner:** nobody
**Attachments:**

- [add-3.11.patch](https://sourceforge.net/p/docutils/patches/198/attachment/add-3.11.patch) (1.1 kB; application/octet-stream)


Python 3.11 was [released on 2022-10-24](https://discuss.python.org/t/python-3-11-0-final-is-now-available/20291?u=hugovk) 🚀 

![image](https://user-images.githubusercontent.com/1324225/198233993-5b79523b-370f-4316-a4be-9403c8209068.png)


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] Re: #197 MathML: support pandoc(1) as an external converter

[Günter M. <mi...@us...>]

> One big advantage of pandoc is that it supports {align} and {gather}.

I see. Active development and widespread use is another bonus (especially
over other external tools).

> Pandoc and latex2mathml.py supports different subsets of latex. For
> example, latex2mathml.py does not support \gsime, but pandoc does.

> The pandoc list of supported symbols is here [1] and it appears to be
> bigger (~2800 symbols) than the one in latex2mathml.py (~100 symbols)

latex2mathml uses the same database and extracts the about 600 symbols
supported by the most common math packages. It also supports literal
Unicode characters in the input.

> Actually it appears [1] is written by yourself, and that version is 8
> years old, so if you know where a newer version is I can go file a PR
> to them to support newer stuff like \underleftrightarrow.

The database and related work is available under
https://milde.users.sourceforge.net/LUCR/Math/
The latest revision is used in latex2mathml but not published yet.

The "unimathsymbols" database only contains LaTeX math macros that map
directly to Unicode code points. (\underleftrightarrow is implemented
using ↔ (\leftrightarrow) in a `<munder>` element.)


---

** [patches:#197] MathML: support pandoc(1) as an external converter**

**Status:** open
**Group:** None
**Created:** Fri Oct 28, 2022 02:21 PM UTC by Ximin Luo
**Last Updated:** Tue Nov 01, 2022 11:44 AM UTC
**Owner:** nobody
**Attachments:**

- [0001-MathML-support-pandoc-1-as-an-external-converter.patch](https://sourceforge.net/p/docutils/patches/197/attachment/0001-MathML-support-pandoc-1-as-an-external-converter.patch) (3.5 kB; text/x-patch)


In my personal experience, `pandoc` is more reliable than `latexml`, e.g. the current support for the latter in docutils does not automatically include amsfonts which prevents things like mathbb from working. In `pandoc` it Just Works.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

Dear Adam,

thank you for the fast reply.

On 2022-11-01, Adam Turner wrote:
...

> The overall aim with the test changes is to be able to use the standard
> library's ``unittest`` runner, which will alleviate some of the points
> you raise.

I understand and support the overall aim to use standard tools for the tests.
I would have preferred a "feature branch" (as mandated by the "policies")
for this change set, though. (I agree that with git this would have been
easier and have to apologise for the delay of my respective reviews.)


>> r9135
>>   tex2unichar.py is a generated file, changes will be lost with an
>>   update from the database.
>       
>>   (I updated the generating script and prepared a fixup patch.)

> I saw the note that it is generated, but I could not find the
> generating script in the repository (even in ``sandbox/``) -- please
> may you let me know where it is?    

It is at the place indicated in the script,
http://milde.users.sourceforge.net/LUCR/Math/ in the "tools" subdir.
The latest version is not published yet, though, and I plan moving the
project to codeberg.org.

Generally, if a file is created or maintained by an active member of the
Docutils developer team, I'd prefer an "ask before change" (except for
emergencies and small typos).


>> r9140, r9141: Import ``DocutilsTestSupport`` from ``test``

>>   This commit breaks the ability to run individual test scripts from
>>   their respective directories. This is (was) an important help when
>>   working on the test scripts.

> The unittest runner resolves this, though we are not quite at that
> point. We could add an option to ``alltests.py`` to run only tests in a
> specific directory?

When editing a test script, I regularly run it from my editor (it is
just one click). This is a feature I don't want to miss, the actual
implementation is less important (but an option to alltests.py would not
help, it should work from the scrip's "shebang line"). Is there a way
to call the "unittest runner" in the shebang line of the individual
scripts?

...

>> r9146
>>  DocutilsTestSupport must be imported before docutils
>>  so that imports from docutils use the test-dirs parent directory
>>  (in case we want to test a not installed Docutils version)

> If alltests.py is used to run tests, this is still the case.
> We could add an import of DTS to those modules again, though my aim
> eventually is to remove DTS. Is testing an uninstalled Docutils a
> usecase that is widely relied upon?

At least for me, it is a "natural" thing to expect (as I got used to it over
the last decade).

...


>> r9167
Introduces a backwards incompatible API change! (see below)

>>   IMV, an encoded string ("bytes" object) is no "binary" output.
...

> Python 3 bytes objects are sequences of 8-bit bytes, so I would say
> that it is binary output, but perhaps we could improve here.

The Python 3.10 documentation for `bytes` calls it a "binary sequence
type", so I may have to adapt.

The naming of the `core.publish_string()` API function pre-dates Python3
and uses the term "string" as a superordinate for "Unicode strings"
(`str`/`unicode`) and "encoded strings" (`bytes`/`str`).
The documentation is explicit about the return value
(defaulting to 'utf-8' encoded `bytes` instances).

The function is part of the "first level API": intended for use by
3rd-party code. Changing the return value has the potential of many
breaks. 
Please revert the changes to core.py and io.py.

Unicode `str` output can be achieved passing
``settings_overwrites={'output_encoding': 'unicode'}`` to
publish_string().

(In the long run, I'd like to replace the "homegrown" docutils.io classes
with standard objects, but this is a separate topic.)


>> r9168 Use ``.rstrip()`` instead

>>   This fails with multi-lined output if line endings in output and extended
>>   differ.

> For tests, I believe we should ensure that we have as little 'magic' as
> possible -- what we say we expect should be bit-for-bit identical to
> what we generate. In this sense I fixed the newline errors to ensure
> correctness.

This depends on the context: over-specified tests contribute to noise due to
false positives. Test results should not depend on the "native" line endings
of the system running the tests.


>> r9178
8 lines of doctest code seems overly verbose in the context of an
"overview of the Docutils architecture".

>>   In case we don't want to ship "quicktest.py" with the Docutils
>>   distribution, we can still keep it in the Docutils repo and simply change:
>  
>>   -can be found in the ``tools/`` directory of the Docutils distribution)
>>   +can be found in the ``tools/`` directory of the Docutils repository)

> I was actually going to suggest deleting "quicktest.py"
> entirely--finding the version can now be done with "docutils --version"
> and the last substantive change in functionality was 2006.
> "EasyDialogs" was removed in Python 2.7.

It can still be useful for development: testing the reST parser (including
the option to generate input/output pairs for parser test scripts).
I'd move it to tools/dev and remove the EasyDialogs part.


Günter

[Docutils-develop] [docutils:patches] #197 MathML: support pandoc(1) as an external converter

[Ximin L. <inf...@us...>]

One big advantage of pandoc is that it supports {align} and {gather}.

Pandoc and latex2mathml.py supports different subsets of latex. For example, latex2mathml.py does not support \gsime, but pandoc does.

The pandoc list of supported symbols is here [1] and it appears to be bigger (~2800 symbols) than the one in latex2mathml.py (~100 symbols), although it does not support some stuff like \underleftrightarrow that latex2mathml does.

Actually it appears [1] is written by yourself, and that version is 8 years old, so if you know where a newer version is I can go file a PR to them to support newer stuff like \underleftrightarrow.

(Another advantage on top of the other external tools, is that it is still under active development; the last commit to the texmath component was 24 days ago.)

[1] https://github.com/jgm/texmath/blob/master/lib/totexmath/unimathsymbols.txt


---

** [patches:#197] MathML: support pandoc(1) as an external converter**

**Status:** open
**Group:** None
**Created:** Fri Oct 28, 2022 02:21 PM UTC by Ximin Luo
**Last Updated:** Tue Nov 01, 2022 11:42 AM UTC
**Owner:** nobody
**Attachments:**

- [0001-MathML-support-pandoc-1-as-an-external-converter.patch](https://sourceforge.net/p/docutils/patches/197/attachment/0001-MathML-support-pandoc-1-as-an-external-converter.patch) (3.5 kB; text/x-patch)


In my personal experience, `pandoc` is more reliable than `latexml`, e.g. the current support for the latter in docutils does not automatically include amsfonts which prevents things like mathbb from working. In `pandoc` it Just Works.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #197 MathML: support pandoc(1) as an external converter

[Ximin L. <inf...@us...>]

One big advantage of pandoc is that it supports {align} and {gather}.

Pandoc and latex2mathml.py supports different subsets of latex. For example, latex2mathml.py does not support \gsime, but pandoc does.

The pandoc list of supported symbols is here [1] and it appears to be bigger (~2800 symbols) than the one in latex2mathml.py (~100 symbols), although it does not support some stuff like \underleftrightarrow that latex2mathml does.

Actually it appears [1] is written by yourself, and that version is 8 years old, so if you know where a newer version is I can go file a PR to them to support newer stuff like \underleftrightarrow.

[1] https://github.com/jgm/texmath/blob/master/lib/totexmath/unimathsymbols.txt


---

** [patches:#197] MathML: support pandoc(1) as an external converter**

**Status:** open
**Group:** None
**Created:** Fri Oct 28, 2022 02:21 PM UTC by Ximin Luo
**Last Updated:** Tue Nov 01, 2022 06:48 AM UTC
**Owner:** nobody
**Attachments:**

- [0001-MathML-support-pandoc-1-as-an-external-converter.patch](https://sourceforge.net/p/docutils/patches/197/attachment/0001-MathML-support-pandoc-1-as-an-external-converter.patch) (3.5 kB; text/x-patch)


In my personal experience, `pandoc` is more reliable than `latexml`, e.g. the current support for the latter in docutils does not automatically include amsfonts which prevents things like mathbb from working. In `pandoc` it Just Works.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #197 MathML: support pandoc(1) as an external converter

[Günter M. <mi...@us...>]

Thank you for the patch.

What are the advantages over the native LaTeX -> MathML converter?

A test of the new conversion route with the maths documentation source
[mathematics.txt](https://docutils.sourceforge.io/docs/ref/rst/mathematics.txt)  revealed:

* it takes long to run (considerably longer than the native converter but far shorter than `latexml`),
* there are > 120 conversion errors (unknown AMSmath macros),
* the error messages need getting used to
   (instead of reporting an unknown macro, it tells about expecting "%", "\\label", "\\nonumber" or whitespace)
* the successfull conversions are similar to the native MathML support.

BTW: a fix for "amsfonts" commands with `latexml` is ready and will be soon in the repository.






---

** [patches:#197] MathML: support pandoc(1) as an external converter**

**Status:** open
**Group:** None
**Created:** Fri Oct 28, 2022 02:21 PM UTC by Ximin Luo
**Last Updated:** Fri Oct 28, 2022 02:21 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-MathML-support-pandoc-1-as-an-external-converter.patch](https://sourceforge.net/p/docutils/patches/197/attachment/0001-MathML-support-pandoc-1-as-an-external-converter.patch) (3.5 kB; text/x-patch)


In my personal experience, `pandoc` is more reliable than `latexml`, e.g. the current support for the latter in docutils does not automatically include amsfonts which prevents things like mathbb from working. In `pandoc` it Just Works.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Recent commit activity

[Adam T. <aat...@ou...>]

Dear Günter,

Thank you for the note. I shall aim to add more rationale in commit messages in future.
Thank you also for the comments, which I've briefly addressed below.

The overall aim with the test changes is to be able to use the standard library's ``unittest`` runner, which will alleviate some of the points you raise.

> r9135
>   tex2unichar.py is a generated file, changes will be lost with an
>   update from the database.
       
>   (I updated the generating script and prepared a fixup patch.)

I saw the note that it is generated, but I could not find the generating script in the repository (even in ``sandbox/``) -- please may you let me know where it is?
    
> r9140, r9141: Import ``DocutilsTestSupport`` from ``test``
> 
>   This commit breaks the ability to run individual test scripts from
>   their respective directories. This is (was) an important help when
>   working on the test scripts.

The unittest runner resolves this, though we are not quite at that point.
We could add an option to ``alltests.py`` to run only tests in a specific directory?

> r9145
>   Since when is _format_str() unused?
>   Does this affect reporting of test failures?

r9143 removed the redefinitions of ``assertEqual``, ``assertNotEqual``, etc, which were the only uses of  _format_str()
This allows using ``TestCase``'s specialised equality methods by type.
Differences in strings with newlines (the reported purpose of _format_str()) are handled well by TestCase

> r9146
>  DocutilsTestSupport must be imported before docutils
>  so that imports from docutils use the test-dirs parent directory
>  (in case we want to test a not installed Docutils version)

If alltests.py is used to run tests, this is still the case.
We could add an import of DTS to those modules again, though my aim eventually is to remove DTS.
Is testing an uninstalled Docutils a usecase that is widely relied upon?

> r9163
>   What happend to sys.stdout.flush()  ?

``.flush()`` is still called before running tests, I didn't see a reason to call it before the environment information print-outs, though if it is needed we should add it back in.

> r9165
>   compare_output() no longer prints the corresponding input in case of
>   assertion errors which makes finding problems in parser test suites
>   harder.

My aim for parser test suites is to improve the test harnesses further, so this is not a permanent removal.

> r9167
>   IMV, an encoded string ("bytes" object) is no "binary" output.
>   
>   I need some more time to understand the changes here.
>   
>   A StringOutput.encode() method that *decodes* bytes data seems
>   odd:
>  
>   @@ -581,6 +600,11 @@ class StringOutput(Output):
>   
>   +    def encode(self, data):
>   +        if isinstance(data, bytes):
>   +            return data.decode(self.encoding, self.error_handler)
>   +        return str(data)
>   +

Python 3 bytes objects are sequences of 8-bit bytes, so I would say that it is binary output, but perhaps we could improve here.
I think the naming of ``.encode()`` is unfortunate in this context, but StringOutput should return strings (text content) and BytesOutput should return bytes (binary content).

> r9168 Use ``.rstrip()`` instead
> 
>   This fails with multi-lined output if line endings in output and extended
>   differ.

For tests, I believe we should ensure that we have as little 'magic' as possible -- what we say we expect should be bit-for-bit identical to what we generate. In this sense I fixed the newline errors to ensure correctness.

> r9178 
>   In case we don't want to ship "quicktest.py" with the Docutils
>   distribution, we can still keep it in the Docutils repo and simply change:
  
>   -can be found in the ``tools/`` directory of the Docutils distribution)
>   +can be found in the ``tools/`` directory of the Docutils repository)

I was actually going to suggest deleting "quicktest.py" entirely--finding the version can now be done with "docutils --version" and the last substantive change in functionality was 2006. "EasyDialogs" was removed in Python 2.7.

> r9179, r9180
>   there should be a way to avoid this code repetition

In general for tests I am more open to repetition as it is useful to be explicit about what is being tested, though I intend to refactor the functional tests, which hopefully will help achieve this goal.

Thanks,
Adam

[Docutils-develop] Recent commit activity

[Guenter M. <mi...@us...>]

Dear Docutils developers, dear Adam,

after a long period of calm, I realize a flurry of commits over the last
couple of days -- far more than I can reasonably review in time.
It is nice to see development activity but also alarming to see it at this
pace, as even a short looking over revealed some pitfalls.

For other changes I would have preferred a discussion or at least some
rationale in the commit messages (not just a one-liner telling what is done).

Changes to the test suite should not only keep the tests running but also
ensure intelligable feedback in case of test failures.
Did you test how the changes affect error reports?

r9135
  tex2unichar.py is a generated file, changes will be lost with an
  update from the database.
       
  (I updated the generating script and prepared a fixup patch.)
       
r9140, r9141: Import ``DocutilsTestSupport`` from ``test``

  This commit breaks the ability to run individual test scripts from
  their respective directories. This is (was) an important help when
  working on the test scripts.

r9145
  Since when is _format_str() unused?
  Does this affect reporting of test failures?
  
r9146
  DocutilsTestSupport must be imported before docutils
  so that imports from docutils use the test-dirs parent directory
  (in case we want to test a not installed Docutils version)
  
r9163
  What happend to sys.stdout.flush()  ?
  
r9165
  compare_output() no longer prints the corresponding input in case of
  assertion errors which makes finding problems in parser test suites
  harder.
  
r9167
  IMV, an encoded string ("bytes" object) is no "binary" output.
  
  I need some more time to understand the changes here.
  
  A StringOutput.encode() method that *decodes* bytes data seems
  odd:
 
  @@ -581,6 +600,11 @@ class StringOutput(Output):
  
  +    def encode(self, data):
  +        if isinstance(data, bytes):
  +            return data.decode(self.encoding, self.error_handler)
  +        return str(data)
  +
  
r9168 Use ``.rstrip()`` instead

  This fails with multi-lined output if line endings in output and extended
  differ.
  
r9178 
  In case we don't want to ship "quicktest.py" with the Docutils
  distribution, we can still keep it in the Docutils repo and simply change:
  
  -can be found in the ``tools/`` directory of the Docutils distribution)
  +can be found in the ``tools/`` directory of the Docutils repository)

r9179, r9180
  there should be a way to avoid this code repetition
  
...


Günter