#436 docutils doesn't build with Python 3.11

[Tomáš Hrnčiar]

Hello,

in Fedora we started with rebuilding Python packages with preleases of Python 3.11, currently it is 2nd alpha.

Docutils doesn't build because 3.11 adds support for null characters in the csv module, which breaks a test. See reproducer below.

import csv
from docutils.parsers.rst.directives import tables
with open('utf-16.csv', 'rb') as f: csv_data = f.read()
...
csv_data = str(csv_data, 'latin1').splitlines()
reader = csv.reader([tables.CSVTable.encode_for_csv(line + '\n') for line in csv_data])
next(reader)

Python 3.11:

['þÿ\x00"\x00T\x00r\x00e\x00a\x00t\x00"\x00', '\x00 \x00"\x00Q\x00u\x00a\x00n\x00t\x00i\x00t\x00y\x00"\x00', '\x00 \x00"\x00D\x00e\x00s\x00c\x00r\x00i\x00p\x00t\x00i\x00o\x00n\x00"\x00']

Python 3.10:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
_csv.Error: line contains NUL</module></stdin>

[Tomáš Hrnčiar]

The change was introduced in this commit: https://github.com/python/cpython/commit/b454e8e4df73bc73bc1a6f597431f171bfae8abd

[Günter Milde]

Thank you for the report.

Docutils doesn't build because 3.11 adds support for null characters in
the csv module, which breaks a test.

Could you attach a full test log?

[engelbert gruber]

mine breaks because reading utf-16.csv with null bytes does not break
so there is no exception data::

(du311) docutils$ python test/alltests.py 
Traceback (most recent call last):
  File "test/alltests.py", line 93, in <module>
    suite = suite()
            ^^^^^^^
  File "test/alltests.py", line 80, in suite
    suite = package_unittest.loadTestModules(DocutilsTestSupport.testroot,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "test/package_unittest.py", line 102, in loadTestModules
    module = import_module(mod)
             ^^^^^^^^^^^^^^^^^^
  File "test/package_unittest.py", line 133, in import_module
    mod = __import__(name)
          ^^^^^^^^^^^^^^^^
  File "test/test_parsers/test_rst/test_directives/test_tables.py", line 67, in <module>
    null_bytes_exception = DocutilsTestSupport.exception_data(null_bytes)[0]
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

[Günter Milde]

Could you re-try with r8909?

[engelbert gruber]

made it pass in r8910
reading a utf-16 file in latin1 gives funny results.
maybe breaking processing of the file would be better than producing garbled output

[Günter Milde]

reading a utf-16 file in latin1 gives funny results.

In the test, this is intentional. In praxi, utf-16 is recognized by the BOM and correctly decoded, if the "input_encoding" setting is left at its default (None).

maybe breaking processing of the file would be better than producing garbled output

See [feature-requests:#92].

[Günter Milde]

Fixed in [r8910]

[Günter Milde]

Fixed in release 0.19.
Thank you for reporting and testing.