#302 distutils' check() using docutils failing without specific error message

[Matt Bogosian]

Immediately after releasing 0.13.1, formerly passing builds of dropbox/stone started failing on Travis CI:

• passing example
• failing example

The error message is missing/non-descript:

$ python setup.py check -m -r -s -v
warning: check: Could not finish the parsing.

error: Please correct your package.

Note that pinning docutils==0.12.0 fixes the error. (In other words, this only started showing up with 0.13.1.)

Unfortunately, I cannot reproduce this locally in OS X (I have Python 2.7.10 and Python 3.5.2), so it may be limited to certain versions of Python or certain environments.

I'm happy to assist with additional testing if needed/desired.

[Matt Bogosian]

Back-reference to original discovery/discussion: https://github.com/dropbox/stone/pull/28#discussion-diff-91844629L44

[engelbert gruber]

I did not see the error description in your git log ?
Did i overlook something ?

Am 13.12.2016 01:59 schrieb "Matt Bogosian" mbogosian@users.sf.net:

Back-reference to original discovery/discussion:
https://github.com/dropbox/stone/pull/28#discussion-diff-91844629L44

---

• [bugs:#302] https://sourceforge.net/p/docutils/bugs/302/ docutils
  failing without specific error message*

Status: open
Group: Default
Created: Tue Dec 13, 2016 12:54 AM UTC by Matt Bogosian
Last Updated: Tue Dec 13, 2016 12:54 AM UTC
Owner: nobody

Immediately after releasing 0.13.1, formerly passing builds of
dropbox/stone started failing on Travis CI:

• passing example
  https://travis-ci.org/dropbox/stone/jobs/182914480#L193
• failing example
  https://travis-ci.org/dropbox/stone/jobs/182914200#L193

The error message is missing/non-descript:

$ python setup.py check -m -r -s -v
warning: check: Could not finish the parsing.

error: Please correct your package.

Note that pinning docutils==0.12.0 fixes the error. (In other words, this
only started showing up with 0.13.1.)

Unfortunately, I cannot reproduce this locally in OS X (I have Python
2.7.10 and Python 3.5.2), so it may be limited to certain versions of
Python or certain environments.

I'm happy to assist with additional testing if needed/desired.

Sent from sourceforge.net because you indicated interest in
https://sourceforge.net/p/docutils/bugs/302/

To unsubscribe from further messages, please visit
https://sourceforge.net/auth/subscriptions/

[Matt Bogosian]

I did not see the error description in your git log ?
Did i overlook something ?

If you're referring to the log of the failing Travis CI build, then the only output is:

$ python setup.py check -m -r -s -v
running check
warning: check: Could not finish the parsing.

error: Please correct your package.

There is no detailed error message. My apologies if this was unclear from the original post or if I've misunderstood your question.

[Matt Bogosian]

Note: this appears to happen in the Travis CI environment with docutils==0.13.1, even with setup(…, license='', …, long_description='', …). In other words, I don't think it's a reStructuredText parsing problem.

[Günter Milde]

In this case, it is a Doctuils rst-parser setup problem. Distutils' check() function misses to pass default options, the option "use_character_level_inline_markup" is tested during the initialization of the parser.

[Günter Milde]

Thank you for your report. We are sorry for the inconvenience but without error message, we can only guess.
It seems the problem is an incompatibility between docutils 0.13 and the application running it. A similar problem was detected and solved with the Sphinx documentation builder (see Bugs: #301).
In case dropbox/stone/travis uses Docutils via Sphinx, the issue may be solved by upgrading to Sphinx 1.5.1.

[Matt Bogosian]

Our project doesn't use Sphinx. Are you saying that docutils 0.13 is not backward-compatible with 0.12? That is the only difference between our passing and failing builds.

[Günter Milde]

What I say is that Docutils 0.13 is incompatible with Sphinx < 5.1.5 and possibly incopatible with your application or some middleware at Travis.

Docutils 0.13.1 is backward-compatible to 0.12 in the sense that all
documented features are kept and the all tests pass. There is no change to
the rST format.
However, depending on the way Docutils is used, internal changes can lead to incompatibilities (or make existing incompatibilities lead to crashes/failure).
This cannot be solved without better knowledge about the actual way your
project uses Docutils.

Using Docutils via the provided front-ends results in error messages we can
interpret. With just a vague "Could not finish the parsing. Correct your package" we are left to wild guesses. My first guess (Sphinx) was wrong, so the following questings may lead to a better guess:

• who could not parse (lint, tox, python, docutils?) and what?
• Where does the warning come from (it does not look like a Docutils warning)?
• Why only on Travis but not locally?
• where are all the error messages? Is there a way to get a Python backtrace?
• Where does the setup.py come from, what is in it, what is the check function there.
• How does Docutils come into this? Where is it used, what for is it used and how is it called?

[Matt Bogosian]

Okay, this is about as simple as I can get things, and I can still repro the problem:

https://github.com/posita/stone/tree/posita/repro-docutils-snafu

That has a single (empty) package (a zero-byte stone/__init__.py), and about as simple a setup.py as one can get. The only dependency (pulled in from tox) is docutils. As you can see from the builds, docutils==0.12 succeeds, docutils=0.13.1 fails, with the exception of Python 3.5:

https://travis-ci.org/dropbox/stone/builds/183778095

Back-reference: https://github.com/travis-ci/travis-ci/issues/7028

[Matt Bogosian]

The error is happening here:

AttributeError: "Values instance has no attribute 'character_level_inline_markup'"
> /tmp/stone/.tox/docutils-0.13.1/local/lib/python2.7/site-packages/docutils/parsers/rst/states.py(470)init_customizations()
-> if settings.character_level_inline_markup:
(Pdb) b /tmp/stone/.tox/docutils-0.13.1/local/lib/python2.7/site-packages/docutils/parsers/rst/states.py:470
Breakpoint 4 at /tmp/stone/.tox/docutils-0.13.1/local/lib/python2.7/site-packages/docutils/parsers/rst/states.py:470
(Pdb) c
warning: check: Could not finish the parsing.

The program exited via sys.exit(). Exit status:  error: Please correct your package.
> /tmp/stone/setup.py(3)<module>()
-> from __future__ import absolute_import, division, print_function
(Pdb) c
running check
> /tmp/stone/.tox/docutils-0.13.1/local/lib/python2.7/site-packages/docutils/parsers/rst/states.py(470)init_customizations()
-> if settings.character_level_inline_markup:
(Pdb) l
465         def __init__(self):
466             pass
467
468         def init_customizations(self, settings):
469             # lookahead and look-behind expressions for inline markup rules
470 B->         if settings.character_level_inline_markup:
471                 start_string_prefix = u'(^|(?<!\x00))'
472                 end_string_suffix = u''
473             else:
474                 start_string_prefix = (u'(^|(?<=\\s|[%s%s]))' %
475                                        (punctuation_chars.openers,
(Pdb) settings
<Values at 0x1587050: {'footnote_backlinks': 1, 'record_dependencies': DependencyList(None, []), 'language_code': 'en', 'traceback': None, 'pep_references': None, 'strip_comments': None, 'toc_backlinks': 'entry', 'dump_internals': None, 'datestamp': None, 'report_level': 2, '_destination': None, 'halt_level': 4, 'strip_classes': None, 'title': None, 'error_encoding_error_handler': 'backslashreplace', 'debug': None, 'rfc_references': None, 'tab_width': 4, 'dump_transforms': None, 'warning_stream': None, 'exit_status_level': 5, 'config': None, 'strict_visitor': None, 'dump_pseudo_xml': None, 'expose_internals': None, 'source_link': None, 'output_encoding_error_handler': 'strict', 'source_url': None, 'input_encoding': None, '_disable_config': None, 'id_prefix': '', 'sectnum_xform': 1, 'error_encoding': 'UTF-8', 'output_encoding': 'utf-8', 'generator': None, '_source': None, 'input_encoding_error_handler': 'strict', 'auto_id_prefix': 'id', 'strip_elements_with_classes': None, '_config_files': [], 'dump_settings': None}>

See also: https://github.com/travis-ci/travis-ci/issues/7028#issuecomment-266940801

[David Goodger]

Please save the following as a script, add it to your tox.ini as a command in each environment, and reply with the results:

import docutils
print(docutils.__version__)

from docutils.parsers.rst import Parser
from docutils import frontend
parser = Parser()
settings = frontend.OptionParser(components=(Parser,)).get_default_values()
import pprint
pprint.pprint(settings.__dict__)

This isolates what I believe will indicate the root problem. The settings object is lacking an attribute, causing the error. Why is still a mystery, which hopefully the results of the above will help clarify.

Thanks!

[Günter Milde]

Thank you for looking into this and giving more input.

On 14.12.16, Matt Bogosian wrote:

The error is happening here:

/tmp/stone/.tox/docutils-0.13.1/local/lib/python2.7/site-packages/docutils/parsers/rst/init.py(185)parse()
-> self.statemachine.run(inputlines, document, inliner=self.inliner)
AttributeError: "Values instance has no attribute 'character_level_inline_markup'"

This helps a lot to solve the mystery:

The character_level_inline_markup setting is new in 0.13:

• docutils/parsers/

  • Apply [ 103 ] Recognize inline markups without word boundaries.

-- HISTORY.txt

It is set in parsers/rst/init.py::

     ('Inline markup recognized at word boundaries only '
      '(adjacent to punctuation or whitespace). '
      'Force character-level inline markup recognition with '
      '"\\ " (backslash + space). Default.',
      ['--word-level-inline-markup'],
      {'action': 'store_false', 'dest': 'character_level_inline_markup'}),
     ('Inline markup recognized anywhere, regardless of surrounding '
      'characters. Backslash-escapes must be used to avoid unwanted '
      'markup recognition. Useful for East Asian languages. '
      'Experimental.',
      ['--character-level-inline-markup'],
      {'action': 'store_true', 'default': False,
       'dest': 'character_level_inline_markup'}),

and used in parsers/rst/states.py::

def init_customizations(self, settings):
    # lookahead and look-behind expressions for inline markup rules
    if settings.character_level_inline_markup:
        start_string_prefix = u'(^|(?<!\x00))'
        end_string_suffix = u''
    else:

The latter seems to be the problematic spot: settings could be the
Values instance that "has no attribute 'character_level_inline_markup'".

This does not (yet) solve the mystery, why this is a problem in your use
case but not in our tests. Maybe some middleware is passing settings as a
dictionary or similar.

Anyway, you may try the following patch that ensures the new setting is
present with its default (backwards compatible) value:

Exec: svn 'diff' 'states.py' 2>&1
Dir: /home/milde/Code/Python/docutils-svn/docutils/docutils/parsers/rst/

Index: states.py

--- states.py (Revision 7996)
+++ states.py (Arbeitskopie)
@@ -467,7 +467,7 @@

 def init_customizations(self, settings):
     # lookahead and look-behind expressions for inline markup rules

• if settings.character_level_inline_markup:
• if settings.ensure_value("character_level_inline_markup", False):
  start_string_prefix = u'(^|(?<!\x00))'
  end_string_suffix = u''
  else:

[Matt Bogosian]

As an aside, this (from distutils/command/check.py) probably risks hiding a lot of errors like the above:

140             try:
141  ->             parser.parse(data, document)
142             except AttributeError:  # <<-- this could happen anywhere in call to parser.parse (or any sub-call)
143                 reporter.messages.append((-1, 'Could not finish the parsing.',
144                                           '', {}))

[David Goodger]

Matt, thanks for the investigation. But please don't mischaracterize: "UPDATE: docutils maintainers do not appear interested in helping without additional information from our end. They are blaming it on our application." (https://github.com/dropbox/stone/pull/28#discussion_r92260654) That is false, and doesn't help.

Nobody blamed anyone; we're just mystified as to what's going on and need more information to investigate.

The replies above look very helpful though. Thanks!

[Matt Bogosian]

Corrected with my sincere apologies. 😔

[David Goodger]

Searching my email for "finish the parsing", I found something possibly related: https://sourceforge.net/p/docutils/bugs/270/

[Matt Bogosian]

I found that as well, but we are able to reproduce with empty strings passed to long_description (see our setup.py) and without any code blocks whatsoever, which led me to believe it was a different issue?

[Günter Milde]

It looks very much like a reoccurence of the generic problem - in #270: the way "distutils" calls and uses Docutils in its check() function.

The Python issue (and patch) http://bugs.python.org/issue23063
could explain, why in your case setup.py check() works with
Docutils 0.13 under Python 3.5.

Could you try whether applying marcman's Docutils-workaround
https://sourceforge.net/p/docutils/bugs/_discuss/thread/c5444752/f668/attachment/bug_270_2.patch
solves the problem with setup.py check() and Docutils 0.13 ?

[David Goodger]

I concur with Günter. It looks like the issue you're seeing is because Distutils is not setting up Docutils objects properly. This has been fixed in Python in http://bugs.python.org/issue23063. It looks like #270 was just another expression of this. I'm guessing that the Python 2.7.x in the failing environment is not up to date, and does not have the fix to the Python bug above.

So the question is, what specific version of Python is failing? Please include the micro version: 2.7.what?

[Matt Bogosian]

My apologies for not fully understanding #270. I thought it was more specific than it actually was. Travis CI appears to have 2.7.9 and 3.4.2. I'll try the patch and report back.

[David Goodger]

Yup, Python 2.7.9 and 3.4.2 are the proximal reason. The Python bugfix was applied to 2.7.10 and 3.4.3. Any chance of updating the Python versions (currently 2.7.12 and 3.4.5) on the system running Travis CI? That ought to solve it without resorting to ugly workarounds.

[Matt Bogosian]

Here is the output of the requested debug pprint script:

https://travis-ci.org/dropbox/stone/builds/183841292

Specifically (e.g., from the Python 2.7 job):

docutils-0.12 create: /home/travis/build/dropbox/stone/.tox/docutils-0.12
docutils-0.12 installdeps: docutils==0.12
docutils-0.12 inst: /home/travis/build/dropbox/stone/.tox/dist/stone-0.1.zip
docutils-0.12 installed: docutils==0.12,stone==0.1
docutils-0.12 runtests: PYTHONHASHSEED='1744110672'
docutils-0.12 runtests: commands[0] | python docutils-test-script.py
0.12
{'_config_files': [],
 '_destination': None,
 '_disable_config': None,
 '_source': None,
 'auto_id_prefix': 'id',
 'config': None,
 'datestamp': None,
 'debug': None,
 'dump_internals': None,
 'dump_pseudo_xml': None,
 'dump_settings': None,
 'dump_transforms': None,
 'error_encoding': 'UTF-8',
 'error_encoding_error_handler': 'backslashreplace',
 'exit_status_level': 5,
 'expose_internals': None,
 'file_insertion_enabled': 1,
 'footnote_backlinks': 1,
 'generator': None,
 'halt_level': 4,
 'id_prefix': '',
 'input_encoding': None,
 'input_encoding_error_handler': 'strict',
 'language_code': 'en',
 'output_encoding': 'utf-8',
 'output_encoding_error_handler': 'strict',
 'pep_base_url': 'http://www.python.org/dev/peps/',
 'pep_file_url_template': 'pep-%04d',
 'pep_references': None,
 'raw_enabled': 1,
 'record_dependencies': DependencyList(None, []),
 'report_level': 2,
 'rfc_base_url': 'http://www.faqs.org/rfcs/',
 'rfc_references': None,
 'sectnum_xform': 1,
 'smart_quotes': False,
 'source_link': None,
 'source_url': None,
 'strict_visitor': None,
 'strip_classes': None,
 'strip_comments': None,
 'strip_elements_with_classes': None,
 'syntax_highlight': 'long',
 'tab_width': 8,
 'title': None,
 'toc_backlinks': 'entry',
 'traceback': None,
 'trim_footnote_reference_space': None,
 'warning_stream': None}
docutils-0.12 runtests: commands[1] | python setup.py check -m -r -s -v
running check
docutils-0.13.1 create: /home/travis/build/dropbox/stone/.tox/docutils-0.13.1
docutils-0.13.1 installdeps: docutils==0.13.1
docutils-0.13.1 inst: /home/travis/build/dropbox/stone/.tox/dist/stone-0.1.zip
docutils-0.13.1 installed: docutils==0.13.1,stone==0.1
docutils-0.13.1 runtests: PYTHONHASHSEED='1744110672'
docutils-0.13.1 runtests: commands[0] | python docutils-test-script.py
0.13.1
{'_config_files': [],
 '_destination': None,
 '_disable_config': None,
 '_source': None,
 'auto_id_prefix': 'id',
 'character_level_inline_markup': False,
 'config': None,
 'datestamp': None,
 'debug': None,
 'dump_internals': None,
 'dump_pseudo_xml': None,
 'dump_settings': None,
 'dump_transforms': None,
 'error_encoding': 'UTF-8',
 'error_encoding_error_handler': 'backslashreplace',
 'exit_status_level': 5,
 'expose_internals': None,
 'file_insertion_enabled': 1,
 'footnote_backlinks': 1,
 'generator': None,
 'halt_level': 4,
 'id_prefix': '',
 'input_encoding': None,
 'input_encoding_error_handler': 'strict',
 'language_code': 'en',
 'output_encoding': 'utf-8',
 'output_encoding_error_handler': 'strict',
 'pep_base_url': 'http://www.python.org/dev/peps/',
 'pep_file_url_template': 'pep-%04d',
 'pep_references': None,
 'raw_enabled': 1,
 'record_dependencies': DependencyList(None, []),
 'report_level': 2,
 'rfc_base_url': 'http://tools.ietf.org/html/',
 'rfc_references': None,
 'sectnum_xform': 1,
 'smart_quotes': False,
 'source_link': None,
 'source_url': None,
 'strict_visitor': None,
 'strip_classes': None,
 'strip_comments': None,
 'strip_elements_with_classes': None,
 'syntax_highlight': 'long',
 'tab_width': 8,
 'title': None,
 'toc_backlinks': 'entry',
 'traceback': None,
 'trim_footnote_reference_space': None,
 'warning_stream': None}
docutils-0.13.1 runtests: commands[1] | python setup.py check -m -r -s -v
running check
warning: check: Could not finish the parsing.

error: Please correct your package.

Oddly enough, character_level_inline_markup seems to be present (and False)? 😕

[Günter Milde]

This is another instance of the problem in #270