#369 Change in 0.15 breaks3rd party code.

[Alexander Heger]

A change introduced in version 0.15 breaks my code. This is related to #353.

I get the error message

Exception occurred:
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 882, in interpreted
    nodes[0][0].rawsource = unescape(text, True)
AttributeError: 'str' object has no attribute 'rawsource'

and the full log of the error is

# Sphinx version: 2.1.2
# Python version: 3.7.4 (CPython)
# Docutils version: 0.15.1 release
# Jinja2 version: 2.10.1
# Last messages:
#   building [html]: targets for 0 source files that are out of date
#   updating environment:
#   46 added, 0 changed, 0 removed
#   reading sources... [  2%] Adapnet
#   reading sources... [  4%] Aliases
#   reading sources... [  6%] Command
#   reading sources... [  8%] Dump
#   reading sources... [ 10%] Environment
#   reading sources... [ 13%] Formats
#   reading sources... [ 15%] History
# Loaded extensions:
#   sphinx.ext.mathjax (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.1) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.1) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.2) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.3) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.2) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/alex/Python/lib/python3.7/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/todo.py
#   sphinxext (unknown version) from /home/alex/kepler/source/doc/sphinxext/__init__.py
#   sphinx.ext.coverage (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.imgmath (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/imgmath.py
#   sphinx.ext.ifconfig (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.viewcode (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/viewcode.py
Traceback (most recent call last):
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/cmd/build.py", line 284, in build_main
    app.build(args.force_all, filenames)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/application.py", line 345, in build
    self.builder.build_update()
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 319, in build_update
    len(to_build))
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 332, in build
    updated_docnames = set(self.read())
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 438, in read
    self._read_serial(docnames)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 460, in _read_serial
    self.read_doc(docname)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 504, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/io.py", line 325, in read_doc
    pub.publish()
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/io.py", line 113, in read
    self.parse()
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/parsers.py", line 94, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2346, in explicit_markup
    self.explicit_list(blank_finish)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2376, in explicit_list
    match_titles=self.state_machine.match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 319, in nested_list_parse
    node=node, match_titles=match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2649, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2356, in explicit_construct
    return method(self, expmatch)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2099, in directive
    directive_class, match, type_name, option_presets)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2148, in run_directive
    result = directive_instance.run()
  File "/home/alex/kepler/source/doc/sphinxext/parm.py", line 502, in run
    self.state.nested_parse(self.content, self.content_offset, text_node)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2785, in text
    paragraph, literalnext = self.paragraph(lines, startline)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 418, in paragraph
    textnodes, messages = self.inline_text(text, lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 428, in inline_text
    self.memo, self.parent)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 647, in parse
    lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 791, in interpreted_or_phrase_ref
    lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 882, in interpreted
    nodes[0][0].rawsource = unescape(text, True)
AttributeError: 'str' object has no attribute 'rawsource'

The identical example works in version 0.14; I do not know what other changes have been made in 0.15 that could cause ir, #353 looks as if was a related chain of changes to unescape text.

[Günter Milde]

Thank you for the report.

I suppose that the problem is a doctree node inserted by the parm.py extension (or some other 4th party code) that does not follow the Docutils specs but worked "accidentially" in version 0.14:

Only Text nodes contain strings directly, all other nodes have the text content wrapped in Text nodes
(see nodes.py and the transformations that come with Docutils).

Docutils 0.15 adds the "sourcecode" attribute to some more nodes than 0.14 (as a debugging help).

We could catch also AttributeErrors in

    if role_fn:
        nodes, messages2 = role_fn(role, rawsource, text, lineno, self)
        try:
            nodes[0][0].rawsource = unescape(text, True)
        except IndexError:
            pass
        return nodes, messages + messages2

to make the code more forgiving with non-standard nodes. You may expect other problems with non-standard nodes, though.

[Alexander Heger]

Dear Günter,

yes, for me I could fix by changing

        if role_fn:
            nodes, messages2 = role_fn(role, rawsource, text, lineno, self)
            try:
                nodes[0][0].rawsource = unescape(text, True)
            except IndexError:
                pass
            return nodes, messages + messages2

to

        if role_fn:
            nodes, messages2 = role_fn(role, rawsource, text, lineno, self)
            try:
                nodes[0][0].rawsource = unescape(text, True)
            except (IndexError, AttributeError):
                pass
            return nodes, messages + messages2

But to help me change code to conforming, what should I be using for something that just contains plain text other than a Tdext node? Where would I find the corresponding development guide?

[Günter Milde]

The overview of existing docs is http://docutils.sourceforge.net/docs/index.html#api-api-reference-material-for-client-developers)
Unfortunately, there is no comprehensive development guide. However, for new inline roles, there is http://docutils.sourceforge.net/docs/howto/rst-roles.html.
Generally, looking into the Python source and find out how similar tasks are done is most helpfull.

what should I be using for something that just contains plain text other than a Text node?
You may look up the list of existing nodes for something that matches the requirement.
Or look for a syntax construct in standard rST that is similar to your new feature,
create a test document that contains this construct,
translate it with rst2pseudoxml or rst2xml and
check the resulting doctree stucture.

BTW: The problematic code is removed from "states.py" in the repository at [r8197].

[David Alphus]

This seems to happen if your role returns a list of nodes.Text nodes since nodes[0][0] directly refers to the string content. For an AttributeError, you could attempt to set nodes[0].rawsource instead.