Menu ▾ ▴

#353 Backslash escape ignored for some markup constructs

closed-fixed
nobody
None
5
2020-03-03
2018-11-27
Günter Milde
No

The reStructuredText Markup Specification states that

A backslash followed by any character (except whitespace characters in non-URI contexts) escapes that character. The escaped character represents the character itself, and is prevented from playing a role in any markup interpretation.
--- http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#escaping-mechanism

However, the "classifier delimiter" in definition list terms (" : ") and the "author separator" in the "authors" Docinfo field (";" or ",") are not escaped, when preceded by a backslash. (The backslash is removed before the relevant source part is parsed.)

An attempt to fix this situation in the 0.15 development version relied on the undocumented "rawsource" attribute. Because this attribute is not part of the Docutils API but a developer aid not to be relied on in document generation, the respective patches were reversed.

A solution would be to store NULL-escaped text strings in the doctree nodes , e.g. 

-  return [self.node_class(rawtext, utils.unescape(text), **options)], []
+  return [self.node_class(rawtext, text, **options)], []

and unescape in nodes.Text.astext():

    def astext(self):
-   return reprunicode(self)
+   return reprunicode(docutils.utils.unescape(self))

This enables functions and transforms processing Text nodes
to respect backslash escapes.
The attached patch to r8241 fixes the problem and passes all tests (including 2 new tests for escaped markup).

1 Attachments
do-not-unescape-text-when-generating-Text-nodes.patch

Discussion

  • Günter Milde

    Günter Milde - 2018-11-27
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -4,7 +4,7 @@

 However, the "classifier delimiter" in definition list terms (" : ") and the "author separator" in the "authors" Docinfo field (";" or ",") are not escaped, when preceded by a backslash. (The backslash is removed before the relevant source part is parsed.)

-An attempt to fix this situation in the 0.15 development version relied on the undocumented "sourcetext" attribute. Because this attribute is not part of the Docutils API but a developer aid  not to be relied on in document generation, the respective patches were reversed.
+An attempt to fix this situation in the 0.15 development version relied on the undocumented "rawsource" attribute. Because this attribute is not part of the Docutils API but a developer aid  not to be relied on in document generation, the respective patches were reversed.

 A solution would be to store NULL-escaped text strings in the doctree nodes , e.g. 
 ~~~
@@ -19,3 +19,4 @@
 ~~~
 This enables functions and transforms processing Text nodes
 to respect backslash escapes.
+The attached patch to r8241 fixes the problem and passes all tests (including 2 new tests for escaped markup).
    • Attachments has changed:

    Diff:

    --- old
+++ new
@@ -0,0 +1 @@
+do-not-unescape-text-when-generating-Text-nodes.patch (13.6 kB; text/x-patch)
     

  • Günter Milde

    Günter Milde - 2019-07-22
    • status: open --> closed-fixed
     

  • Günter Milde

    Günter Milde - 2019-07-22

    Patch applied in [r8284] .

     

    Related

    Commit: [r8284]

  • Alexander Heger

    Alexander Heger - 2019-07-25

    This fix now breaks my code.

    Exception occurred:
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 882, in interpreted
    nodes[0][0].rawsource = unescape(text, True)
AttributeError: 'str' object has no attribute 'rawsource'

    with the log file

    # Sphinx version: 2.1.2
# Python version: 3.7.4 (CPython)
# Docutils version: 0.15.1 release
# Jinja2 version: 2.10.1
# Last messages:
#   building [html]: targets for 0 source files that are out of date
#   updating environment:
#   46 added, 0 changed, 0 removed
#   reading sources... [  2%] Adapnet
#   reading sources... [  4%] Aliases
#   reading sources... [  6%] Command
#   reading sources... [  8%] Dump
#   reading sources... [ 10%] Environment
#   reading sources... [ 13%] Formats
#   reading sources... [ 15%] History
# Loaded extensions:
#   sphinx.ext.mathjax (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.1) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.1) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.2) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.3) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.2) from /home/alex/Python/lib/python3.7/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/alex/Python/lib/python3.7/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/todo.py
#   sphinxext (unknown version) from /home/alex/kepler/source/doc/sphinxext/__init__.py
#   sphinx.ext.coverage (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.imgmath (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/imgmath.py
#   sphinx.ext.ifconfig (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.viewcode (2.1.2) from /home/alex/Python/lib/python3.7/site-packages/sphinx/ext/viewcode.py
Traceback (most recent call last):
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/cmd/build.py", line 284, in build_main
    app.build(args.force_all, filenames)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/application.py", line 345, in build
    self.builder.build_update()
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 319, in build_update
    len(to_build))
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 332, in build
    updated_docnames = set(self.read())
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 438, in read
    self._read_serial(docnames)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 460, in _read_serial
    self.read_doc(docname)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 504, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/io.py", line 325, in read_doc
    pub.publish()
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/io.py", line 113, in read
    self.parse()
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/alex/Python/lib/python3.7/site-packages/sphinx/parsers.py", line 94, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2346, in explicit_markup
    self.explicit_list(blank_finish)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2376, in explicit_list
    match_titles=self.state_machine.match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 319, in nested_list_parse
    node=node, match_titles=match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2649, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2356, in explicit_construct
    return method(self, expmatch)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2099, in directive
    directive_class, match, type_name, option_presets)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2148, in run_directive
    result = directive_instance.run()
  File "/home/alex/kepler/source/doc/sphinxext/parm.py", line 502, in run
    self.state.nested_parse(self.content, self.content_offset, text_node)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2785, in text
    paragraph, literalnext = self.paragraph(lines, startline)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 418, in paragraph
    textnodes, messages = self.inline_text(text, lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 428, in inline_text
    self.memo, self.parent)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 647, in parse
    lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 791, in interpreted_or_phrase_ref
    lineno)
  File "/home/alex/Python/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 882, in interpreted
    nodes[0][0].rawsource = unescape(text, True)
AttributeError: 'str' object has no attribute 'rawsource'

    Could you please undo this change or provide a workaround, e.g., a "try...except" (as would be reasonable to do)?

     

  • Günter Milde

    Günter Milde - 2019-07-27

    Actually, this fix is only in the development version (0.16.dev). Your problem with the code is due to
    the (partially reversed) addition of "sourcecode" to some more nodes in 0.15 in combination with a non-standard node added to the doctree. (See discussion in #369.)

     

  • Andrey K.

    Andrey K. - 2019-09-02

    Hi folks!
    I started faicing the similar issue as @alx posted. I guess first time it had happened at our project's CI several week ago. Downgrading to docutils<0.15.0 helps.

    The code that triggers docutils to fail is a custom role. The simplified version looks like:

    from doctutils import nodes

def custome_role(*args, **kwargs):
     return [nodes.Text("Foo")], []
     

    • Günter Milde

      Günter Milde - 2019-09-02

      As said in the discussion in [#369], this should be solved by nesting the Text node inside an inline node.
      (Besides this, the cited code should work with the repository version 0.16.dev (try a snapshot).)

       

      Related

      Bugs: #369

Log in to post a comment.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.