docutils-users

[Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ant...@be...>]

Hi all,

Currently, RST specifies that entries in a definition list may be followed
by a series of classifiers all on the same line, as in

    term : classifier one : classifier 2
        Definition

This syntax is used by numpydoc (the de facto docstring standard in the
scientific Python world) to document individual parameters of callables, as
in

    parameter_name : type_of_parameter
        Description of the parameter.

(see
https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections
for details).

In general, this approach yields highly legible raw docstrings, and can
easily be converted into the Sphinx standard field list format (tools to do
so include numpydoc and sphinx.ext.napoleon).

However, it is awkward to include very long parameter "types" (e.g., an
enumeration of allowable string values -- which I know is not strictly a
type, but remains a common API) using this method, due to the restriction
that the classifiers need to stay in one line.  In practice, this means
that it is necessary to use a line continuation in the docstring, either

    parameter_name : some long type \
description
        Description of parameter.

or

    parameter_name : some long type \
                     description
        Description of the parameter.

The first form is ugly in the source, the second form is ugly when the
docstring is viewed as a string (e.g. by pydoc) as all the spaces before
``description`` end up in the literal docstring.  See e.g.
https://github.com/numpy/numpydoc/issues/87 for a longer discussion of the
issue.

While numpydoc can of course invent its own syntax, it would seem
preferable if a standard way to write multi-line classifiers was specified
by the RST standard.  Just checking for indentation is actually a bit
ambiguous, because it is not clear how to parse

    a : b
        c

(is ``c`` a continuation or a description?).  A possibility would be to
support backslashed-line continuations at the RST level, so the Python
docstring would look like

    parameter_name : some long type \\
                     description
        Description of the parameter

but RST would see a single backslash and know what to do; or require that
continued classifier lines start also with ``: ``:

    parameter_name : some long type
                   : description
        Description of the parameter

Neither option is strictly fully back-compatible, but I hope one of them,
or some similar solution, can be adopted.

Antony Lee

PS: As suggested on the docutils mailing list page, I am not subscribed to
the list, so please CC me in the reply.  Thanks!

Re: [Docutils-users] Multi-line classifiers in definition lists

[Guenter M. <mi...@us...>]

On 2017-10-04, Antony Lee wrote:

> Hi all,

> Currently, RST specifies that entries in a definition list may be followed
> by a series of classifiers all on the same line, as in

>     term : classifier one : classifier 2
>         Definition

> This syntax is used by numpydoc (the de facto docstring standard in the
> scientific Python world) to document individual parameters of
> callables, as in

>     parameter_name : type_of_parameter
>         Description of the parameter.

> (see
> https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections
> for details).

> In general, this approach yields highly legible raw docstrings, and can
> easily be converted into the Sphinx standard field list format (tools to do
> so include numpydoc and sphinx.ext.napoleon).

> However, it is awkward to include very long parameter "types" (e.g., an
> enumeration of allowable string values -- which I know is not strictly a
> type, but remains a common API) using this method, due to the restriction
> that the classifiers need to stay in one line.  In practice, this means
> that it is necessary to use a line continuation in the docstring, either

>     parameter_name : some long type \
> description
>         Description of parameter.

> or

>     parameter_name : some long type \
>                      description
>         Description of the parameter.

> The first form is ugly in the source, the second form is ugly when the
> docstring is viewed as a string (e.g. by pydoc) as all the spaces before
> ``description`` end up in the literal docstring.  See e.g.
> https://github.com/numpy/numpydoc/issues/87 for a longer discussion of the
> issue.

> While numpydoc can of course invent its own syntax, it would seem
> preferable if a standard way to write multi-line classifiers was specified
> by the RST standard.

Are the docstrings also used with Docutils or only with Sphinx?

Maybe it suffices to have the standard way not specified but compatible with
the rST standard.

> Just checking for indentation is actually a bit
> ambiguous, because it is not clear how to parse

>     a : b
>         c

> (is ``c`` a continuation or a description?).  A possibility would be to
> support backslashed-line continuations at the RST level, so the Python
> docstring would look like

>     parameter_name : some long type \\
>                      description
>         Description of the parameter

> but RST would see a single backslash and know what to do; or require that
> continued classifier lines start also with ``: ``:

>     parameter_name : some long type
>                    : description
>         Description of the parameter

> Neither option is strictly fully back-compatible, but I hope one of them,
> or some similar solution, can be adopted.

Another option is a "type" role as alternative markup for long types::

.. role:: type

parameter_name
    :type:`some long type description`
    
    Description of the parameter.
        

Numpydoc and sphinx.ext.napoleon could then implement a transform that
converts the result to be identical to the simple syntax for short "types".

Günter

Re: [Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ant...@be...>]

Replying to Günter Milde:

Are the docstrings also used with Docutils or only with Sphinx?
> Maybe it suffices to have the standard way not specified but compatible
> with
> the rST standard.


Can you clarify what you mean?  My point is essentially that (independently
of sphinx/numpydoc, but simply when embedding rst into a python string
literal) there is no way to have a long classifier list that looks good
both at the level of the python source, and at the level of printed string
(as displayed by ``print(docstring)``).

Another option is a "type" role as alternative markup for long types


This is essentially the approach of "standard" Sphinx docstrings, which are
arguably(?) less legible (at the source level) than both numpydoc and
google style docstrings (see
http://www.sphinx-doc.org/en/stable/ext/napoleon.html#napoleon-marching-toward-legible-docstrings
for discussion).  As a tangent, note that google style docstrings are *not*
rst-compliant (due to the incompatble indent in multiline argument
descriptions), which makes it difficult to decide e.g. what is the correct
syntax when trying to embed a bullet list in an argument description.  It
is for this reason (having well spec'ed rules for including "complex"
syntax items in an argument description) that I'd prefer numpydoc to stay
rst compliant.

Antony

2017-10-04 10:57 GMT-07:00 Antony Lee <ant...@be...>:

> Hi all,
>
> Currently, RST specifies that entries in a definition list may be followed
> by a series of classifiers all on the same line, as in
>
>     term : classifier one : classifier 2
>         Definition
>
> This syntax is used by numpydoc (the de facto docstring standard in the
> scientific Python world) to document individual parameters of callables, as
> in
>
>     parameter_name : type_of_parameter
>         Description of the parameter.
>
> (see https://github.com/numpy/numpy/blob/master/doc/HOWTO_
> DOCUMENT.rst.txt#sections for details).
>
> In general, this approach yields highly legible raw docstrings, and can
> easily be converted into the Sphinx standard field list format (tools to do
> so include numpydoc and sphinx.ext.napoleon).
>
> However, it is awkward to include very long parameter "types" (e.g., an
> enumeration of allowable string values -- which I know is not strictly a
> type, but remains a common API) using this method, due to the restriction
> that the classifiers need to stay in one line.  In practice, this means
> that it is necessary to use a line continuation in the docstring, either
>
>     parameter_name : some long type \
> description
>         Description of parameter.
>
> or
>
>     parameter_name : some long type \
>                      description
>         Description of the parameter.
>
> The first form is ugly in the source, the second form is ugly when the
> docstring is viewed as a string (e.g. by pydoc) as all the spaces before
> ``description`` end up in the literal docstring.  See e.g.
> https://github.com/numpy/numpydoc/issues/87 for a longer discussion of
> the issue.
>
> While numpydoc can of course invent its own syntax, it would seem
> preferable if a standard way to write multi-line classifiers was specified
> by the RST standard.  Just checking for indentation is actually a bit
> ambiguous, because it is not clear how to parse
>
>     a : b
>         c
>
> (is ``c`` a continuation or a description?).  A possibility would be to
> support backslashed-line continuations at the RST level, so the Python
> docstring would look like
>
>     parameter_name : some long type \\
>                      description
>         Description of the parameter
>
> but RST would see a single backslash and know what to do; or require that
> continued classifier lines start also with ``: ``:
>
>     parameter_name : some long type
>                    : description
>         Description of the parameter
>
> Neither option is strictly fully back-compatible, but I hope one of them,
> or some similar solution, can be adopted.
>
> Antony Lee
>
> PS: As suggested on the docutils mailing list page, I am not subscribed to
> the list, so please CC me in the reply.  Thanks!
>

Re: [Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ant...@be...>]

>
> As for the suggestions to extend the rST syntax, I think David is the
> expert that can tell whether this fits into the overal picture and can be

implemented without too many complications.


David has privately replied to me (I assume this was a mistake of
reply-to-me vs. reply-to-mailing list -- apologizing in advance if this was
not the case) indicating cautious interest in the idea of
backslash-line-continuations.  Let's wait until he comments on the mailing
list.
I guess the same mechanism could be used for multi-line section headings,
etc.
Antony

2017-10-07 21:57 GMT-07:00 Antony Lee <ant...@be...>:

> Replying to Günter Milde:
>
> Are the docstrings also used with Docutils or only with Sphinx?
>> Maybe it suffices to have the standard way not specified but compatible
>> with
>> the rST standard.
>
>
> Can you clarify what you mean?  My point is essentially that
> (independently of sphinx/numpydoc, but simply when embedding rst into a
> python string literal) there is no way to have a long classifier list that
> looks good both at the level of the python source, and at the level of
> printed string (as displayed by ``print(docstring)``).
>
> Another option is a "type" role as alternative markup for long types
>
>
> This is essentially the approach of "standard" Sphinx docstrings, which
> are arguably(?) less legible (at the source level) than both numpydoc and
> google style docstrings (see http://www.sphinx-doc.org/en/
> stable/ext/napoleon.html#napoleon-marching-toward-legible-docstrings for
> discussion).  As a tangent, note that google style docstrings are *not*
> rst-compliant (due to the incompatble indent in multiline argument
> descriptions), which makes it difficult to decide e.g. what is the correct
> syntax when trying to embed a bullet list in an argument description.  It
> is for this reason (having well spec'ed rules for including "complex"
> syntax items in an argument description) that I'd prefer numpydoc to stay
> rst compliant.
>
> Antony
>
> 2017-10-04 10:57 GMT-07:00 Antony Lee <ant...@be...>:
>
>> Hi all,
>>
>> Currently, RST specifies that entries in a definition list may be
>> followed by a series of classifiers all on the same line, as in
>>
>>     term : classifier one : classifier 2
>>         Definition
>>
>> This syntax is used by numpydoc (the de facto docstring standard in the
>> scientific Python world) to document individual parameters of callables, as
>> in
>>
>>     parameter_name : type_of_parameter
>>         Description of the parameter.
>>
>> (see https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMEN
>> T.rst.txt#sections for details).
>>
>> In general, this approach yields highly legible raw docstrings, and can
>> easily be converted into the Sphinx standard field list format (tools to do
>> so include numpydoc and sphinx.ext.napoleon).
>>
>> However, it is awkward to include very long parameter "types" (e.g., an
>> enumeration of allowable string values -- which I know is not strictly a
>> type, but remains a common API) using this method, due to the restriction
>> that the classifiers need to stay in one line.  In practice, this means
>> that it is necessary to use a line continuation in the docstring, either
>>
>>     parameter_name : some long type \
>> description
>>         Description of parameter.
>>
>> or
>>
>>     parameter_name : some long type \
>>                      description
>>         Description of the parameter.
>>
>> The first form is ugly in the source, the second form is ugly when the
>> docstring is viewed as a string (e.g. by pydoc) as all the spaces before
>> ``description`` end up in the literal docstring.  See e.g.
>> https://github.com/numpy/numpydoc/issues/87 for a longer discussion of
>> the issue.
>>
>> While numpydoc can of course invent its own syntax, it would seem
>> preferable if a standard way to write multi-line classifiers was specified
>> by the RST standard.  Just checking for indentation is actually a bit
>> ambiguous, because it is not clear how to parse
>>
>>     a : b
>>         c
>>
>> (is ``c`` a continuation or a description?).  A possibility would be to
>> support backslashed-line continuations at the RST level, so the Python
>> docstring would look like
>>
>>     parameter_name : some long type \\
>>                      description
>>         Description of the parameter
>>
>> but RST would see a single backslash and know what to do; or require that
>> continued classifier lines start also with ``: ``:
>>
>>     parameter_name : some long type
>>                    : description
>>         Description of the parameter
>>
>> Neither option is strictly fully back-compatible, but I hope one of them,
>> or some similar solution, can be adopted.
>>
>> Antony Lee
>>
>> PS: As suggested on the docutils mailing list page, I am not subscribed
>> to the list, so please CC me in the reply.  Thanks!
>>
>
>

Re: [Docutils-users] Multi-line classifiers in definition lists

[Guenter M. <mi...@us...>]

On 2017-10-08, Antony Lee wrote:

> Replying to Günter Milde:

>> Are the docstrings also used with Docutils or only with Sphinx?

>From the context below, it seems the docstrings are only used as-is or
with Sphinx, not with standard Docutils.
I am asking, because Sphinx extends the rST syntax with additional roles
and directives.

>> Maybe it suffices to have the standard way not specified but
>> compatible with the rST standard.

> Can you clarify what you mean?

The `napoleon documentation`__ says:

  ReStructuredText is great, but it creates visually dense, hard to read
  docstrings. 

However, reStructuredText is not to blame here, as the dense, hard to read
docstrings are a consequence of the Sphinx docstring conventions. The Numpy
Style is valid rST and extracted docstrings could be used with standard
Docutils (rst2html, say) -- it works but you will loose some of the bells
and whistles like Sphinx-added cross-links.

__ http://www.sphinx-doc.org/en/stable/ext/napoleon.html

> My point is essentially that (independently of sphinx/numpydoc, but
> simply when embedding rst into a python string literal) there is no way
> to have a long classifier list that looks good both at the level of the
> python source, and at the level of printed string (as displayed by
> ``print(docstring)``).

I see your point -- the restriction that term and classifiers in a
definition list need to be on one line.
(Note that similar restrictions exist for, e.g. section headings.)

We have to consider whether use cases that suffer from this restriction
are strong enough to merit a complication of the specification and code.
(This is not blanck rejection of any change to rST syntax. However, 
simpler alternatives may exist for the problematic use cases.)


>> Another option is a "type" role as alternative markup for long types

> This is essentially the approach of "standard" Sphinx docstrings, which are
> arguably(?) less legible (at the source level) than both numpydoc and
> google style docstrings

In the given example, I agree with you and napoleon.

However, in case of long argument type list, falling back to the explicit
markup provided by the Sphinx docstring conventions may even be considered
as improving clarity. Compare ::

    Parameters
    ----------
    arg1 : a long list of argument types that does not fit on a line and\
           therfore must be wrapped
        Description of arg1
    arg2 : str
        Description of arg2

to ::

    Parameters
    ----------
    arg1
        :type:`a long list of argument types that does not fit on a line
	 and therfore must be wrapped`
	 
        Description of arg1
	
    arg2 : str
        Description of arg2

or a convention with nested definition lists::

    Parameters
    ----------
    arg1
        type:
	  a long list of argument types that does not fit on a line
	  and therfore must be wrapped
	 
        Description of arg1
	
    arg2 : str
        Description of arg2


As for the suggestions to extend the rST syntax, I think David is the
expert that can tell whether this fits into the overal picture and can be
implemented without too many complications.

>> A possibility would be to support backslashed-line continuations at
>> the RST level, so the Python docstring would look like

>>     parameter_name : some long type \\
>>                      description
>>         Description of the parameter

>> but RST would see a single backslash and know what to do; or require that
>> continued classifier lines start also with ``: ``:

>>     parameter_name : some long type
>>                    : description
>>         Description of the parameter

Günter

Re: [Docutils-users] Multi-line classifiers in definition lists

[David G. <go...@py...>]

Oops, I meant to reply-all:

On Oct 7, 2017 10:48 PM, "David Goodger" <dgo...@gm...> wrote:

I like the idea of backslash-escape line continuations at the reST level.
It's a general purpose change that could help in other areas too. It might
have side effects though.

Can't work on it right now: on vacation, cycle touring. I'll take a look
when I get back in a week or so.

David Goodger


On Oct 4, 2017 1:37 PM, "Antony Lee" <ant...@be...> wrote:

Hi all,

Currently, RST specifies that entries in a definition list may be followed
by a series of classifiers all on the same line, as in

    term : classifier one : classifier 2
        Definition

This syntax is used by numpydoc (the de facto docstring standard in the
scientific Python world) to document individual parameters of callables, as
in

    parameter_name : type_of_parameter
        Description of the parameter.

(see https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMEN
T.rst.txt#sections for details).

In general, this approach yields highly legible raw docstrings, and can
easily be converted into the Sphinx standard field list format (tools to do
so include numpydoc and sphinx.ext.napoleon).

However, it is awkward to include very long parameter "types" (e.g., an
enumeration of allowable string values -- which I know is not strictly a
type, but remains a common API) using this method, due to the restriction
that the classifiers need to stay in one line.  In practice, this means
that it is necessary to use a line continuation in the docstring, either

    parameter_name : some long type \
description
        Description of parameter.

or

    parameter_name : some long type \
                     description
        Description of the parameter.

The first form is ugly in the source, the second form is ugly when the
docstring is viewed as a string (e.g. by pydoc) as all the spaces before
``description`` end up in the literal docstring.  See e.g.
https://github.com/numpy/numpydoc/issues/87 for a longer discussion of the
issue.

While numpydoc can of course invent its own syntax, it would seem
preferable if a standard way to write multi-line classifiers was specified
by the RST standard.  Just checking for indentation is actually a bit
ambiguous, because it is not clear how to parse

    a : b
        c

(is ``c`` a continuation or a description?).  A possibility would be to
support backslashed-line continuations at the RST level, so the Python
docstring would look like

    parameter_name : some long type \\
                     description
        Description of the parameter

but RST would see a single backslash and know what to do; or require that
continued classifier lines start also with ``: ``:

    parameter_name : some long type
                   : description
        Description of the parameter

Neither option is strictly fully back-compatible, but I hope one of them,
or some similar solution, can be adopted.

Antony Lee

PS: As suggested on the docutils mailing list page, I am not subscribed to
the list, so please CC me in the reply.  Thanks!

------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Docutils-users mailing list
Doc...@li...
https://lists.sourceforge.net/lists/listinfo/docutils-users

Please use "Reply All" to reply to the list.

Re: [Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ant...@be...>]

Hi,
Sorry re-raising an old thread, but I had a quick revisit at this again...
As a reminder, this was about adding backslash-escaped line continuations,
so that one could write

term : info more info \

       and more info

    the definition


the idea being that numpydoc docstrings (by far the most widely used
docstring convention in the scientific python ecosystem) would benefit from
it.
As it turns out the patch to achieve this is relatively simple:

diff --git i/docutils/docutils/statemachine.py
w/docutils/docutils/statemachine.py
index ec5351887..70aa9baea 100644
--- i/docutils/docutils/statemachine.py
+++ w/docutils/docutils/statemachine.py
@@ -311,6 +311,13 @@ class StateMachine(object):
            except IndexError:
                self.line = None
                raise EOFError
+            while self.line.endswith("\\"):
+                try:
+                    self.line_offset += 1
+                    self.line +=
self.input_lines[self.line_offset].lstrip()
+                except IndexError:
+                    self.line = None
+                    raise EOFError
            return self.line
        finally:
            self.notify_observers()


i.e. joining the backslash-escaped lines relatively early in the process.
Please let me know the best way to proceed forward.

Cheers,
Antony

On Mon, Oct 9, 2017 at 5:21 AM David Goodger <go...@py...> wrote:

> Oops, I meant to reply-all:
>
> On Oct 7, 2017 10:48 PM, "David Goodger" <dgo...@gm...> wrote:
>
> I like the idea of backslash-escape line continuations at the reST level.
> It's a general purpose change that could help in other areas too. It might
> have side effects though.
>
> Can't work on it right now: on vacation, cycle touring. I'll take a look
> when I get back in a week or so.
>
> David Goodger
>
>
> On Oct 4, 2017 1:37 PM, "Antony Lee" <ant...@be...> wrote:
>
> Hi all,
>
> Currently, RST specifies that entries in a definition list may be followed
> by a series of classifiers all on the same line, as in
>
>     term : classifier one : classifier 2
>         Definition
>
> This syntax is used by numpydoc (the de facto docstring standard in the
> scientific Python world) to document individual parameters of callables, as
> in
>
>     parameter_name : type_of_parameter
>         Description of the parameter.
>
> (see
> https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#sections
> for details).
>
> In general, this approach yields highly legible raw docstrings, and can
> easily be converted into the Sphinx standard field list format (tools to do
> so include numpydoc and sphinx.ext.napoleon).
>
> However, it is awkward to include very long parameter "types" (e.g., an
> enumeration of allowable string values -- which I know is not strictly a
> type, but remains a common API) using this method, due to the restriction
> that the classifiers need to stay in one line.  In practice, this means
> that it is necessary to use a line continuation in the docstring, either
>
>     parameter_name : some long type \
> description
>         Description of parameter.
>
> or
>
>     parameter_name : some long type \
>                      description
>         Description of the parameter.
>
> The first form is ugly in the source, the second form is ugly when the
> docstring is viewed as a string (e.g. by pydoc) as all the spaces before
> ``description`` end up in the literal docstring.  See e.g.
> https://github.com/numpy/numpydoc/issues/87 for a longer discussion of
> the issue.
>
> While numpydoc can of course invent its own syntax, it would seem
> preferable if a standard way to write multi-line classifiers was specified
> by the RST standard.  Just checking for indentation is actually a bit
> ambiguous, because it is not clear how to parse
>
>     a : b
>         c
>
> (is ``c`` a continuation or a description?).  A possibility would be to
> support backslashed-line continuations at the RST level, so the Python
> docstring would look like
>
>     parameter_name : some long type \\
>                      description
>         Description of the parameter
>
> but RST would see a single backslash and know what to do; or require that
> continued classifier lines start also with ``: ``:
>
>     parameter_name : some long type
>                    : description
>         Description of the parameter
>
> Neither option is strictly fully back-compatible, but I hope one of them,
> or some similar solution, can be adopted.
>
> Antony Lee
>
> PS: As suggested on the docutils mailing list page, I am not subscribed to
> the list, so please CC me in the reply.  Thanks!
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Docutils-users mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-users
>
> Please use "Reply All" to reply to the list.
>
>
>
>

Re: [Docutils-users] Multi-line classifiers in definition lists

[Guenter M. <mi...@us...>]

On 2019-12-11, Antony Lee wrote:

> Hi,
> Sorry re-raising an old thread, but I had a quick revisit at this again...
> As a reminder, this was about adding backslash-escaped line continuations,
> so that one could write

> term : info more info \

>        and more info

>     the definition


> the idea being that numpydoc docstrings (by far the most widely used
> docstring convention in the scientific python ecosystem) would benefit from
> it.

I like the idea but want it to be general: also long section headings,
say, may benefit from this.

> As it turns out the patch to achieve this is relatively simple:

> diff --git i/docutils/docutils/statemachine.py
> w/docutils/docutils/statemachine.py
> index ec5351887..70aa9baea 100644
> --- i/docutils/docutils/statemachine.py
> +++ w/docutils/docutils/statemachine.py
> @@ -311,6 +311,13 @@ class StateMachine(object):
>             except IndexError:
>                 self.line = None
>                 raise EOFError
> +            while self.line.endswith("\\"):
> +                try:
> +                    self.line_offset += 1
> +                    self.line +=
> self.input_lines[self.line_offset].lstrip()
> +                except IndexError:
> +                    self.line = None
> +                    raise EOFError
>             return self.line
>         finally:
>             self.notify_observers()


> i.e. joining the backslash-escaped lines relatively early in the process.

Unfortunately, this breaks our test suite and shows nasty side-effects in a
small dedicated test file.

> Please let me know the best way to proceed forward.

We would need to sort out the problems and side-effects,
test properly,
document,
and then implement it in the development version.

Thanks for the proposal and remainder,

Günter


Test backslash escaping of line ends
------------------------------------

There are side effects, the test suite fails.

Lets have a closer look:

Section headings
----------------

Breaking long section headings fails:

long and broken section heading\
!
-----------------------------------

long section heading with broken underline
-------------------------------\
-------------------


Paragraphs
----------

This is a paragraph. \
It's quite short.

The text of all lines should be there,
even if a line end is
escaped with a backslash \
somewhere in the middle.

The second line of this paragraph
contains an escaped line break \
and is missing in the output!!

Lists
-----

definition lists
````````````````

using a backslash \
escape
    you can break the definition line

escaping line ends: works in the \
                 additional explanations
    too.

definition term
   and definition with backslash \
   escape. Again a line missing!!

field lists
```````````

:the field \
 name:
   and the content

:a long and broken \
 field name: start of content missing if placed on same line!!
 
:field name: and broken \
   content

literal text
--------------

An example::

      Whitespace, newlines, blank lines, 
        and all kinds of markup \
      must be  preserved by literal blocks.

Also ``inline \
literals`` must keep the backslash and the escaped character!

Re: [Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ant...@in...>]

Thanks for providing a nice list of test cases; I'll work on them.
Antony

On Fri, Feb 7, 2020 at 1:41 PM Guenter Milde via Docutils-users <
doc...@li...> wrote:

> On 2019-12-11, Antony Lee wrote:
>
> > Hi,
> > Sorry re-raising an old thread, but I had a quick revisit at this
> again...
> > As a reminder, this was about adding backslash-escaped line
> continuations,
> > so that one could write
>
> > term : info more info \
>
> >        and more info
>
> >     the definition
>
>
> > the idea being that numpydoc docstrings (by far the most widely used
> > docstring convention in the scientific python ecosystem) would benefit
> from
> > it.
>
> I like the idea but want it to be general: also long section headings,
> say, may benefit from this.
>
> > As it turns out the patch to achieve this is relatively simple:
>
> > diff --git i/docutils/docutils/statemachine.py
> > w/docutils/docutils/statemachine.py
> > index ec5351887..70aa9baea 100644
> > --- i/docutils/docutils/statemachine.py
> > +++ w/docutils/docutils/statemachine.py
> > @@ -311,6 +311,13 @@ class StateMachine(object):
> >             except IndexError:
> >                 self.line = None
> >                 raise EOFError
> > +            while self.line.endswith("\\"):
> > +                try:
> > +                    self.line_offset += 1
> > +                    self.line +=
> > self.input_lines[self.line_offset].lstrip()
> > +                except IndexError:
> > +                    self.line = None
> > +                    raise EOFError
> >             return self.line
> >         finally:
> >             self.notify_observers()
>
>
> > i.e. joining the backslash-escaped lines relatively early in the process.
>
> Unfortunately, this breaks our test suite and shows nasty side-effects in a
> small dedicated test file.
>
> > Please let me know the best way to proceed forward.
>
> We would need to sort out the problems and side-effects,
> test properly,
> document,
> and then implement it in the development version.
>
> Thanks for the proposal and remainder,
>
> Günter
>
>
> Test backslash escaping of line ends
> ------------------------------------
>
> There are side effects, the test suite fails.
>
> Lets have a closer look:
>
> Section headings
> ----------------
>
> Breaking long section headings fails:
>
> long and broken section heading\
> !
> -----------------------------------
>
> long section heading with broken underline
> -------------------------------\
> -------------------
>
>
> Paragraphs
> ----------
>
> This is a paragraph. \
> It's quite short.
>
> The text of all lines should be there,
> even if a line end is
> escaped with a backslash \
> somewhere in the middle.
>
> The second line of this paragraph
> contains an escaped line break \
> and is missing in the output!!
>
> Lists
> -----
>
> definition lists
> ````````````````
>
> using a backslash \
> escape
>     you can break the definition line
>
> escaping line ends: works in the \
>                  additional explanations
>     too.
>
> definition term
>    and definition with backslash \
>    escape. Again a line missing!!
>
> field lists
> ```````````
>
> :the field \
>  name:
>    and the content
>
> :a long and broken \
>  field name: start of content missing if placed on same line!!
>
> :field name: and broken \
>    content
>
> literal text
> --------------
>
> An example::
>
>       Whitespace, newlines, blank lines,
>         and all kinds of markup \
>       must be  preserved by literal blocks.
>
> Also ``inline \
> literals`` must keep the backslash and the escaped character!
>
>
>
> _______________________________________________
> Docutils-users mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-users
>
> Please use "Reply All" to reply to the list.
>

Re: [Docutils-users] Multi-line classifiers in definition lists

[Guenter M. <mi...@us...>]

On 2020-02-06, Antony Lee wrote:

> Thanks for providing a nice list of test cases; I'll work on them.

This is good news.

Are you aware of the `Docutils test suite`__?
I did not include examples of every test failure there in my test document.

__ https://docutils.sourceforge.io/docs/dev/testing.html


Günter

Re: [Docutils-users] Multi-line classifiers in definition lists

[Antony L. <ann...@gm...>]

No, but thanks for pointing to it.
Antony

On Fri, Feb 7, 2020 at 5:45 PM Guenter Milde via Docutils-users <
doc...@li...> wrote:

> On 2020-02-06, Antony Lee wrote:
>
> > Thanks for providing a nice list of test cases; I'll work on them.
>
> This is good news.
>
> Are you aware of the `Docutils test suite`__?
> I did not include examples of every test failure there in my test document.
>
> __ https://docutils.sourceforge.io/docs/dev/testing.html
>
>
> Günter
>
>
>
> _______________________________________________
> Docutils-users mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-users
>
> Please use "Reply All" to reply to the list.
>