docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:bugs] Re: #427 Inconsistent sentence spacing in man pages using rst2man.py

[Günter M. <mi...@us...>]

> most important is to not break post-processors, the layout is random anyway.

It is not completely random (even less in Postscript or PDF output where the increased spacing after sentences is clearly visible, I don't know whether/how it is preserved in HTML).

I suggest documenting the issue of sentence spacing (recommending to follow the (g|t)roff [input conventions](https://www.gnu.org/software/groff/manual/groff.html#Input-Conventions)) and then closing this bug report.

Adding support for "raw " directive and role or a "preamble" option may be valid [feature-requests](https://sourceforge.net/p/docutils/feature-requests).


---

** [bugs:#427] Inconsistent sentence spacing in man pages using rst2man.py**

**Status:** open
**Labels:** manpage writer 
**Created:** Wed Oct 13, 2021 07:00 PM UTC by jei23jkfd
**Last Updated:** Wed Sep 07, 2022 12:09 PM UTC
**Owner:** nobody


## Operating system info

I'm running Docutils 0.18b2.dev r8848 with Python 3.9.

## Description of bug

The roff language has a very subtle requirement. It enforces semantic line breaks. That is, any roff document must end a sentence with a line break.

For example, this is incorrect:

~~~
This is a sentence. This is another sentence.
~~~

We have to insert a line break after each sentence:

~~~
This is a sentence.
This is another sentence.
~~~

The semantic line breaks are used to add optional sentence spacing. It uses the line breaks to detect when a period (or question or exclamation mark) represents the end of a sentence and then adds an optional extra space when displaying it. The relevant documentation for groff(1) and mandoc(1) is at

- https://www.gnu.org/software/groff/manual/groff.html#Sentences
- https://mandoc.bsd.lv/man/roff.7.html#Sentence_Spacing

## Minimal example

Here is a minimal example that shows how the reST man page writer has this bug:

~~~rst
###
mwe
###
a minimal example
#################
:Date: October 13, 2021
:Manual section: 1
:Manual group: Testing Docutils
:Version: mwe 0.1.0

Synopsis
========
| mwe [**-aq**] [**-b** *file*] [**\--long-long** *which*] *file \...*

Description
===========
To find the common attributes of a variety of objects, it is necessary
to begin, by surveying the *objects* themselves in the concrete. Let us
therefore advert successively to the various modes of action, and
arrangements of human affairs, which are classed, by universal or widely
spread opinion, as Just or as Unjust. The things well known to excite
the sentiments associated with those names, are of a very multifarious
character. I shall pass them rapidly in review, without studying any
particular arrangement.
The previous line will have been spaced with two spaces.

Options
=======
Its arguments are as follows:

-a                         Do all.
-q                         Be quiet.
-b file                    Do everything to *file*.
--long-meme which          Chooses the long named *which*.

Environment
===========
mwe is not affected by environment variables.

Exit status
===========
mwe exits 0 on success.
~~~

If you convert this with `rst2man.py mwe.rst mwe.1` and view it with `man ./mwe.1` you will notice the issue easily: some sentences in the DESCRIPTION section end with 1 space and some sentences end with 2 spaces.

## Possible solutions

The most complete solution would be to automatically detect where sentences end in the input and add line breaks as required in the man page output. This is what Pandoc does. However, it requires keeping a list of abbreviations for every language so as to not have false positives: you don't want to add a line break when someone uses "e.g.".

Another solution would be to use the `.ss` macro to remove the extra space at the end of any sentences. Then Docutils wouldn't have to conform to roff syntax. However, this would not work with mandoc(1) as the developer of that program has decided not to support `.ss`. This is what Asciidoctor does.

Please let me know if you have any questions.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #456 Latex Blank lines in Block Math Environment

[Günter M. <mi...@us...>]

- **status**: pending-works-for-me --> closed-invalid
- **Comment**:

Closing this now, as there is no example of invalid LaTeX code generation.



---

** [bugs:#456] Latex Blank lines in Block Math Environment**

**Status:** closed-invalid
**Created:** Thu Sep 01, 2022 09:14 PM UTC by Daniel James Perry
**Last Updated:** Wed Nov 30, 2022 09:52 PM UTC
**Owner:** nobody


The latex writer has a serious bug regarding block math displays.
According to the book *More Math into Latex*:
> No blank lines are permitted within an equation or equation* environment.

But if you look at the code that's **exactly what it does.** The resulting Latex output is incorrect.
Please fix this as soon as possible.

BTW get with the times and start using git please.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #34 incomplete filtering of FilterMessages

[Günter M. <mi...@us...>]

- **status**: open --> closed-fixed
- **Group**:  --> None
- **Comment**:

The reported problem cannot be reproduced with the current repository version.
It seems to be fixed by an alternative patch.



---

** [patches:#34] incomplete filtering of FilterMessages**

**Status:** closed-fixed
**Group:** None
**Created:** Mon Sep 18, 2006 11:47 PM UTC by Santiago Gala
**Last Updated:** Mon Sep 18, 2006 11:47 PM UTC
**Owner:** nobody


FilterMessages is supposed to filter messages below the
report\_level. It has two problems, though:

\* It does not remove nodes in 
self.document.transform\_messages
\* It does not remove the refids in 
the backrefs to the removed nodes

The attached patch does both things.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #82 Fix odf_odt fails on some inline elements

[Günter M. <mi...@us...>]

- **status**: open --> closed-out-of-date
- **Group**:  --> None
- **Comment**:

Closing this because of long silence.

Still, a test case would be fine.



---

** [patches:#82] Fix odf_odt fails on some inline elements**

**Status:** closed-out-of-date
**Group:** None
**Created:** Sat Aug 13, 2011 12:30 AM UTC by Anonymous
**Last Updated:** Sat Aug 13, 2011 12:30 AM UTC
**Owner:** nobody


If the odf\_odt writer is used on an "inline" element with no class names, it dies with an UnboundLocalError. Checking for 0 classes fixes this.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #113 writers/odf_odt: Use only ASCII filenames in ODF packages

[Günter M. <mi...@us...>]

- **status**: open --> closed-fixed
- **Comment**:

The related Debian bug was closed in 2013. Thanks for the patch.



---

** [patches:#113] writers/odf_odt: Use only ASCII filenames in ODF packages**

**Status:** closed-fixed
**Group:** None
**Created:** Mon Aug 05, 2013 02:07 PM UTC by Michael Schutte
**Last Updated:** Wed Feb 11, 2015 12:59 PM UTC
**Owner:** nobody
**Attachments:**

- [odt-writer-ascii-filenames.diff](https://sourceforge.net/p/docutils/patches/113/attachment/odt-writer-ascii-filenames.diff) (933 Bytes; text/x-patch)


The odf_odt writer embeds images in its output files and uses the original filenames as part of the embedded filenames.  Since the OpenDocument standard does not specify the filename charset, recode to ASCII (dropping non-representable characters) to be on the safe side.

The actual reason that brought about this patch is an invalid assumption about character sets in docutils.writers.odf_odt.Writer.store_embedded_files().  This has been reported as Debian bug <http://bugs.debian.org/714317>.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #114 rst2odt_prepstyles.py: Port to ElementTree

[Günter M. <mi...@us...>]

- **status**: open --> pending-remind
- **Comment**:

With [bugs:#255] solved and changes to the source (to make it Py3k-conform), is there an advantage of using elementtree that makes it worth updating, testing, and applying the patch ? 




---

** [patches:#114] rst2odt_prepstyles.py: Port to ElementTree**

**Status:** pending-remind
**Group:** None
**Created:** Mon Aug 05, 2013 02:12 PM UTC by Michael Schutte
**Last Updated:** Sat Aug 23, 2014 09:32 PM UTC
**Owner:** nobody
**Attachments:**

- [rst2odt_prepstyles-elementtree.diff](https://sourceforge.net/p/docutils/patches/114/attachment/rst2odt_prepstyles-elementtree.diff) (1.8 kB; text/x-patch)


rst2odt_prepstyles.py currently uses lxml to parse, modify and write XML.  The attached patch makes it use ElementTree instead, which has been part of Python's standard library since 2.5.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #194 Deprecate PEP 263 coding slugs support

[Günter M. <mi...@us...>]

- **status**: open --> closed-accepted
- **Group**:  --> None
- **Comment**:

The way forward is now specified in the RELEASE-NOTES.



---

** [patches:#194] Deprecate PEP 263 coding slugs support**

**Status:** closed-accepted
**Group:** None
**Created:** Thu Jun 09, 2022 10:48 PM UTC by Adam  Turner
**Last Updated:** Fri Dec 02, 2022 11:30 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-Deprecate-PEP-263-coding-slugs.patch](https://sourceforge.net/p/docutils/patches/194/attachment/0001-Deprecate-PEP-263-coding-slugs.patch) (5.5 kB; application/octet-stream)


Python 3 uses utf-8 as the encoding for Python source files, there is no longer a compelling use-case for the support, which adds complexity to the IO implementation.

I propose deprecating support for removal in 1.0, but 2.0 might be a better option.

Support was added in [r4506].

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #201 Adapt tests to Sphinx 6.0.0

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed
- **Comment**:

A version of the patch with Pygments-version-dependent data is now applied to the repository.



---

** [patches:#201] Adapt tests to Sphinx 6.0.0**

**Status:** open-fixed
**Group:** None
**Created:** Wed Jan 04, 2023 03:51 PM UTC by Daniel Garcia Moreno
**Last Updated:** Tue Jan 10, 2023 04:42 PM UTC
**Owner:** nobody
**Attachments:**

- [sphinx-6.0.0.patch](https://sourceforge.net/p/docutils/patches/201/attachment/sphinx-6.0.0.patch) (5.8 kB; text/x-patch)


Sphinx 6.0.0 changes the format a bit, this patch fixes some tests that are broken with the latest release.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] Re: #202 RST parser: Add source lines to admonitions and definitionlists

[Günter M. <mi...@us...>]

Bug 465 is a duplicate of  [feature-requests:#41].


---

** [patches:#202] RST parser: Add source lines to admonitions and definitionlists**

**Status:** open
**Group:** None
**Created:** Mon Jan 23, 2023 07:44 PM UTC by Hood Chatham
**Last Updated:** Mon Jan 23, 2023 07:45 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-Add-source-lines-to-admonitions-and-definitionlists.patch](https://sourceforge.net/p/docutils/patches/202/attachment/0001-Add-source-lines-to-admonitions-and-definitionlists.patch) (1.8 kB; text/x-patch)


Work on bug #465


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #36 prevent accidential file overwrites by wildcard expansion

[Günter M. <mi...@us...>]

- **summary**: option for disallowing file overwrites --> prevent accidential file overwrites by wildcard expansion
- **status**: pending --> open-accepted
- **Group**:  --> Default
- **Comment**:

The upcoming change to the command line API is now announced in the RELEASE-NOTES.



---

** [feature-requests:#36] prevent accidential file overwrites by wildcard expansion**

**Status:** open-accepted
**Group:** Default
**Created:** Fri Mar 15, 2013 10:43 PM UTC by Jakub Wilk
**Last Updated:** Fri Dec 02, 2022 01:07 AM UTC
**Owner:** nobody


A Debian user complained that if you use a wildcard as input file name, and the wildcard unexpectedly expands to two names, then rst2html will overwrite your files:
http://bugs.debian.org/654690

Would if be possible to add an option for not overwriting output files, that you could add to your configuration file, and later override from command line if needed?



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #94 Should docutils whine about nesting simple body elements?

[Günter M. <mi...@us...>]

Ticket moved from /p/docutils/bugs/466/


---

** [feature-requests:#94] Should docutils whine about nesting simple body elements?**

**Status:** open
**Group:** None
**Created:** Thu Feb 09, 2023 09:55 AM UTC by Julien Palard
**Last Updated:** Tue Apr 18, 2023 11:29 AM UTC
**Owner:** nobody


Context:

[This](https://github.com/python/cpython/blob/45fa12aec8f508c224a1521cfe3ae597f1026264/Doc/tools/extensions/pyspecific.py#L149) call to `nested_parse` adds a `Paragraph` node to a `Paragraph` node,  this look forbidden by [the doc](https://docutils.sourceforge.io/docs/ref/doctree.html#body-elements), and it also look to misbehave sphinx-side, when translating the document, where we completly loose the nested paragraph once translated.

It renders properly though when **not** translated.

The linked `nested_parse` returns:

```xml
<paragraph classes="availability">
    <pending_xref refdoc="index" refdomain="std" refexplicit="True" reftarget="availability" reftype="ref" refwarn="True">
        <inline classes="xref std std-ref">
            Availability
    : 
    Unix, not Emscripten, not WASI.
    <paragraph>
        Hello world I'm the content of the "availability" directive.
```

If it's really something that should not be done, maybe some kind of warning at build time could help debugging those issues?



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #467 sdist is missing tox.ini

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed
- **Comment**:

Thank you for pointing this out.
Packaging the repository now includes tox.ini, so it should be fine in the next release.



---

** [bugs:#467] sdist is missing tox.ini**

**Status:** open-fixed
**Created:** Thu Feb 23, 2023 12:48 PM UTC by Marcel Telka
**Last Updated:** Thu Feb 23, 2023 12:48 PM UTC
**Owner:** nobody


The sdist package at PyPI is missing tox.ini file.  Please add the file to sdist to make downstream testing easier.  Thank you.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Update tests for Pygments 2.14

[Guenter M. <mi...@us...>]

Dear Adam,

On 2023-04-06, Adam Turner wrote:

> Pygments 2.14, released in January__, contains changes__ in the Python
> lexer which break ``test_code``, ``test_code_long``, and ``test_include``.

> __ https://pygments.org/docs/changelog/#version-2-14-0
> __ https://github.com/pygments/pygments/commit/147b22face65617514ccfa8512c6b097b07cad4c

> I have attached a sample patch to fix these and maintain support in the
> tests for Pygments 2.13 and earlier. 

Thank you. Do you think this should go to 0.20?

In the proposed patch, the Pygments version parsing fails for pre-release
versions (for examples, c.f. https://pygments.org/docs/changelog/).

The patch below demonstrates an alternative parsing that is a bit more robust.


> A different approach would be to declare that the tests only support
> the latest version of Pygments (it is currently unclear which version
> of Pygments is our minimum supported).

This would mean to skip the tests not only if Pygments is not installed at
all but also for a non-matching version.

Would be possible, but has the disadvantage of not testing versions that
OTOH still work fine "in praxi". (And that developers with
distribution-installed Pygments that is not bleeding edge may fail to find
regressions in the syntax parsing.)
I favour the "versioned" tests (keeping the old version until Pagments 2.14
is in Debian/stable).


Thanks,
Günter


diff --git a/docutils/test/test_parsers/test_rst/test_directives/test_code.py b/docutils/test/test_parsers/test_rst/test_directives/test_code.py
index 7985d1b65..6fab49644 100755
--- a/docutils/test/test_parsers/test_rst/test_directives/test_code.py
+++ b/docutils/test/test_parsers/test_rst/test_directives/test_code.py
@@ -9,6 +9,7 @@
 """
 
 from pathlib import Path
+import re
 import sys
 import unittest
 
@@ -23,12 +24,12 @@
 from docutils.utils.code_analyzer import with_pygments
 
 try:
-    from pygments import __version__ as _pygments_ver
+    import pygments
 except ImportError:
-    _pygments_ver = ''
-    PYGMENTS_2_14_PLUS = False
+    PYGMENTS_2_14_PLUS = None
 else:
-    PYGMENTS_2_14_PLUS = tuple(map(int, _pygments_ver.split('.'))) >= (2, 14)
+    _pv = re.match(r'^([0-9]+)\.([0-9]*)', pygments.__version__)
+    PYGMENTS_2_14_PLUS = (int(_pv[1]), int(_pv[2])) >= (2, 14)
 
 
 class ParserTestCase(unittest.TestCase):

Re: [Docutils-develop] Release 0.20

[Guenter M. <g....@qu...>]

Dear Adam, dear Docutils developers,

Am  6.04.23 schrieb Adam Turner:

...
> > I left the decision about the end state of this transition open...

> A decision to make later, and one that doesn't block the 0.20 release!

Yes and no: if we want to give users advise on a stable recipe to avoid
beeing hit by the default-change, we would need agreement of what will
(most likely) be kept stable.

The API documentation "publisher.txt" now has the example

    output = bytes(publish_string(…))

(which depends on `OutputString` features).


...

> > (How about starting with annotating "core.py". [...]

> I have a patch for adding type hints to Docutils, but wanted to wait
> until releasing Docutils 0.20 so as not to add major new changes to
> the repository.

Agreed.

> >> As far as I can tell, the only unresolved point at the moment ahead
> >> of releasing Docutils 0.20 is the future of ``publish_bytes``.

In the course of the latest changes and documentation updates, I found about
two issues:

* The parts returned by publish_parts() have (since ages) an item
  "encoding" (holding the `output_encoding` setting).

  This can (and IMO should) be easily complemented with "errors"
  (holding the `output_encoding_error_handler` setting).

* The "null" writer currently produces the output ``None``, which is not
  `str` nor `bytes`.

  Currently, ``publish_string(writer_name="null", auto_encode=True)``
  returns ``OutputString("None", encoding='utf-8', errors='strict')``.

  This should be either ``None`` (which requires special casing for None)
  or the empty string (which would make the output value type conforming to
  the documentation).

I have prepared two small patches, that could either go into 0.20 or 0.21
(see below).
What do you think?

Günter

---

>From 4f1b402a033cfc51c215469bfe89e0fbfabbc8af Mon Sep 17 00:00:00 2001
From: milde <mi...@us...>
Date: Wed, 12 Apr 2023 17:21:17 +0200
Subject: [PATCH] Add "errors" to the parts provided by all writers.

The new generic part "errors" contains the
`output_encoding_error_handler` setting,
which may make-or-brake encoding ``parts['whole']``.
---
 docutils/docs/api/publisher.txt                         | 5 ++++-
 docutils/docutils/core.py                               | 2 +-
 docutils/docutils/writers/__init__.py                   | 2 ++
 docutils/test/test_writers/test_html4css1_parts.py      | 1 +
 docutils/test/test_writers/test_html5_polyglot_parts.py | 1 +
 docutils/test/test_writers/test_latex2e_misc.py         | 1 +
 6 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/docutils/docs/api/publisher.txt b/docutils/docs/api/publisher.txt
index 89810ff..a18e67d 100644
--- a/docutils/docs/api/publisher.txt
+++ b/docutils/docs/api/publisher.txt
@@ -136,5 +136,8 @@ Parts Provided By All Writers

 _`encoding`
-    The output encoding setting.
+    The `output_encoding`_ setting.
+
+_`errors`
+    The `output_encoding_error_handler`_ setting.

 _`version`
diff --git a/docutils/docutils/core.py b/docutils/docutils/core.py
index e19d6ce..085e5cd 100644
--- a/docutils/docutils/core.py
+++ b/docutils/docutils/core.py
@@ -491,5 +491,5 @@ def publish_parts(source, source_path=None, source_class=io.StringInput,

        parts = publish_parts(...)
-       body = parts['body'].encode(parts['encoding'])
+       body = parts['body'].encode(parts['encoding'], parts['errors'])

     See the `API documentation`__ for details on the provided parts.
diff --git a/docutils/docutils/writers/__init__.py b/docutils/docutils/writers/__init__.py
index 84be8f6..cba09f4 100644
--- a/docutils/docutils/writers/__init__.py
+++ b/docutils/docutils/writers/__init__.py
@@ -95,4 +95,6 @@ def assemble_parts(self):
         self.parts['whole'] = self.output
         self.parts['encoding'] = self.document.settings.output_encoding
+        self.parts['errors'] = (
+            self.document.settings.output_encoding_error_handler)
         self.parts['version'] = docutils.__version__

diff --git a/docutils/test/test_writers/test_html4css1_parts.py b/docutils/test/test_writers/test_html4css1_parts.py
index 8bd0dee..6dcfbd7 100755
--- a/docutils/test/test_writers/test_html4css1_parts.py
+++ b/docutils/test/test_writers/test_html4css1_parts.py
@@ -76,4 +76,5 @@ def format_output(self, parts):
         del parts['head_prefix']
         del parts['encoding']
+        del parts['errors']
         del parts['version']
         # remove standard portions:
diff --git a/docutils/test/test_writers/test_html5_polyglot_parts.py b/docutils/test/test_writers/test_html5_polyglot_parts.py
index acd959d..799ee98 100644
--- a/docutils/test/test_writers/test_html5_polyglot_parts.py
+++ b/docutils/test/test_writers/test_html5_polyglot_parts.py
@@ -74,4 +74,5 @@ def format_output(self, parts):
         del parts['head_prefix']
         del parts['encoding']
+        del parts['errors']
         del parts['version']
         # remove standard portions:
diff --git a/docutils/test/test_writers/test_latex2e_misc.py b/docutils/test/test_writers/test_latex2e_misc.py
index 32b1edc..74b2813 100644
--- a/docutils/test/test_writers/test_latex2e_misc.py
+++ b/docutils/test/test_writers/test_latex2e_misc.py
@@ -69,4 +69,5 @@ def test_publish_parts(self):
             'docinfo',
             'encoding',
+            'errors',
             'fallbacks',
             'head_prefix',
--
libgit2 1.1.0


>From 30d5554ee8d1015d1c3ff7c0d1862bd003fcb5b7 Mon Sep 17 00:00:00 2001
From: milde <mi...@us...>
Date: Wed, 12 Apr 2023 17:30:10 +0200
Subject: [PATCH] The "null" writer now sets the output to the empty string ''.

Bring the "null" writer behaviour in line with other writers
to facilitate keeping a consistent "core" API.

Makes `io.StringOutput` and `core.publish_string()` conform to the
documented API without requiring a special case for ``output == None``.
---
 docutils/docutils/writers/null.py       | 2 +-
 docutils/test/test_writers/test_null.py | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/docutils/docutils/writers/null.py b/docutils/docutils/writers/null.py
index 6c30627..0d4a919 100644
--- a/docutils/docutils/writers/null.py
+++ b/docutils/docutils/writers/null.py
@@ -19,3 +19,3 @@ class Writer(writers.UnfilteredWriter):
 
     def translate(self):
-        pass
+        self.output = ''
diff --git a/docutils/test/test_writers/test_null.py b/docutils/test/test_writers/test_null.py
index 47890fd..5880ac3 100755
--- a/docutils/test/test_writers/test_null.py
+++ b/docutils/test/test_writers/test_null.py
@@ -34,7 +34,6 @@ def test_publish(self):
                             'strict_visitor': True,
                         },
+                        auto_encode=False,
                     )
-                    if isinstance(output, bytes):
-                        output = output.decode('utf-8')
                     self.assertEqual(output, case_expected)
 
@@ -46,5 +45,5 @@ def test_publish(self):
 This is a paragraph.
 """,
-None]
+'']
 ]
 
--
libgit2 1.1.0

Re: [Docutils-develop] Release 0.20

[Guenter M. <mi...@us...>]

Dear Adam, dear Docutils developers,

Am  6.04.23 schrieb Adam Turner:

...
> > I left the decision about the end state of this transition open...

> A decision to make later, and one that doesn't block the 0.20 release!

Yes and no: if we want to give users advise on a stable recipe to avoid
beeing hit by the default-change, we would need agreement of what will
(most likely) be kept stable.

The API documentation "publisher.txt" now has the example

    output = bytes(publish_string(…))

(which depends on `OutputString` features).


...

> > (How about starting with annotating "core.py". [...]

> I have a patch for adding type hints to Docutils, but wanted to wait
> until releasing Docutils 0.20 so as not to add major new changes to
> the repository.

Agreed.

> >> As far as I can tell, the only unresolved point at the moment ahead
> >> of releasing Docutils 0.20 is the future of ``publish_bytes``.

In the course of the latest changes and documentation updates, I found about
two issues:

* The parts returned by publish_parts() have (since ages) an item
  "encoding" (holding the `output_encoding` setting).

  This can (and IMO should) be easily complemented with "errors"
  (holding the `output_encoding_error_handler` setting).

* The "null" writer currently produces the output ``None``, which is not
  `str` nor `bytes`.

  Currently, ``publish_string(writer_name="null", auto_encode=True)``
  returns ``OutputString("None", encoding='utf-8', errors='strict')``.

  This should be either ``None`` (which requires special casing for None)
  or the empty string (which would make the output value type conforming to
  the documentation).

I have prepared two small patches, that could either go into 0.20 or 0.21
(see below).
What do you think?

Günter

---

>From 4f1b402a033cfc51c215469bfe89e0fbfabbc8af Mon Sep 17 00:00:00 2001
From: milde <mi...@us...>
Date: Wed, 12 Apr 2023 17:21:17 +0200
Subject: [PATCH] Add "errors" to the parts provided by all writers.

The new generic part "errors" contains the
`output_encoding_error_handler` setting,
which may make-or-brake encoding ``parts['whole']``.
---
 docutils/docs/api/publisher.txt                         | 5 ++++-
 docutils/docutils/core.py                               | 2 +-
 docutils/docutils/writers/__init__.py                   | 2 ++
 docutils/test/test_writers/test_html4css1_parts.py      | 1 +
 docutils/test/test_writers/test_html5_polyglot_parts.py | 1 +
 docutils/test/test_writers/test_latex2e_misc.py         | 1 +
 6 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/docutils/docs/api/publisher.txt b/docutils/docs/api/publisher.txt
index 89810ff..a18e67d 100644
--- a/docutils/docs/api/publisher.txt
+++ b/docutils/docs/api/publisher.txt
@@ -136,5 +136,8 @@ Parts Provided By All Writers

 _`encoding`
-    The output encoding setting.
+    The `output_encoding`_ setting.
+
+_`errors`
+    The `output_encoding_error_handler`_ setting.

 _`version`
diff --git a/docutils/docutils/core.py b/docutils/docutils/core.py
index e19d6ce..085e5cd 100644
--- a/docutils/docutils/core.py
+++ b/docutils/docutils/core.py
@@ -491,5 +491,5 @@ def publish_parts(source, source_path=None, source_class=io.StringInput,

        parts = publish_parts(...)
-       body = parts['body'].encode(parts['encoding'])
+       body = parts['body'].encode(parts['encoding'], parts['errors'])

     See the `API documentation`__ for details on the provided parts.
diff --git a/docutils/docutils/writers/__init__.py b/docutils/docutils/writers/__init__.py
index 84be8f6..cba09f4 100644
--- a/docutils/docutils/writers/__init__.py
+++ b/docutils/docutils/writers/__init__.py
@@ -95,4 +95,6 @@ def assemble_parts(self):
         self.parts['whole'] = self.output
         self.parts['encoding'] = self.document.settings.output_encoding
+        self.parts['errors'] = (
+            self.document.settings.output_encoding_error_handler)
         self.parts['version'] = docutils.__version__

diff --git a/docutils/test/test_writers/test_html4css1_parts.py b/docutils/test/test_writers/test_html4css1_parts.py
index 8bd0dee..6dcfbd7 100755
--- a/docutils/test/test_writers/test_html4css1_parts.py
+++ b/docutils/test/test_writers/test_html4css1_parts.py
@@ -76,4 +76,5 @@ def format_output(self, parts):
         del parts['head_prefix']
         del parts['encoding']
+        del parts['errors']
         del parts['version']
         # remove standard portions:
diff --git a/docutils/test/test_writers/test_html5_polyglot_parts.py b/docutils/test/test_writers/test_html5_polyglot_parts.py
index acd959d..799ee98 100644
--- a/docutils/test/test_writers/test_html5_polyglot_parts.py
+++ b/docutils/test/test_writers/test_html5_polyglot_parts.py
@@ -74,4 +74,5 @@ def format_output(self, parts):
         del parts['head_prefix']
         del parts['encoding']
+        del parts['errors']
         del parts['version']
         # remove standard portions:
diff --git a/docutils/test/test_writers/test_latex2e_misc.py b/docutils/test/test_writers/test_latex2e_misc.py
index 32b1edc..74b2813 100644
--- a/docutils/test/test_writers/test_latex2e_misc.py
+++ b/docutils/test/test_writers/test_latex2e_misc.py
@@ -69,4 +69,5 @@ def test_publish_parts(self):
             'docinfo',
             'encoding',
+            'errors',
             'fallbacks',
             'head_prefix',
--
libgit2 1.1.0


>From 30d5554ee8d1015d1c3ff7c0d1862bd003fcb5b7 Mon Sep 17 00:00:00 2001
From: milde <mi...@us...>
Date: Wed, 12 Apr 2023 17:30:10 +0200
Subject: [PATCH] The "null" writer now sets the output to the empty string ''.

Bring the "null" writer behaviour in line with other writers
to facilitate keeping a consistent "core" API.

Makes `io.StringOutput` and `core.publish_string()` conform to the
documented API without requiring a special case for ``output == None``.
---
 docutils/docutils/writers/null.py       | 2 +-
 docutils/test/test_writers/test_null.py | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/docutils/docutils/writers/null.py b/docutils/docutils/writers/null.py
index 6c30627..0d4a919 100644
--- a/docutils/docutils/writers/null.py
+++ b/docutils/docutils/writers/null.py
@@ -19,3 +19,3 @@ class Writer(writers.UnfilteredWriter):
 
     def translate(self):
-        pass
+        self.output = ''
diff --git a/docutils/test/test_writers/test_null.py b/docutils/test/test_writers/test_null.py
index 47890fd..5880ac3 100755
--- a/docutils/test/test_writers/test_null.py
+++ b/docutils/test/test_writers/test_null.py
@@ -34,7 +34,6 @@ def test_publish(self):
                             'strict_visitor': True,
                         },
+                        auto_encode=False,
                     )
-                    if isinstance(output, bytes):
-                        output = output.decode('utf-8')
                     self.assertEqual(output, case_expected)
 
@@ -46,5 +45,5 @@ def test_publish(self):
 This is a paragraph.
 """,
-None]
+'']
 ]
 
--
libgit2 1.1.0

Re: [Docutils-develop] Release 0.20

[Adam T. <aat...@ou...>]

Dear Günter,

> While I favour keeping "auto_encode" over a seprate `publish_bytes()`
> function, I left the decision about the end state of this transition open...

A decision to make later, and one that doesn't block the 0.20 release!

> This will become easier with type annotations in the source.

> (How about starting with annotating "core.py". I only postponed this because
> I don't know whether a partially type-hinted module will interfere with the
> present 3rd-party type hint stubs.)

I have a patch for adding type hints to Docutils, but wanted to wait
until releasing Docutils 0.20 so as not to add major new changes to
the repository.

>> If you have time, perhaps you could commit your patch (or the latest
>> version thereof), and we could make progress from there? As far as I
>> can tell, the only unresolved point at the moment ahead of releasing
>> Docutils 0.20 is the future of ``publish_bytes``.

> This is now [r9336].
> A patch removing `publish_bytes` waits in a branch in my local Git repo.

I think to un-block the release we should apply the patch removing
``publish_bytes``.

Thanks,
Adam

[Docutils-develop] Change "math-output" default to MathML for HTML5 writer.

[Guenter M. <mi...@us...>]

Dear Docutils developers,

since January 2023, Chrome supports MathML (again), so now MathML support
is in Firefox, Chrome, Edge, Safari, the Android HTML view, and webkit.

As the Latex2MathML converter supports more math commands, is easier to
maintain and (potentially) offers better rendering, I propose to change
the default math-output for HTML5 from "HTML" to "MathML" in Docutils 0.22
and announce this change in 0.20.

For the HTML4 writer, I propose to keep the default at "HTML math.css",
because MathML is not officially supported by its output format
XHTML 1.0 Transitional.

Günter

Re: [Docutils-develop] Release 0.20

[Guenter M. <mi...@us...>]

Dear Adam,

On 2023-04-06, Adam Turner wrote:

>>> The main use(s) here would be for publishing binary formats (e.g. ODT)
>>> to a ``bytes`` object in memory rather than writing to disk, or for
>>> when call-sites use a non-unicode ``output_encoding`` setting. 

>> IMV, the extended ``publish_string()`` providing

>>    publish_string(..., auto_encode=False) --> OutString
>>    publish_string(..., auto_encode=True) --> bytes
>     
>> with `OutString` beeing 100% compatible with `str` and easily convertible
>> to bytes via ``bytes(result)`` can cater for such needs. 

>> (Also, is should not be too surprising that `publish_string` returns a
>> `bytes` instance if the user tells it to "auto_encode".)

> Fair enough, though I suppose I had seen ``auto_encode`` as part of
> moving to a position where (eventually) ``publish_string`` always
> returns an instance of Python's ``str`` class, rather than the current
> overloaded return type. 

While I favour keeping "auto_encode" over a seprate `publish_bytes()`
function, I left the decision about the end state of this transition open...

> In the scenario I was imagining, users of
> Docutils who know that they want ``bytes`` output could change to using
> ``publish_bytes`` in Docutils 0.20, and not need to worry about the
> future of ``publish_string`` or the ``auto_encode`` argument. (There
> is also an argument for self-documenting code, in that if you know that
> you want ``bytes``, using the function with ``bytes`` in the name helps
> a non-expert reader to understand what is going on.)

I imagine that in absence of `publish_bytes` users knowing they need an
"encoded string" (i.e. `bytes`) will find `publish_string` and either
just use ``bytes(publish_string(...))`` or read the help string or docs and
use ``publish_string(..., auto_encode=True)``.
This will become easier with type annotations in the source.

(How about starting with annotating "core.py". I only postponed this because
I don't know whether a partially type-hinted module will interfere with the
present 3rd-party type hint stubs.)

> If you have time, perhaps you could commit your patch (or the latest
> version thereof), and we could make progress from there? As far as I
> can tell, the only unresolved point at the moment ahead of releasing
> Docutils 0.20 is the future of ``publish_bytes``.

This is now [r9336].
A patch removing `publish_bytes` waits in a branch in my local Git repo.

Thanks,
Günter

Re: [Docutils-develop] Release 0.20

[Adam T. <aat...@ou...>]

Dear Günter,

>> The main use(s) here would be for publishing binary formats (e.g. ODT)
>> to a ``bytes`` object in memory rather than writing to disk, or for
>> when call-sites use a non-unicode ``output_encoding`` setting. 

> IMV, the extended ``publish_string()`` providing

>    publish_string(..., auto_encode=False) --> OutString
>    publish_string(..., auto_encode=True) --> bytes
    
> with `OutString` beeing 100% compatible with `str` and easily convertible
> to bytes via ``bytes(result)`` can cater for such needs. 

> (Also, is should not be too surprising that `publish_string` returns a
> `bytes` instance if the user tells it to "auto_encode".)

Fair enough, though I suppose I had seen ``auto_encode`` as part of
moving to a position where (eventually) ``publish_string`` always
returns an instance of Python's ``str`` class, rather than the current
overloaded return type. In the scenario I was imagining, users of
Docutils who know that they want ``bytes`` output could change to using
``publish_bytes`` in Docutils 0.20, and not need to worry about the
future of ``publish_string`` or the ``auto_encode`` argument. (There
is also an argument for self-documenting code, in that if you know that
you want ``bytes``, using the function with ``bytes`` in the name helps
a non-expert reader to understand what is going on.)

>> If it is to be removed, perhaps we could provide a recipie in the
>> documentation for how to manage publishing to an in-memory byte
>> sequence.

> This is part of the `publish_string()` docstring in my patch::

>    If `auto_encode` is True, the output is encoded according to the
>    `output_encoding`_ setting; the return value is a `bytes` instance
>    (unless `output_encoding`_ is "unicode",
>    cf. `docutils.io.StringOutput.write()`).

Sorry, I had forgotten about this part of your patch.

-----------

If you have time, perhaps you could commit your patch (or the latest
version thereof), and we could make progress from there? As far as I
can tell, the only unresolved point at the moment ahead of releasing
Docutils 0.20 is the future of ``publish_bytes``.


Thanks,
Adam

[Docutils-develop] Update tests for Pygments 2.14

[Adam T. <aat...@ou...>]

Dear all,

Pygments 2.14, released in January__, contains changes__ in the Python
lexer which break ``test_code``​​, ``test_code_long``, and ``test_include``.

__ https://pygments.org/docs/changelog/#version-2-14-0
__ https://github.com/pygments/pygments/commit/147b22face65617514ccfa8512c6b097b07cad4c

I have attached a sample patch to fix these and maintain support in the
tests for Pygments 2.13 and earlier. A different approach would be to
declare that the tests only support the latest version of Pygments (it
is currently unclear which version of Pygments is our minimum supported).

Thanks,
Adam


-------------


From 8ad1babfdab6b3ba715a1aa8712e2044740fd2ba Mon Sep 17 00:00:00 2001
From: Adam Turner <908...@us...>
Date: Thu, 6 Apr 2023 17:24:40 +0100
Subject: [PATCH] Support Pygments 2.14 and newer in tests

---
 .../test_rst/test_directives/test_code.py     |  62 +++++++
 .../test_directives/test_code_long.py         |  62 +++++++
 .../test_rst/test_directives/test_include.py  | 154 ++++++++++++++++++
 3 files changed, 278 insertions(+)

diff --git a/docutils/test/test_parsers/test_rst/test_directives/test_code.py b/docutils/test/test_parsers/test_rst/test_directives/test_code.py
index 81efaaefa..7985d1b65 100755
--- a/docutils/test/test_parsers/test_rst/test_directives/test_code.py
+++ b/docutils/test/test_parsers/test_rst/test_directives/test_code.py
@@ -22,6 +22,14 @@
 from docutils.utils import new_document
 from docutils.utils.code_analyzer import with_pygments
 
+try:
+    from pygments import __version__ as _pygments_ver
+except ImportError:
+    _pygments_ver = ''
+    PYGMENTS_2_14_PLUS = False
+else:
+    PYGMENTS_2_14_PLUS = tuple(map(int, _pygments_ver.split('.'))) >= (2, 14)
+
 
 class ParserTestCase(unittest.TestCase):
     def test_parser(self):
@@ -153,6 +161,60 @@ def my_function():
       print(8/2)
 """,
 """\
+<document source="test data">
+    <literal_block classes="code python3 testclass" ids="my-function" names="my_function" xml:space="preserve">
+        <inline classes="ln">
+             7 \n\
+        <inline classes="keyword">
+            def
+         \n\
+        <inline classes="name function">
+            my_function
+        <inline classes="punctuation">
+            ():
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+             8 \n\
+        <inline classes="whitespace">
+                \n\
+        <inline classes="literal string doc">
+            \'\'\'Test the lexer.
+        <inline classes="ln">
+             9 \n\
+        <inline classes="literal string doc">
+                \'\'\'
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            10 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            11 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="comment single">
+            # and now for something completely different
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            12 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="name builtin">
+            print
+        <inline classes="punctuation">
+            (
+        <inline classes="literal number integer">
+            8
+        <inline classes="operator">
+            /
+        <inline classes="literal number integer">
+            2
+        <inline classes="punctuation">
+            )
+""" if PYGMENTS_2_14_PLUS else """\
 <document source="test data">
     <literal_block classes="code python3 testclass" ids="my-function" names="my_function" xml:space="preserve">
         <inline classes="ln">
diff --git a/docutils/test/test_parsers/test_rst/test_directives/test_code_long.py b/docutils/test/test_parsers/test_rst/test_directives/test_code_long.py
index 07f1b9f36..641a5f1d9 100755
--- a/docutils/test/test_parsers/test_rst/test_directives/test_code_long.py
+++ b/docutils/test/test_parsers/test_rst/test_directives/test_code_long.py
@@ -22,6 +22,14 @@
 from docutils.utils import new_document
 from docutils.utils.code_analyzer import with_pygments
 
+try:
+    from pygments import __version__ as _pygments_ver
+except ImportError:
+    _pygments_ver = ''
+    PYGMENTS_2_14_PLUS = False
+else:
+    PYGMENTS_2_14_PLUS = tuple(map(int, _pygments_ver.split('.'))) >= (2, 14)
+
 
 @unittest.skipUnless(with_pygments, 'needs Pygments')
 class ParserTestCase(unittest.TestCase):
@@ -54,6 +62,60 @@ def my_function():
       print(8/2)
 """,
 """\
+<document source="test data">
+    <literal_block classes="code python3" xml:space="preserve">
+        <inline classes="ln">
+             7 \n\
+        <inline classes="keyword">
+            def
+         \n\
+        <inline classes="name function">
+            my_function
+        <inline classes="punctuation">
+            ():
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+             8 \n\
+        <inline classes="whitespace">
+                \n\
+        <inline classes="literal string doc">
+            \'\'\'Test the lexer.
+        <inline classes="ln">
+             9 \n\
+        <inline classes="literal string doc">
+                \'\'\'
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            10 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            11 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="comment single">
+            # and now for something completely different
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            12 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="name builtin">
+            print
+        <inline classes="punctuation">
+            (
+        <inline classes="literal number integer">
+            8
+        <inline classes="operator">
+            /
+        <inline classes="literal number integer">
+            2
+        <inline classes="punctuation">
+            )
+""" if PYGMENTS_2_14_PLUS else """\
 <document source="test data">
     <literal_block classes="code python3" xml:space="preserve">
         <inline classes="ln">
diff --git a/docutils/test/test_parsers/test_rst/test_directives/test_include.py b/docutils/test/test_parsers/test_rst/test_directives/test_include.py
index d5db404c5..0667e0e55 100755
--- a/docutils/test/test_parsers/test_rst/test_directives/test_include.py
+++ b/docutils/test/test_parsers/test_rst/test_directives/test_include.py
@@ -23,6 +23,14 @@
 from docutils.utils import new_document
 from docutils.utils.code_analyzer import with_pygments
 
+try:
+    from pygments import __version__ as _pygments_ver
+except ImportError:
+    _pygments_ver = ''
+    PYGMENTS_2_14_PLUS = False
+else:
+    PYGMENTS_2_14_PLUS = tuple(map(int, _pygments_ver.split('.'))) >= (2, 14)
+
 TEST_ROOT = Path(__file__).resolve().parents[3]
 
 
@@ -1097,6 +1105,23 @@ def mydir(path):
    :code: rst
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        Included code
+    <literal_block classes="code rst" source="{include1}" xml:space="preserve">
+        <inline classes="generic heading">
+            Inclusion 1
+        \n\
+        <inline classes="generic heading">
+            -----------
+        \n\
+        <inline classes="whitespace">
+            \n\
+        This file is used by \n\
+        <inline classes="literal string">
+            ``test_include.py``
+        .
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         Included code
@@ -1121,6 +1146,32 @@ def mydir(path):
    :number-lines:
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        Included code
+    <literal_block classes="code rst" source="{include1}" xml:space="preserve">
+        <inline classes="ln">
+            1 \n\
+        <inline classes="generic heading">
+            Inclusion 1
+        \n\
+        <inline classes="ln">
+            2 \n\
+        <inline classes="generic heading">
+            -----------
+        \n\
+        <inline classes="ln">
+            3 \n\
+        <inline classes="whitespace">
+            \n\
+        <inline classes="ln">
+            4 \n\
+        <inline classes="whitespace">
+        This file is used by \n\
+        <inline classes="literal string">
+            ``test_include.py``
+        .
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         Included code
@@ -1152,6 +1203,34 @@ def mydir(path):
    :code: rst
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        TAB expansion with included code:
+    <literal_block classes="code rst" source="{include_literal}" xml:space="preserve">
+        Literal included this should \n\
+        <inline classes="generic strong">
+            **not**
+         be \n\
+        <inline classes="generic emph">
+            *marked*
+         \n\
+        <inline classes="name variable">
+            `up`
+        .
+        <inline classes="whitespace">
+            \n\
+                <- leading raw tab.
+        <inline classes="whitespace">
+            \n\
+            \n\
+        Newlines
+        <inline classes="whitespace">
+            \n\
+        are
+        <inline classes="whitespace">
+            \n\
+        normalized.
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         TAB expansion with included code:
@@ -1180,6 +1259,34 @@ def mydir(path):
    :tab-width: 2
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        Custom TAB expansion with included code:
+    <literal_block classes="code rst" source="{include_literal}" xml:space="preserve">
+        Literal included this should \n\
+        <inline classes="generic strong">
+            **not**
+         be \n\
+        <inline classes="generic emph">
+            *marked*
+         \n\
+        <inline classes="name variable">
+            `up`
+        .
+        <inline classes="whitespace">
+            \n\
+          <- leading raw tab.
+        <inline classes="whitespace">
+            \n\
+            \n\
+        Newlines
+        <inline classes="whitespace">
+            \n\
+        are
+        <inline classes="whitespace">
+            \n\
+        normalized.
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         Custom TAB expansion with included code:
@@ -1208,6 +1315,34 @@ def mydir(path):
    :tab-width: -1
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        Custom TAB expansion with included code:
+    <literal_block classes="code rst" source="{include_literal}" xml:space="preserve">
+        Literal included this should \n\
+        <inline classes="generic strong">
+            **not**
+         be \n\
+        <inline classes="generic emph">
+            *marked*
+         \n\
+        <inline classes="name variable">
+            `up`
+        .
+        <inline classes="whitespace">
+            \n\
+        \t<- leading raw tab.
+        <inline classes="whitespace">
+            \n\
+            \n\
+        Newlines
+        <inline classes="whitespace">
+            \n\
+        are
+        <inline classes="whitespace">
+            \n\
+        normalized.
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         Custom TAB expansion with included code:
@@ -1234,6 +1369,25 @@ def mydir(path):
 .. include:: {include14}
 """,
 f"""\
+<document source="test data">
+    <paragraph>
+        Including includes/include14.txt
+    <paragraph>
+        Including more/include6.txt as rst-code from includes/include14.txt:
+    <literal_block classes="code rst" source="{include6}" xml:space="preserve">
+        In includes/more/include6.txt
+        <inline classes="whitespace">
+            \n\
+            \n\
+        <inline classes="punctuation">
+            ..
+         \n\
+        <inline classes="operator word">
+            include
+        <inline classes="punctuation">
+            ::
+         ../sibling/include7.txt
+""" if PYGMENTS_2_14_PLUS else f"""\
 <document source="test data">
     <paragraph>
         Including includes/include14.txt
-- 
2.40.0.windows.1

Re: [Docutils-develop] Release 0.20

[Guenter M. <mi...@us...>]

Dear Adam,

On 2023-04-06, Adam Turner wrote:

...

> ---------------------------------------------------------------------

>>> I think we should keep the ``publish_bytes()`` function in either case.

>> I don't see a convincing use case for ``publish_bytes()`` and would
>> prefer to keep the "core" interface as small as sensible.

Rationale: 

``publish_bytes()`` makes sense alongside ``publish_str()``.

However, ``publish_str()`` and the existing ``publish_string()`` are so
close that confusion is to be expected.


> The main use(s) here would be for publishing binary formats (e.g. ODT)
> to a ``bytes`` object in memory rather than writing to disk, or for
> when call-sites use a non-unicode ``output_encoding`` setting. 

IMV, the extended ``publish_string()`` providing

    publish_string(..., auto_encode=False) --> OutString
    publish_string(..., auto_encode=True) --> bytes
    
with `OutString` beeing 100% compatible with `str` and easily convertible
to bytes via ``bytes(result)`` can cater for such needs. 

(Also, is should not be too surprising that `publish_string` returns a
`bytes` instance if the user tells it to "auto_encode".)

> If it is to be removed, perhaps we could provide a recipie in the
> documentation for how to manage publishing to an in-memory byte
> sequence.

This is part of the `publish_string()` docstring in my patch::

    If `auto_encode` is True, the output is encoded according to the
    `output_encoding`_ setting; the return value is a `bytes` instance
    (unless `output_encoding`_ is "unicode",
    cf. `docutils.io.StringOutput.write()`).

Thanks,
Günter

Re: [Docutils-develop] Release 0.20

[Adam T. <aat...@ou...>]

Dear Günter,

> I'd like to keep support for the default Python version of Debian/stable
> (even if this is longer than 42 months old).

Fair enough!

---------------------------------------------------------------------


>> I am content to go with option (c), but I would want to simultaneously announce
>> the version where the default would change to ``auto_encode=False`` and the
>> version where ``publish_string()`` would only support returning ``str``
>> instances.

> I suggest to do the default switch 2 versions after announcement, i.e. 0.22
> or 1.0 (announce as 0.22 or later).
> I would not announce removal of the option now (shold not be earlier than
> 3.0, maybe never).

Ok, this sounds good -- introduce the ``auto_encode`` argument now,
defaulting to ``True``, and announce that it will switch to ``False`` in
Docutils 0.22 or later. 

---------------------------------------------------------------------

>> I think that it might be possible to implement the new ``auto_encode``
>> parameter without a custom ``str`` subclass,

> The problem with exporting an output document as `str` instance
> (auto_encode=False) is that

> * The output document by default contains an "encoding indicator" (at least
>   in HTML and LaTeX) which is determined by the "output_encoding" setting
>   (depending on a set of configuration files or command line input which may
>   be "programatically overwritten").
  
> * A `str` instance has no meta-data storing the "intended encoding".
> 
>   The application calling `publish_string()` would have to re-enact the
>   configuration parsing or to grep in the string to find out the
>   right encoding.

> The old approach is to use publish_parts() and from the returned dictionary
> use the "document" and "encoding" items.

> I am open for other suggestions to solve this problem.  

I think here we should say "practicality beats purity" and go with the subclass,
viewing it as a transitional measure. You make good arguments!

---------------------------------------------------------------------

>> I think we should keep the ``publish_bytes()`` function in either case.

> I don't see a convincing use case for ``publish_bytes()`` and would
> prefer to keep the "core" interface as small as sensible.

The main use(s) here would be for publishing binary formats (e.g. ODT) to a ``bytes``
object in memory rather than writing to disk, or for when call-sites use a non-unicode
``output_encoding`` setting. If it is to be removed, perhaps we could provide a recipie
in the documentation for how to manage publishing to an in-memory byte sequence.

Thanks,
Adam

Re: [Docutils-develop] Release 0.20

[Guenter M. <mi...@us...>]

Dear Adam,

I am glad to hear from you again.

On 2023-03-24, Adam Turner wrote:

...

> For interest, Sphinx has a policy__ to support:
> "all minor versions of Python released in the past 42 months from the ...
> release date with a minimum of 3 minor versions of Python"

> __ https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy

Thank you for the pointer. 
I'd like to keep support for the default Python version of Debian/stable
(even if this is longer than 42 months old).


>>> * Future of ``core.publish_string()`` API function:

>>>   a) Keep current behaviour ::

>>>       def publish_string(source: Union[bytes, str],
>>>                          [...]
>>>                          enable_exit_status=False) -> Union[bytes, str]

>>>      as "wart", just improve documentation?

>>>   b) Deprecate ``publish_string()`` and provide new ``publish_str()``
>>>      and ``publish_bytes()`` functions?

>>>   c) Return a sub-class of ``str`` with ``__bytes__()`` method that
>>>      encodes with ``encoding`` and ``encoding_errors`` set to the
>>>      "output_encoding" and "output_encoding_errors" setting values?

>>>      - as "subtly" changed behaviour, or
>>>      - with a new function replacing ``publish_string()``
>>>        (find a good name!)?

>>>   Proposal [GM]
>>>     - explore c)
>>>     - remove ``core.publish_bytes()`` before releasing 0.20.

>> I prepared a patch for option c) with a new function attribute
>> 'auto_encode for `core.publish_string()`. This would allow to keep the
>> name and switch to a behaviour matching it (returning a string, not
>> bytes) gradually (by switching the default value and eventually
>> removing the option later). See below.

> I am content to go with option (c), but I would want to simultaneously announce
> the version where the default would change to ``auto_encode=False`` and the
> version where ``publish_string()`` would only support returning ``str``
> instances.

I suggest to do the default switch 2 versions after announcement, i.e. 0.22
or 1.0 (announce as 0.22 or later).
I would not announce removal of the option now (shold not be earlier than
3.0, maybe never).

> I think that it might be possible to implement the new ``auto_encode``
> parameter without a custom ``str`` subclass,

The problem with exporting an output document as `str` instance
(auto_encode=False) is that

* The output document by default contains an "encoding indicator" (at least
  in HTML and LaTeX) which is determined by the "output_encoding" setting
  (depending on a set of configuration files or command line input which may
  be "programatically overwritten").
  
* A `str` instance has no meta-data storing the "intended encoding".

  The application calling `publish_string()` would have to re-enact the
  configuration parsing or to grep in the string to find out the
  right encoding.

The old approach is to use publish_parts() and from the returned dictionary
use the "document" and "encoding" items.

I am open for other suggestions to solve this problem.  

> I think we should keep the ``publish_bytes()`` function in either case.

I don't see a convincing use case for ``publish_bytes()`` and would
prefer to keep the "core" interface as small as sensible.

Thanks,
Günter

[Docutils-develop] [docutils:feature-requests] #93 Mod operator for math2html

[Günter M. <mi...@us...>]

The *LaTeX math to html/css* converter is contributed code without an active maintainer, unfortunately.
The `\mod` command is was added to the *LaTeX math to MathML* converter in Nov 2022 [r9213].
I recommend using "--math-output=MathML" (or set [math-output](https://docutils.sourceforge.io/docs/user/config.html#math-output) in a configuration file. 
It works nice with Firefox and since January 23 also in Crome.



---

** [feature-requests:#93] Mod operator for math2html**

**Status:** open
**Group:** Default
**Created:** Sat Sep 03, 2022 08:43 PM UTC by Daniel James Perry
**Last Updated:** Mon Oct 31, 2022 12:10 PM UTC
**Owner:** nobody


The latex to html/css code is quite impressive, but it's missing the `\\mod` command (as well as other similar commands like `\\pmod`. It should be a trivial thing to do.

BTW please get with the times and start using git PLEASE.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #466 Should docutils whine about nesting simple body elements?

[Günter M. <mi...@us...>]

Nesting paragraphs in paragraphs is indeed a violation of the Docutils document model [docutils.dtd](https://docutils.sourceforge.io/docs/ref/docutils.dtd).
Unfortunately, this is just one example of an "infinite" number of ways you can build invalid documents -- testing for all sorts of violations by custom code may seriously affect performance.

You may consider exporting a test document to XML and test for validity against the docutils.dtd.


---

** [bugs:#466] Should docutils whine about nesting simple body elements?**

**Status:** open
**Created:** Thu Feb 09, 2023 09:55 AM UTC by Julien Palard
**Last Updated:** Thu Feb 09, 2023 09:55 AM UTC
**Owner:** nobody


Context:

[This](https://github.com/python/cpython/blob/45fa12aec8f508c224a1521cfe3ae597f1026264/Doc/tools/extensions/pyspecific.py#L149) call to `nested_parse` adds a `Paragraph` node to a `Paragraph` node,  this look forbidden by [the doc](https://docutils.sourceforge.io/docs/ref/doctree.html#body-elements), and it also look to misbehave sphinx-side, when translating the document, where we completly loose the nested paragraph once translated.

It renders properly though when **not** translated.

The linked `nested_parse` returns:

```xml
<paragraph classes="availability">
    <pending_xref refdoc="index" refdomain="std" refexplicit="True" reftarget="availability" reftype="ref" refwarn="True">
        <inline classes="xref std std-ref">
            Availability
    : 
    Unix, not Emscripten, not WASI.
    <paragraph>
        Hello world I'm the content of the "availability" directive.
```

If it's really something that should not be done, maybe some kind of warning at build time could help debugging those issues?



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.