docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

Unfortunately, I don't know what the "test failure on Windows" actually is, could you provide the relevant test output, please?

> Is it possible to use the simpler unquote() here? 

If it works on Windows, using unquote() is a good idea. Could you test with
~~~
diff --git a/docutils/docutils/writers/_html_base.py b/docutils/docutils/writers/_html_base.py
index ead930e31..2eab36510 100644
--- a/docutils/docutils/writers/_html_base.py
+++ b/docutils/docutils/writers/_html_base.py
@@ -626,7 +626,7 @@ def uri2imagepath(self, uri):
         uri_parts = urllib.parse.urlparse(uri)
         if uri_parts.scheme not in ('', 'file'):
             raise ValueError('Can only read local images.')
-        imagepath = urllib.request.url2pathname(uri_parts.path)
+        imagepath = urllib.parse.unquote(uri_parts.path)
         if imagepath.startswith('/'):  # cf. config.html#root-prefix
             root_prefix = Path(self.settings.root_prefix)
             imagepath = (root_prefix/imagepath[1:]).as_posix()
~~~
and adapt the test script if required?

> Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

Per definitionem, there is no way to provide a system file-path as image ressource, both [rST](https://docutils.sourceforge.io/docs/ref/rst/directives.html#image) and the [Docutils Doctree](https://docutils.sourceforge.io/docs/ref/doctree.html#image)  explicitely expect an URI.

As URIs are capable of representing local file paths, I dont see the advantage.
Interpreting an image URI with undefined scheme as system path would be a complication of both, specification and implementation and a backwards-incompatible change. We would als have to think about images that are only read to get the size but kept external in HTML output.





---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 02:25 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #492 Git repo lacking tags

[Adam T. <aa-...@us...>]

> a new release is due once remaining issues are settled

At an appropriate time, I would be happy to help make the release, if useful -- an effort to increase our [bus factor](https://en.wikipedia.org/wiki/Bus_factor) perhaps! @grubert has written a very comprehensive set of notes in  `release.txt` & `releasing-log.txt` to follow!

A


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Wed Aug 07, 2024 01:45 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 02:25 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[Adam T. <aa-...@us...>]

> Rather, we shall move the upstream SVN repo to a Git version and pay attention to correctly convert tags.

Over the last week or so I have made a series of improvements  the reposurgeon conversion process [r9856] -- I now consider this "done". Next to look at is migrating & preserving history for the issue tracker.

A


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Tue Aug 06, 2024 08:17 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[Günter M. <mi...@us...>]

Thank you for the extensive analysis.
It seems some non-standard setup or handling of either the upstream SVN repo or the git-svn version provided in repo.or.cz is causing errors. (In my repo.or.cz checkout, all tags are broken, too.)

Caveat: the repository in repo.or.cz is a 3rd party contribution, so we cannot change things there.
Rather, we shall move the upstream SVN repo to a Git version and pay attention to correctly convert tags.


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Tue Aug 06, 2024 07:54 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

Sorry for one more, but I went too fast.  No there is a difference between the git commit messages. The first one (the one tagged by docutils-0.21.-2) contains
~~~
git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@9649 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~
whereas the one in master branch (a direct ancestor to master tip) contains
~~~
git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk@9649 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~

i.e. former has `code/trunk/docutils@9649` and latter has `code/trunk@9649`.


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Tue Aug 06, 2024 07:46 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

The parent of `docutils-0.21.2` refers to the same revision as a more direct ancestor in master branch:
~~~
docutils$ git log -1 docutils-0.21.2~
commit b43eb031ecb2de7d2d1a5e4d160ebe15a0f9a7f0
Author: grubert <grubert@929543f6-e4f2-0310-98a6-ba3bd3dd1d04>
Date:   Tue Apr 23 18:54:26 2024 +0000

    Version 0.21.2
    
    
    git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk/docutils@9649 929543f6-e4f2-0310-98a6-ba3bd3dd1d04

$ git log -1 323957acb
commit 323957acb17369d38bd31c81dfe543ae46ae9113
Author: grubert <grubert@929543f6-e4f2-0310-98a6-ba3bd3dd1d04>
Date:   Tue Apr 23 18:54:26 2024 +0000

    Version 0.21.2
    
    
    git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk@9649 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~

These are the two same commits but they have distinct SHAs and in master branch I see the second one not the first one.  The git-svn-id however always refers to ba3bd3dd1d04 i.e. to the actually tagged commits (the parent of the annotated tag itself).

All I am lacking is the branch where this ba3bd3dd1d04 belongs, because it is not in master branch.


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Tue Aug 06, 2024 07:39 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

I git cloned again with https://repo.or.cz/docutils.git not git protocol and got (as expected) the same result.

Notice that ab29cf78ae24a9  is indeed the doctuils-0.21.2 tag but it is not the commit or one of the commits in recent history referring to 0.21.2 release.  

~~~
$ git log --oneline --grep 0.21.2
0353066fb Fix headers for release 0.21.2
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
6729e38e3 version 0.21.2b.dev
~~~

It is as if all `docutils-...` tags refer to some other branch, and the origin remote does contain many branches other than master but none has a name which seems encouraging to me to try out.  The 0.21.2 release seems to about 200 commits back in history, not 9344 commits.  So we can't use `docutils-...` prefixed tags for bisecting. 

If I  count commits separating docutils-0.21.2 from master HEAD:
~~~
$ git log --oneline docutils-0.21.2...master | wc -l
   15385
~~~
It looks as if all these tags are on a separate "release" branch which I do not see ats available at my check out.

When I diff between master HEAD and such tags, I see in particular that there is one less layer of `docutils/` nesting.  Maybe there is some business of a git sub-module (but that exceeds greatly my git knowledge).  In brief the tags seem to belong to some other `.git` structure but somehow are visible from the public svn-to-git mirror repo.



---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Mon Aug 05, 2024 09:22 PM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[Günter M. <mi...@us...>]

I don't know why you get a different result, however, on https://repo.or.cz/docutils.git/tags the tag for 0.21.2 seems to be correctly set:
> commit	ab29cf78ae24a9970d9884de40926dceb855bd32
[...] Tue, 23 Apr 2024 20:55:21 +0200 (23 18:55 +0000)


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Mon Aug 05, 2024 04:28 PM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #87 [FR] Giving type annotations to docutils

[Günter M. <mi...@us...>]

Thanks for the first commits with type hints.

We should, however, mention, that using the type hints (with e.g. mypy) requires Python 3.10 or newer. With 3.9 and mypy 0.812, I get, e.g., for `mypy docutils/core.py` the result:

      Found 392 errors in 14 files (checked 1 source file)

Also,  the boilerplate (TYPE_CHECKING blocks) are quite large. How about a dedicated module for Docutils-specific type definitions? 
We may also define shortcuts/aliases for often used Unions like `str | os.PathLike[str]` which may offer a better intelligible way of representing
`pathdict: dict[str, list[str | os.PathLike[str]] | str | os.PathLike[str]]`, say.


---

**[feature-requests:#87] [FR] Giving type annotations to docutils**

**Status:** open
**Group:** Default
**Created:** Mon Jan 03, 2022 03:49 AM UTC by Adam  Turner
**Last Updated:** Mon Jan 03, 2022 12:39 PM UTC
**Owner:** nobody


*Sorry for reposting this as a feature request, I couldn't work out how to respond to Günter's post [1] through Sourceforge's web interface* 

[1]: https://sourceforge.net/p/docutils/mailman/message/37410168/

> > Hi all,
> 
> > I noticed py2 and py36- support will be dropped in the next release.
> 
> Yes, I am currently working on this, ... soon in the repository.

Günter @milde -- happy to help with this if you'd like?

> > It helps to integrate type annotations in docutils-stub package into
> > upstream.
> > I believe it helps to develop third-party extensions (including Sphinx).
> > Is there any chance to merge it into the docutils?
> 
> In my view, it would make sense to have annotations right in the "master".
> 
> This could also be used as an indicator for the public API: 
> 
> In the Docutils Backwards Compatibility Policy, we could change the
> clause about the "scope of the public API" to an "opt-in" scheme: only
> function/class interfaces with annotations belong to the stable, public
> API while not annotated function interfaces may be changed or removed in
> future versions if this helps to improve or clean up the code base.
> 
> What do other Docutils developers/contributors/users think?

In my experienece, type annotations are often most useful in private / internal code -- the public API is often well specified, and used often enough that you remember the types. Internal methods may be touched less frequently, or the code paths from caller to callee may be less clear, so type hints as a self documenting contract are really useful.

I also think that it would be useful to be explicit about the public API in docs/dev/policies.txt -- currently the scope clause links to CPython's backwards compatability rules -- I think that explicit is better than implicit here, in my opinion.

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

Hello, but docutils-0.21.2 is not resolving to f344d4daf but to ab29cf78ae24

~~~
docutils$ git log --oneline f344d4daf..HEAD | wc -l
     201
docutils$ git log --oneline docutils-0.21.2..HEAD | wc -l
    9344
~~~

??

~~~
git log -1 f344d4daf
commit f344d4dafcd68cc9c315a4700b5f0efb5dc2e9e7
Author: grubert <grubert@929543f6-e4f2-0310-98a6-ba3bd3dd1d04>
Date:   Tue Apr 23 19:10:06 2024 +0000

    releasing 0.21.2
    
    
    git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk@9651 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Mon Aug 05, 2024 12:16 PM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[Günter M. <mi...@us...>]

Docutils 0.21.2 is indeed the last release.
And yes, there are a lot of commits after this - a new release is due once remaining issues are settled.


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Mon Aug 05, 2024 09:47 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Mon Aug 05, 2024 09:47 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] manpage option "use_reference_macros"

[Guenter M. <mi...@us...>]

Dear Engelbert,

is the manpage writer option "use_reference_macros" intended to stay?

If yes, it would need an entry in docs/user/config.txt.

Also, I suggest adding a command line option to reset the default
in case a config file sets it to True.

Suggestion:
  rename "use_reference_macros" to "macro_references" and use the pair

  --macro-references
  --text-references
   
See, e.g., "--link-stylesheet" / "--embed-stylesheet" in latex2e/__init__.py
for an example.

Günter

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Adam T. <aa-...@us...>]

> In my opinion, the project should stop honoring the "preferred encoding" and instead expect UTF-8 unless otherwise specified, as that's going to become the default behavior in Python 3.14 for most IO operations.

I agree, however...

> I'm unsure of compatibility implications.

There was fairly extensive discussion of this in April last year. The core issue is that Docutils serialises to formats that have internal charset/encoding declarations (e.g. TeX, HTML, XML). If everything is UTF-8 then all is fine and simple, but if the user wants e.g. latin1 encoding then Docutils 'should' encode that in the relevant places in the output documents. Docutils also chooses whether to embed a Unicode character directly vs using an escape or macro (e.g. the dagger † footnote symbol) based on the chosen encoding.

I am of the opinion that Docutils should remove support for encodings other than Unicode (UTF-8) in text mode for both input and output. UTF-8 is so ubiquitous that anyone running a modern enough Python to use this version of Docutils will either support UTF-8 or know how to work-around any problems.

The only writer that makes runtime use of the output encoding setting is LaTeX. LuaTex and XeTeX have always supported UTF-8 in source files, and LaTeX has [since 2018](https://tug.org/TUGboat/tb39-1/tb121ltnews28.pdf). 

----

To your original point, running with ``PYTHONWARNDEFAULTENCODING=1`` should now produce no warnings. If you still get warnings please let us know as it would mean we are missing test coverage (Docutils' tests [pass with -Werror and -Xwarn_default_encoding](https://github.com/docutils/docutils/actions/runs/10204589804/job/28233427232)).

A



---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Thu Aug 01, 2024 04:14 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Jason R. C. <ja...@us...>]

> I'm not sure what the right behaviour here should be.

In my opinion, the project should stop honoring the "preferred encoding" and instead expect UTF-8 unless otherwise specified, as that's going to become the default behavior in Python 3.14 for most IO operations. I'm unsure of compatibility implications. It does appear as if this test (`test_fallback_no_utf8`) would no longer be relevant in that regime, so I'd just delete it.

Regarding the non-UTF8 mode, that does sound more complicated, although maybe that functionality too should be deprecated/removed. That is, IMHO, the user should be offered UTF-8 mode by default and an option to specify an encoding, maybe with "locale" as one option, but otherwise remove the implied "locale" behavior.

I have a very weak understanding of docutils, however, so take my advice with a grain of salt.


---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Thu Aug 01, 2024 02:57 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Adam T. <aa-...@us...>]

I tested reverting and re-applying [r9772] in [this PR](https://github.com/AA-Turner/docutils/pull/16) -- note that the 'alltest' failure occurs in the before and after, but pytest and unittest fail when [r9772] is reapplied.

A


---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Thu Aug 01, 2024 02:55 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Adam T. <aa-...@us...>]

[r9772] breaks tests for non-UTF locales on both Linux and Windows (e.g. ISO 88591), when not using Python's UTF-8 mode.

See the following failures (from [GitHub Actions](https://github.com/AA-Turner/docutils/actions/runs/10200475399/job/28220043798?pr=16), scroll up to the first section `Run test suite (pytest ./test)`):

```
_____________________ FileInputTests.test_fallback_no_utf8 _____________________

self = <test.test_io.FileInputTests testMethod=test_fallback_no_utf8>

    @unittest.skipIf(preferredencoding in (None, 'ascii', 'utf-8'),
                     'locale encoding not set or UTF-8')
    def test_fallback_no_utf8(self):
        # If  no encoding is given and decoding with 'utf-8' fails,
        # use the locale's preferred encoding (if not None).
        # Provisional: the default will become 'utf-8'
        # (without auto-detection and fallback) in Docutils 0.22.
        source = du_io.FileInput(
            source_path=os.path.join(DATA_ROOT, 'latin1.txt'))
>       data = source.read()

test/test_io.py:321: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
docutils/io.py:412: in read
    data = self.source.read()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <encodings.utf_8.IncrementalDecoder object at 0x7f023f9906f0>
input = b'Gr\xfc\xdfe\n', final = True

>   ???
E   UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 2: invalid start byte

<frozen codecs>:322: UnicodeDecodeError
```

I'm not sure what the right behaviour here should be.

------

There's also a problem on the same non-UTF-8 locales when not in UTF-8 mode:

```
======================================================================
ERROR: test_publish_cmdline (test_publisher.ConvenienceFunctionTests.test_publish_cmdline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/docutils/docutils/docutils/docutils/io.py", line 525, in write
    self.destination.write(data)
  File "/home/runner/work/docutils/docutils/docutils/test/alltests.py", line 63, in write
    self.stream.write(string)
TypeError: write() argument must be str, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/docutils/docutils/docutils/docutils/io.py", line 529, in write
    self.destination.buffer.write(data)
    ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Tee' object has no attribute 'buffer'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/docutils/docutils/docutils/test/test_publisher.py", line 160, in test_publish_cmdline
    core.publish_cmdline(writer_name='null',
  File "/home/runner/work/docutils/docutils/docutils/docutils/core.py", line 431, in publish_cmdline
    output = publisher.publish(
             ^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/docutils/docutils/docutils/docutils/core.py", line 261, in publish
    output = self.writer.write(self.document, self.destination)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/docutils/docutils/docutils/docutils/writers/__init__.py", line 81, in write
    return self.destination.write(self.output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/docutils/docutils/docutils/docutils/io.py", line 533, in write
    raise ValueError(
ValueError: Encoding of <file> (iso8859-1) differs 
  from specified encoding (utf-8)

----------------------------------------------------------------------
Ran 1872 tests in 4.888s

FAILED (errors=1, skipped=2)
Elapsed time: 5.079 seconds
```

(On Windows it says ``ValueError: Encoding of <file> (cp1252) differs `` instead). 

This failure only happens with ``alltests.py``. Now that both ``pytest`` and ``unittest`` work with our test suite, we could consider removing  ``alltests.py``.

A



---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Mon Jul 22, 2024 11:55 AM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #489 Issues with the Docutils Doctree

[Günter M. <mi...@us...>]

Commit [r9780] announces the change to mandatory labels for footnote and label in Docutils 1.0.
Commit [r9781] implements a rST parser warning for empty footnotes and labels.

TODO: allow figures without caption/legend?


---

**[bugs:#489] Issues with the Docutils Doctree**

**Status:** open
**Created:** Tue Jun 11, 2024 12:13 PM UTC by Günter Milde
**Last Updated:** Tue Jun 11, 2024 12:13 PM UTC
**Owner:** nobody


There are a number of discrepancies between the [Docutils Generic DTD][docutils.dtd]
and the implementation in Docutils code.

footnote
--------

[docutils.dtd][docutils.dtd]: `(label?, (%body.elements;)+)`
→ Label **optional**, content **required**; (not changed since 2002-04-20).

[reStructureText Markup Specification][rST spec]:
> Each footnote consists of an explicit markup start (".. "), a left
> square bracket, **the footnote label**, a right square bracket, and
> whitespace, followed by indented body elements.

The `footnote` class in [docutils.nodes](https://docutils.sourceforge.io/docutils/nodes.py) is a subclass of `Labeled`, 
whose docstring says: "**Contains a label** as its first element."

The rST parser **requires** a label but allows **empty footnotes**
(cf. `test/test_writers/test_latex2e.py`).

citation
--------

[docutils.dtd][docutils.dtd]: `(label, (%body.elements;)+)`
→ Label **required**, content **required**; (not changed since 2002-04-20).

[reStructureText Markup Specification][rST spec]:
> Citations are **identical to footnotes** except that they use only
> non-numeric labels …

The rST parser allows **empty citations** (cf. test_rst/test_citations.py).

figure
------

[docutils.dtd][docutils.dtd]: `(image, ((caption, legend?) | legend))`
→ caption or legend **required**; (not changed since 2002-04-20).

[reStructuredText Directives](https://docutils.sourceforge.io/docs/ref/rst/directives.html#figure)

> A "figure" consists of image data (including image options), an **optional
> caption** (a single paragraph), and an **optional legend** (arbitrary body
> elements). For page-based output media, figures might float to a different
> position if this helps the page layout.

The rST parser allows **figures without caption or legend**.

Docutils HTML and LaTeX writers use a different layout for figures vs.
images. They can handle figures without caption/legend.

Suggestion
----------

* Make *footnote label compulsory`*.
* Let rST report a *warning for empty footnote and citation*.
  Remove tests with empty footnote and citation.
* Allow *figures without caption/legend* in the Docutils Generic DTD.

(Change in Docutils 1.0, announce in 0.22.)

Questions
---------
Are there a use cases for 
* footnotes without label,
* footnotes without content,
* citations without content?

[docutils.dtd]: https://docutils.sourceforge.io/docs/ref/docutils.dtd
[rST spec]: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #491 Don't hardcode interpreter in utils/smartquotes.py

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed
- **Comment**:

Fixed in [r9778]. Thank you for reporting.



---

**[bugs:#491] Don't hardcode interpreter in utils/smartquotes.py**

**Status:** open-fixed
**Created:** Tue Jul 02, 2024 01:39 PM UTC by Ross Burton
**Last Updated:** Tue Jul 02, 2024 01:39 PM UTC
**Owner:** nobody


The interpreter is hardcoded in smartquotes.py:

https://sourceforge.net/p/docutils/code/HEAD/tree/trunk/docutils/docutils/utils/smartquotes.py

Instead of using "/usr/bin/python3" directly, it's recommended to use "/usr/bin/env python3".


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed
- **Comment**:

[r9772] changes the default encoding from ``None`` (auto-detect) to "utf-8" in `docutils.io.Input` and `docutils.io.FileInput`.



---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Sat Jun 29, 2024 09:09 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Günter M. <mi...@us...>]

Thank you for the feedback. The problem is worked on:

* In the repository version, the default encoding is "utf-8".
* In Docutils 1.0, the input encoding detection will be removed. This removal includes the fallback  "getpreferredencoding()" in line 151. 


---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Fri Jun 28, 2024 03:34 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Jason R. C. <ja...@us...>]

---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Fri Jun 28, 2024 03:34 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Waiting for feedback about issues with the Docutils Doctree [#489]

[Guenter M. <mi...@us...>]

Dear Alan, Karl, and Docutils developers,

On 2024-06-20, Karl O. Pinc wrote:
> On Thu, 20 Jun 2024 Alan <ala...@gm...> wrote:

Thanks for the feedback.

It seems there is an emerging consensus regarding footnotes and citations.

Changing the "document tree" description (docutils.dtd) will not prevent
the creation of footnotes/labels without label or content.
The only immediately visibly change for users would be that

   docutils --validate a-document-with-faulty-footnote.rst
   
would generate an "invalid element" warning. 
  Also, it would make clear that there is no longer a guarantee that these
constructs work with Docutils. (We may remove some test cases.)

Developers of writers or plug-ins would gain from a simpler and consistent
documentation. Their task becomes easier, as they do no longer have to make
sure the new component supports footnotes/citations without label or content.
(In the long term, this may also lead to missing support for these
constructs in the Docutils writers.)

>> - citations without content: I've never seen it
>> - footnotes without content or without "label": I've never seen it


Which leaves figures:

>> - figures without caption/legend: no "legend" is reasonably common,
>>   but no caption is extremely rare and is a need met by `image` (I
>>   believe)

> Figures with no caption would still show up in the TOCs list
> of figures.  (Or whereever that list shows up.) At least
> in docbook.  I think.  Although I don't know how useful that'd
> be since no description would be listed.  I suppose it'd have a figure
> number, which is not entirely useless.
...
> Anyway, the big difference between figures and images,
> in my mind, is that figures tend to appear in some sort
> of listing somewhere and images don't.

Also, figures tend to "flow" to a suitable position in page-based media.

Let's have a look at the current behaviour with various writers:

Input::

  .. image:: docs/user/images/default.png

  .. figure:: docs/user/images/default.png

HTML5 
  just uses a <figure> wrapper (like XML and pseudoXML)::

    <img alt="docs/user/images/default.png" src="docs/user/images/default.png" />
    <figure>
    <img alt="docs/user/images/default.png" src="docs/user/images/default.png" />
    </figure>

  The effect depends on CSS styling: a <figure> may float or get a figure
  number. With the default styles, a figure is set with wider margins on
  both sides. The figure number (cf. `numbered figures`__) is only shown
  if there is a caption but this is an CSS implementation detail.

  __ https://docutils.sourceforge.io/test/functional/expected/standalone_rst_html5.html#numbered-figures)
  
LaTeX
  wraps a figure in a figure float and centre-aligns it::
  
    \includegraphics{docs/user/images/default.png}

    \begin{figure}
    \noindent\makebox[\linewidth][c]{\includegraphics{docs/user/images/default.png}}\end{figure}

  As a result, a figure will usually move to a suitable place on the top of
  a page. 

  However, a figure without caption does not get a figure number and is not
  included in the List of Figures.¹
  
  ¹ A list of figures is not supported by Docutils but can be inserted
    with raw LaTeX. 
  
ODT
  uses a figure frame with empty caption field for the figure.

  However, a figure without caption will not get a figure number and not
  be listed in the List of Figures.¹
  
  ¹ A list of figures is not supported by Docutils but can be inserted
    in the LibreOfficeWriter.

A use case would be an illustration with the caption already included in
the image file, that should flow in LaTeX and/or use a consistent CSS
figure-layout in HTML.

It seems a rare case and support is patchy, so it is, IMO, 
be OK if validation issues a warning. 

* Should the rST user documentation 
  https://docutils.sourceforge.io/docs/ref/rst/directives.html#figure
  be updated/expanded to warn that missing both caption and legend results
  in an invalid <figure> element (suggestions on how to formulate this are
  welcome)?
  
* Should the rST parser report a warning or an error in addition to the
  warning with "--validate"?  

Günter

Re: [Docutils-develop] Waiting for feedback about issues with the Docutils Doctree [#489]

[Karl O. P. <ko...@ka...>]

On Thu, 20 Jun 2024 13:22:31 -0400
Alan <ala...@gm...> wrote:

> Since I have not seen any developer feedback,
> here is some user feedback.
> 
> - citations without content: I've never seen it
> - footnotes without content or without "label": I've never seen it
> - figures without caption/legend: no "legend" is reasonably common,
> but no caption is extremely rare and is a need met by `image` (I
> believe)

Figures with no caption would still show up in the TOCs list
of figures.  (Or where ever that list shows up.) At least
in docbook.  I think.  Although I don't know how useful that'd
be since no description would be listed.  I suppose it'd have a figure
number, which is not entirely useless.

Further, I've not tried it and didn't go to the DTD or the
specs and actually look.  So take all this with a big
grain of salt.

Anyway, the big difference between figures and images,
in my mind, is that figures tend to appear in some sort
of listing somewhere and images don't.

Regards,

Karl <ko...@ka...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein