docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:bugs] Re: #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

> I think we should use Path.as_uri() instead of .as_posix().

Done in [r9882].


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Sat Aug 10, 2024 11:04 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

Another example why we should make it clear that the image argument is an "URI reference":
~~~
.. image:: My Documents/first drawing.png
~~~
is parsed to
~~~
.. image:: My Documents/first drawing.png
~~~
because [spaces are dropped from link blocks and image-URIs](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#uri-context) to allow easy breaking of long URIs over several lines.
This behaviour is surprising when you consider an image attribute without scheme a path.


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Sat Aug 10, 2024 02:48 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

Your analysis with  "the `uri_parts.scheme not in ('', 'file')` check" beeing "the root cause" of the test failure is correct. This check was introduced in [r9501] "Refactor "_html_base" writer."

However, even before (since introduction of image embedding in 0.18), the drive part was split from the path. To verify this, try `PS> echo ".. image:: S:/Development/docutils/docutils/test/data/circle.svg" | python rst2html5.py --link-stylesheet --image-loading embed` from a directory in a different drive.

BTW: both, the introduction of the :loading: directive option and of direct embedding of SVG images did not change path handling (you should get the same behaviour with a PNG or JPG image).



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Sat Aug 10, 2024 02:09 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

> the second is needlessly restrictive and hard to explain.

Suppose a user has a directive ``.. image:: /media/circle.svg``. By default this will load the file under `/media/circle.svg`, as the user might expect if they think they are using filesystem paths, and don't realise the nuances of URI references. At some later date, this user sets `root_prefix` to `/home/user/project/`. The image directive will stop working, as it now looks for a file under `/home/user/project/media/circle.svg`.

I think this behaviour is hard to explain and quite  surprising.

I have a proposed fix at https://github.com/AA-Turner/docutils/pull/18 (https://github.com/AA-Turner/docutils/pull/18.patch). This does the following:

* Adds a helper function to convert image URIs to paths
* Updates all writers (HTML, LaTeX, and ODF/ODT) that support images to use this helper for consistent behaviour
* Correctly convert `file:///` URIs to filesystem paths (using Python 3.13's [Path.from_uri()](https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.from_uri) where possible).
* Validate that `file:` URIs are absolute paths
* Correctly convert relative references to filesystem paths
* Changes the default `root_prefix` to `''` (the empty string), and only applies the `root_prefix` behaviour when it is non-empty (i.e. a user has configured the setting)

The second commit in the patch goes further, by rejecting absolute paths when used in relative references. This is optional, but as I argue above I think allowing absolute paths in relative references leads to surprising behaviour, and the resolution is easy if you want to keep using absolute paths (just use `file:///`). I think this would be a straightforward end-state to explain to users: either the argument to `.. image::` (and `.. figure::`) is a URI or it is a relative filesystem path.

A


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Sat Aug 10, 2024 08:29 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

 > I think we have two options. 
 > (1) support absolute paths on every platform, document as such. 
 > (2) make absolute paths return an error on Linux. 
 
 (3) support URI-references: URI | absolute (POSIX) path | relative (POSIX) path

>The first restores backwards compatibility, 

the second is needlessly restrictive and hard to explain.

The third is consistent with what we document. 
It is also in line with "root-prefix" only being used if a path starts with "/".
 
 The different handling of Windows system paths is due to the syntax definition of URIs where the "path" part is represented by a POSIX path.
 
 ---
> I think we should use Path.as_uri() instead of .as_posix().

I prepared a patch. 

BTW: the docs for .as_posix() write:
~~~
>>> p = PureWindowsPath('c:/Windows')
>>> p.as_uri()
'file:///c:/Windows'
 ~~~
 Then `/c:/Windows` may be usable as both, absolute file system path under Windows and "absolute-path reference" [1](https://www.rfc-editor.org/rfc/rfc3986.html#section-4.2)  in a URI-reference like:
 ~~~
 .. image:: /E:/DCIM/parrot.jpg
  ~~~


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Fri Aug 09, 2024 09:55 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

> The parsing is correct: The image argument is a "URI reference" ...

The issue here now becomes why the tests were not failing on Linux, I believe. Both Windows and Linux would have had an absolute path (e.g. `.. image:: /home/user/development/docutils/docutils/test/data/circle-broken.svg`). On Windows, this failed due to misinterpreting the drive letter as a scheme. However, because the default `root_prefix` is `/`, when an absolute path is used on Unix, the scheme is `''`,  `imagepath.startswith('/')` is `True`, and `root_prefix/imagepath[1:]` just strips and restores the leading slash.

I believe that if we are only accepting URI-refereneces, then we should reject absolute paths on UNIX, and the tests should fail.

`--image-loading` was added in Docutils 0.18 and `:loading:` was added in 0.21. Local testing shows that on Linux, absolute paths are allowed with `--image-loading embed` or `:loading: embed` in every released version and the current master branch. On Windows, 0.18-0.20 allowed absolute paths, but since 0.21 an error has been raised.

The following table demonstrates the matrix. `image_loading` means using `--image-loading embed` and `:loading:` means using `:loading: embed`.  My test commands were:

```console
$ printf ".. image:: /mnt/s/Development/docutils/docutils/test/data/circle.svg" | rst2html5 --link-stylesheet --image-loading embed
$ printf ".. image:: /mnt/s/Development/docutils/docutils/test/data/circle.svg\n   :loading: embed" | rst2html5 --link-stylesheet
PS> echo ".. image:: S:/Development/docutils/docutils/test/data/circle.svg" | python rst2html5.py --link-stylesheet --image-loading embed
PS> echo ".. image:: S:/Development/docutils/docutils/test/data/circle.svg`n   :loading: embed" | python rst2html5.py --link-stylesheet
```

*Absolute path support in the image directive with embed mode*

|      | **Linux**     | **Linux** | **Windows**   | **Windows** |
|------|---------------|-----------|---------------|-------------|
|      | `image_loading` | `:loading:` | `image_loading` | `:loading:`   |
| 0.18 | base64        | N/A       | base64        | N/A         |
| 0.19 | base64        | N/A       | base64        | N/A         |
| 0.20 | base64        | N/A       | base64        | N/A         |
| 0.21 | direct        | direct    | error         | error       |
| 0.22 | direct        | direct    | error         | error       |

I think we have two options. (1) support absolute paths on every platform, document as such. (2) make absolute paths return an error on Linux. The first restores backwards compatibility, the second is consistent with what we document.

> [r9880] 

A nit-pick here -- this commit leads to URIs like `file://s:/...` on Windows. From Wikipedia's page on file URIs "*`file://path` (i.e. two slashes, without a hostname) is never correct, but is often used.*". 

I think we should use [Path.as_uri()](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.as_uri) instead of `.as_posix()`.

A



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Fri Aug 09, 2024 08:45 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #187 Rename .txt files to .rst

[Günter M. <mi...@us...>]

We would need to adapt background scripts, too. I am unsure whether the following changes correctly describe the behaviour after applying this patch :
~~~
--- a/sandbox/infrastructure/docutils-update.local
+++ b/sandbox/infrastructure/docutils-update.local
@@ -14,10 +14,10 @@
 #
 # ATTENTION
 #
-# Any .html document with a corresponding .txt file is regenerated 
-# if the .txt has changed, but no new .html files will be generated.
+# Any .html document with a corresponding .rst file is regenerated 
+# if the .rst has changed, but no new .html files will be generated.
 #
-# * Funny thing: sf hides README.txt files.
+# * Funny thing: sf hides README.rst files.
 #
 # ATTENTION
~~~

I downloaded the patch set and tried to commit it to a local branch but failed:
~~~
milde@heinz:/usr/local/src/docutils-git-svn/docutils/docutils > git am -3 /tmp/17.patch 
Applying: Rename all .txt files to .rst
Applying: Update references to .txt files
Using index info to reconstruct a base tree...
M	docutils/test/test_utils/test__init__.py
.git/rebase-apply/patch:7937: trailing whitespace.
# Any .html document with a corresponding .rst file is regenerated 
error: patch failed: docutils/docs/ref/rst/history.rst:1
error: docutils/docs/ref/rst/history.rst: patch does not apply
error: Did you hand edit your patch?
It does not apply to blobs recorded in its index.
Patch failed at 0002 Update references to .txt files
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
~~~



---

**[patches:#187] Rename .txt files to .rst**

**Status:** pending-remind
**Group:** None
**Created:** Wed Jan 05, 2022 05:51 PM UTC by Adam  Turner
**Last Updated:** Fri Aug 09, 2024 05:12 AM UTC
**Owner:** Adam  Turner


This change is in two parts -- the first commit does the rename and the second goes through and updates references to .txt files. All tests pass.

The benefit of this change is primarily for people -- on user interfaces with syntax highlighting (e.g. IntelliJ / VSCode / Notepad++ editors, code mirrors, etc), the text is presented natively as reStructuredText, and editor features can assist with e.g. autocompletion.

Docutils itself of course does not care which file extension is used. 

I have not renamed any of the include files or template files in `docutils.parsers` or `docutils.writers`, as they form part of the public API and would need a deprecation cycle (or aliasing in the parsing code)

A

Please see https://github.com/AA-Turner/docutils/pull/3 and https://github.com/AA-Turner/docutils/pull/3.patch for the commits and patch.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

The parsing is correct: The image argument is a "URI reference":
> A URI-reference is either a URI or a relative reference.  If the
   URI-reference's prefix does not match the syntax of a scheme followed
   by its colon separator, then the URI-reference is a relative
   reference. (RFC 3986)
   
   Hence, the example
   ~~~
   .. image:: S:/Development/data/circle-broken.svg
   ~~~
   describes an image accessible under scheme "S".
   
Our problem is, that we cannot easily use relative paths (to keep flexibility regarding where to run the tests from).
A simple solution would be using a "file" scheme in the offending examples:
~~~
diff --git a/docutils/test/test_writers/test_html5_polyglot_parts.py b/docutils/test/test_writers/test_html5_polyglot_parts.py
index ba139d65c..102273234 100755
--- a/docutils/test/test_writers/test_html5_polyglot_parts.py
+++ b/docutils/test/test_writers/test_html5_polyglot_parts.py
@@ -714,7 +714,7 @@ def format_output(self, parts):
 """,
 }],
 [f"""\
-.. image:: {DATA_ROOT}/circle-broken.svg
+.. image:: file://{DATA_ROOT}/circle-broken.svg
    :loading: embed
 """,
 {'fragment': f"""\
@@ -725,7 +725,7 @@ def format_output(self, parts):
 
 <aside class="system-message">
 <p class="system-message-title">System Message: ERROR/3 (<span class="docutils literal"><string></span>, line 1)</p>
-<p>Cannot parse SVG image "{DATA_ROOT}/circle-broken.svg":
+<p>Cannot parse SVG image "file://{DATA_ROOT}/circle-broken.svg":
   not well-formed (invalid token): line 3, column 48</p>
 </aside>
 """
@@ -831,7 +831,7 @@ def format_output(self, parts):
 """,
 }],
 [f"""\
-.. image:: {DATA_ROOT}/circle-broken.svg
+.. image:: file://{DATA_ROOT}/circle-broken.svg
    :loading: embed
 """,
 {'fragment': """\
~~~


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Thu Aug 08, 2024 08:43 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #187 Rename .txt files to .rst

[Adam T. <aa-...@us...>]

I have recreated the branch/patch for the rename, please have a look:

* https://github.com/AA-Turner/docutils/pull/17
* https://github.com/AA-Turner/docutils/pull/17.patch

All `.txt` files in docutils/, web/, and sandbox/infrastructure/ have been renamed. 

Writer template files remain unchanged.

A


---

**[patches:#187] Rename .txt files to .rst**

**Status:** pending-remind
**Group:** None
**Created:** Wed Jan 05, 2022 05:51 PM UTC by Adam  Turner
**Last Updated:** Thu Aug 08, 2024 07:37 PM UTC
**Owner:** Adam  Turner


This change is in two parts -- the first commit does the rename and the second goes through and updates references to .txt files. All tests pass.

The benefit of this change is primarily for people -- on user interfaces with syntax highlighting (e.g. IntelliJ / VSCode / Notepad++ editors, code mirrors, etc), the text is presented natively as reStructuredText, and editor features can assist with e.g. autocompletion.

Docutils itself of course does not care which file extension is used. 

I have not renamed any of the include files or template files in `docutils.parsers` or `docutils.writers`, as they form part of the public API and would need a deprecation cycle (or aliasing in the parsing code)

A

Please see https://github.com/AA-Turner/docutils/pull/3 and https://github.com/AA-Turner/docutils/pull/3.patch for the commits and patch.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

Sorry, another failure. 

The root cause of this one seems to be the `uri_parts.scheme not in ('', 'file')` check. On Windows,  we get the following, because drives in absolute paths are parsed as URI schemes:

```pycon
>>> uri
'S:/Development/docutils/docutils/test/data/circle-broken.svg'
>>> urllib.parse.urlparse(uri).scheme
's'
```

We could do something like ``Path(uri).is_file()``, but that has the problems with using pathlib that you mentioned earlier...

Full failure below, also on https://github.com/AA-Turner/docutils/actions/runs/10308701375/job/28536730749

```
============================================================================================================= FAILURES ==============================================================================================================
_________________________________________________________________________ Html5WriterPublishPartsTestCase.test_publish (id="totest['system_messages'][1]") __________________________________________________________________________

self = <test.test_writers.test_html5_polyglot_parts.Html5WriterPublishPartsTestCase testMethod=test_publish>

    def test_publish(self):
        if not with_pygments:
            del totest['syntax_highlight']
        writer = 'html5'
        for name, (settings_overrides, cases) in totest.items():
            for casenum, (case_input, case_expected) in enumerate(cases):
                with self.subTest(id=f'totest[{name!r}][{casenum}]'):
                    parts = docutils.core.publish_parts(
                        source=case_input,
                        writer=writer,
                        settings_overrides={
                            '_disable_config': True,
                            'strict_visitor': True,
                            'stylesheet': '',
                            'section_self_link': True,
                            **settings_overrides,
                        }
                    )
>                   self.assertEqual(case_expected, self.format_output(parts))
E                   AssertionError: {'fragment': '<svg xmlns="http://www.w3.org/2000/svg"\n [410 chars]>\n'} != {'fragment': '<img alt="S:/Development/docutils/docutils[396 chars]>\n'}
E                   + {'fragment': '<img '
E                   +              'alt="S:/Development/docutils/docutils/test/data/circle-broken.svg" '
E                   +              'src="S:/Development/docutils/docutils/test/data/circle-broken.svg" '
E                   - {'fragment': '<svg xmlns="http://www.w3.org/2000/svg"\n'
E                   -              '     viewBox="0 0 10 10">\n'
E                   -              '  <circle cx="5" cy="5" r="4" fill="lightblue" x/>\n'
E                   -              '</svg>\n'
E                   -              '\n'
E                   +              '/>\n'
E                   ?               ++
E                   
E                                  '<aside class="system-message">\n'
E                                  '<p class="system-message-title">System Message: ERROR/3 (<span '
E                                  'class="docutils literal"><string></span>, line 1)</p>\n'
E                   -              '<p>Cannot parse SVG image '
E                   ?                         ---- ^^^^
E                   
E                   +              '<p>Cannot embed image '
E                   ?                          ^^^^
E                   
E                                  '"S:/Development/docutils/docutils/test/data/circle-broken.svg":\n'
E                   -              '  not well-formed (invalid token): line 3, column 48</p>\n'
E                   +              '  Can only read local images.</p>\n'
E                                  '</aside>\n'}

docutils\test\test_writers\test_html5_polyglot_parts.py:92: AssertionError
________________________________________________________________________ Html5WriterPublishPartsTestCase.test_publish (id="totest['no_system_messages'][1]") ________________________________________________________________________

self = <test.test_writers.test_html5_polyglot_parts.Html5WriterPublishPartsTestCase testMethod=test_publish>

    def test_publish(self):
        if not with_pygments:
            del totest['syntax_highlight']
        writer = 'html5'
        for name, (settings_overrides, cases) in totest.items():
            for casenum, (case_input, case_expected) in enumerate(cases):
                with self.subTest(id=f'totest[{name!r}][{casenum}]'):
                    parts = docutils.core.publish_parts(
                        source=case_input,
                        writer=writer,
                        settings_overrides={
                            '_disable_config': True,
                            'strict_visitor': True,
                            'stylesheet': '',
                            'section_self_link': True,
                            **settings_overrides,
                        }
                    )
>                   self.assertEqual(case_expected, self.format_output(parts))
E                   AssertionError: {'fragment': '<svg xmlns="http://www.w3.org/2000/svg"\n [85 chars]n\n'} != {'fragment': '<img alt="S:/Development/docutils/docutils[98 chars]>\n'}
E                   + {'fragment': '<img '
E                   +              'alt="S:/Development/docutils/docutils/test/data/circle-broken.svg" '
E                   +              'src="S:/Development/docutils/docutils/test/data/circle-broken.svg" '
E                   - {'fragment': '<svg xmlns="http://www.w3.org/2000/svg"\n'
E                   -              '     viewBox="0 0 10 10">\n'
E                   -              '  <circle cx="5" cy="5" r="4" fill="lightblue" x/>\n'
E                   -              '</svg>\n'
E                   -              '\n'}
E                   +              '/>\n'}
E                   ?               ++

docutils\test\test_writers\test_html5_polyglot_parts.py:92: AssertionError
```

A



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Thu Aug 08, 2024 08:38 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

- **status**: open-fixed --> open



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Thu Aug 08, 2024 04:24 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #58 Migration Docutils from SourceForge to Github

[Adam T. <aa-...@us...>]

As another update, I have recently made several improvements to the draft conversion files (see [r9856] and [r9868]) that I've been working on over the last couple of weeks. I also have written to Günter (@milde) privately to resolve some other matters, relating to migrating issues and the authors map.

A


---

**[feature-requests:#58] Migration Docutils from SourceForge to Github**

**Status:** pending
**Group:** Default
**Created:** Fri Feb 16, 2018 03:23 PM UTC by Yves Chevallier
**Last Updated:** Thu Sep 08, 2022 02:15 PM UTC
**Owner:** nobody


Sourceforge is not really user friendly to report issues, propose pull-request and contribute to the project. I would like to know if it is possible to migrate Docutils to GitHub. 


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #187 Rename .txt files to .rst

[Günter M. <mi...@us...>]

- **assigned_to**: David Goodger --> Adam  Turner



---

**[patches:#187] Rename .txt files to .rst**

**Status:** pending-remind
**Group:** None
**Created:** Wed Jan 05, 2022 05:51 PM UTC by Adam  Turner
**Last Updated:** Thu Aug 08, 2024 07:36 PM UTC
**Owner:** Adam  Turner


This change is in two parts -- the first commit does the rename and the second goes through and updates references to .txt files. All tests pass.

The benefit of this change is primarily for people -- on user interfaces with syntax highlighting (e.g. IntelliJ / VSCode / Notepad++ editors, code mirrors, etc), the text is presented natively as reStructuredText, and editor features can assist with e.g. autocompletion.

Docutils itself of course does not care which file extension is used. 

I have not renamed any of the include files or template files in `docutils.parsers` or `docutils.writers`, as they form part of the public API and would need a deprecation cycle (or aliasing in the parsing code)

A

Please see https://github.com/AA-Turner/docutils/pull/3 and https://github.com/AA-Turner/docutils/pull/3.patch for the commits and patch.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #187 Rename .txt files to .rst

[Günter M. <mi...@us...>]

Thinking about this, I now favour changing the extension of rST source files to `.rst`. The PEP sources at peps.python.org are an "official precedence case".

Writer template files  are *not* rST sources and should keep their extension.

Standard include files shoud change after an advance warning and transition period.
(We can implement a fallback with warning in the "include" directive implementation.)


---

**[patches:#187] Rename .txt files to .rst**

**Status:** pending-remind
**Group:** None
**Created:** Wed Jan 05, 2022 05:51 PM UTC by Adam  Turner
**Last Updated:** Thu Feb 17, 2022 02:07 PM UTC
**Owner:** David Goodger


This change is in two parts -- the first commit does the rename and the second goes through and updates references to .txt files. All tests pass.

The benefit of this change is primarily for people -- on user interfaces with syntax highlighting (e.g. IntelliJ / VSCode / Notepad++ editors, code mirrors, etc), the text is presented natively as reStructuredText, and editor features can assist with e.g. autocompletion.

Docutils itself of course does not care which file extension is used. 

I have not renamed any of the include files or template files in `docutils.parsers` or `docutils.writers`, as they form part of the public API and would need a deprecation cycle (or aliasing in the parsing code)

A

Please see https://github.com/AA-Turner/docutils/pull/3 and https://github.com/AA-Turner/docutils/pull/3.patch for the commits and patch.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #98 Warn on title underline too long

[Günter M. <mi...@us...>]

- **status**: pending --> closed-wont-fix
- **Comment**:

Warnings are created for issues where there is invalid rST or ambiguity about the author's intention. A too short title underline is invalid.
A "too long" title underline is valid rST syntax and the probability that the author intended it to be something other than a title underline is too low to merit a warning.



---

**[feature-requests:#98] Warn on title underline too long**

**Status:** closed-wont-fix
**Group:** Default
**Created:** Sat Oct 28, 2023 10:14 AM UTC by xypron
**Last Updated:** Thu Nov 09, 2023 05:05 PM UTC
**Owner:** nobody


If title underlines are too short, docutils creates a warning:

docutils/parsers/rst/states.py:2750:
        msg = self.reporter.warning('Title underline too short.', 

Many documentation projects want the underline to match the title. It would be great if we could have a warning for titles that are too long too.  Maybe some switch would be needed to disable the warning.



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open-fixed
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 11:37 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #492 Git repo lacking tags

[jfbu <jf...@us...>]

For a real life example where tags would have helped see [this Sphinx comment](https://github.com/sphinx-doc/sphinx/pull/12745#issuecomment-2275315114). There are no tags associated to releases prior to 0.10 and one has to search manually for appropriate commit messages and then check them out.  And for tags since 0.10, with the exception of 0.14, they are as reported seemingly annotating some "mirrored" history, which has the commits but with other SHAs and not direct ancestors of current master tip. For example revision 5862 has two associated commits:
~~~
$ git checkout docutils-0.10
$ git log --grep docutils@5862
commit 59d5b34c2692d494a7bb3d680eb681c8ebbcb42e
Author: milde <milde@929543f6-e4f2-0310-98a6-ba3bd3dd1d04>
Date:   Fri Jan 30 13:37:26 2009 +0000

    latex2e writer: backwards compatible implementation of failsave custom roles
    
    git-svn-id: https://docutils.svn.sourceforge.net/svnroot/docutils/trunk/docutils@5862 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~
and
~~~
$ git checkout master
$ git log --grep trunk@5862
commit 472ada29b7003d67d955b9ad71c9e94f3156a048
Author: milde <milde@929543f6-e4f2-0310-98a6-ba3bd3dd1d04>
Date:   Fri Jan 30 13:37:26 2009 +0000

    latex2e writer: backwards compatible implementation of failsave custom roles
    
    git-svn-id: https://docutils.svn.sourceforge.net/svnroot/docutils/trunk@5862 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
~~~
but you can't find both at same time, and the latter is as shown the one you find on master branch and then it is not an ancestor of the  `docutils-0.10` tag.


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Wed Aug 07, 2024 09:09 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

- **Comment**:

Fixed in [r9866]. Documentation updated in [r9865].
Thank you for report and analysis.



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 02:54 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] manpage option "use_reference_macros"

[engelbert g. <eng...@gm...>]

Hei Günter

the option is here to become default ... maybe

the macros UR/UE and MT/ME are in man-macros since years
but the macros changed to produce OSC8 statements which offer opening the
link in terminals supporting this
but on my ubuntu 2404 it did not work easily so i choose text output for
the previous release
and plan to use text output as default for next release ...

it is still in progress, but as the default output did is still text i do
it piecewise

macro-references / text-references is fine for me.

thanks for your work
 e

On Sun, 4 Aug 2024 at 22:36, Guenter Milde via Docutils-develop <
doc...@li...> wrote:

> Dear Engelbert,
>
> is the manpage writer option "use_reference_macros" intended to stay?
>
> If yes, it would need an entry in docs/user/config.txt.
>
> Also, I suggest adding a command line option to reset the default
> in case a config file sets it to True.
>
> Suggestion:
>   rename "use_reference_macros" to "macro_references" and use the pair
>
>   --macro-references
>   --text-references
>
> See, e.g., "--link-stylesheet" / "--embed-stylesheet" in
> latex2e/__init__.py
> for an example.
>
> Günter
>
>
>
> _______________________________________________
> Docutils-develop mailing list
> Doc...@li...
> https://lists.sourceforge.net/lists/listinfo/docutils-develop
>
> Please use "Reply All" to reply to the list.
>

[Docutils-develop] [docutils:bugs] #490 EncodingWarnings in io module

[Günter M. <mi...@us...>]

> The only writer that makes runtime use of the output encoding setting is LaTeX. 

**Output** encoding defaults to "utf-8" for all writers since several years.
Most writers honour the "output-encoding" setting and encode the output file accordingly. So, you may use "rst2html5 --output-encoding=ASCII:xmlcharrefreplace" to have a pure ASCII file.
The HTML, XML, and LaTeX writers also specify the used encoding in the file,  the LaTeX  writer also provides replacements for Unicode characters that are not encodable if a legacy output encoding is selected.  There should be no cases of `output_encoding == None`.

**Input** encoding was "auto-select" with fallback to utf-8 and "locale encoding" until 0.21. After the discussion last year the transition to utf-8 started: 0.22 uses "utf-8" as input encoding default,
we will remove the input encoding auto-detection code in Docutils 1.0. 

The offending test case, "test_fallback_no_utf8()" is more trouble than help and removed in [r9864].




---

**[bugs:#490] EncodingWarnings in io module**

**Status:** open-fixed
**Created:** Fri Jun 28, 2024 03:34 PM UTC by Jason R. Coombs
**Last Updated:** Thu Aug 01, 2024 08:24 PM UTC
**Owner:** nobody


When running the [distutils](https://github.com/pypa/distutils) tests with `PYTHONWARNDEFAULTENCODING=1`, two warnings are emitted:

```
distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:381: EncodingWarning: 'encoding' argument not specified
    self.source = open(source_path, mode,

distutils/tests/test_check.py::TestCheck::test_check_restructuredtext
  /Users/jaraco/code/pypa/distutils/.tox/py/lib/python3.12/site-packages/docutils/io.py:151: EncodingWarning: UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.
    fallback = locale.getpreferredencoding(do_setlocale=False)
```

Docutils should honor [PEP 597](https://peps.python.org/pep-0597/) and address these warnings (and possibly others). In my experience, adding `encoding='utf-8'` to any io operation is the best approach - it's straight-up compatible with the default on non-Windows systems and usually honoring the Unix convention is suitable if not preferable on Windows. Not only that, but that behavior will become the default in Python 3.15 or so.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

I am in favour of using unquote().

> Now it seems that (at least from wikipedia) this scheme of using | is not standardised/used, and might just be a Python quirk 

The "|" seems to be not Python specific, however, it is obsolete.  Wiki  says:
> On Microsoft Windows systems, the normal colon (:) after a device letter has sometimes been replaced by a vertical bar (|) in file URLs. This reflected the original URL syntax, which made the colon a reserved character in a path part.
Since Internet Explorer 4, file URIs have been standardized on Windows, and should follow the following scheme. 

URI vs. Path

> This seems to be a problem of imprecise language. picture.png as used in the example is not a URI, as it has no scheme. 
> ...
> we accept either a URI-with-scheme (file: or https: etc) or a local file path. 

Actually not: by default, the "uri" attribute is written as-is to the HTML `<img>` element's "src" attribute which expects an URL and may fail for a number of valid system paths.

It seems what Docutils expects is a [URI-Reference](https://datatracker.ietf.org/doc/html/rfc3986#section-4.1): 

  > A URI-reference is either a URI or a relative reference.  
  > If the URI-reference's prefix does not match the syntax of a scheme followed
   by its colon separator, then the URI-reference is a [relative reference](https://datatracker.ietf.org/doc/html/rfc3986#section-4.2).
   
In contrast to a system file path, a relative reference still requires forward slashes and quoting of some characters.


---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 01:12 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

> could you provide the relevant test output, please?

See https://github.com/AA-Turner/docutils/actions/runs/10284219635/job/28459783099 for an online view (I advise collapsing the "alltests" tab and scrolling up to the pytest one, which has better formatted output). The full failure is very verbose, but it is because `if imagepath.startswith('/')` fails.

Using ``unquote()`` does make the current tests pass on Windows, my question was more about what is the range of URIs that we deem valid? If we use ``unquote()``, then e.g. ``file:///C|users/john/mallard.png`` will be improperly translated:

```pycon
>>> from urllib.parse import urlparse, unquote
>>> from urllib.request import url2pathname
>>> print(url2pathname(urlparse('file:///C|users/john/mallard.png').path))
C:\users\john\mallard.png
>>> print(unquote(urlparse('file:///C|users/john/mallard.png').path))
/C|users/john/mallard.png
```

Now it seems that (at least [from wikipedia](https://en.wikipedia.org/wiki/File_URI_scheme)) this scheme of using `|` is not standardised/used, and might just be a Python quirk -- in which case I think just using ``unquote()`` is easier. Of note, ``pathlib`` does not use the ``|`` replacement mechanism.

> Per definitionem, there is no way to provide a system file-path as image ressource, both rST and the Docutils Doctree explicitely expect an URI.

This seems to be a problem of imprecise language. ``picture.png`` as used in the example is not a URI, as it has no scheme. Nor is ``/data/blue square.png`` or etc. Indeed, none of the ``.. image::`` directives in ``docs/`` have a scheme, and hence aren't URIs. The actual check that we do is in ``Image.run()``, ``reference = directives.uri(self.arguments[0])``, which just removes unescaped whitespace.

> Interpreting an image URI with undefined scheme as system path

This goes directly to my point, as the scheme is what makes a URI! (well, the scheme and the path).

I think that there is a strong argument based on historical practice in favour of documenting that we accept either a URI-with-scheme (`file:` or `https:` etc) or a local file path. However, I might be missing something, and I agree we should avoid backwards-incompatible changes. My main motivation is simply to fix the tests, and it seems that using ``unquote()`` will work, unless you forsee any other problems.

A



---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 12:22 PM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Günter M. <mi...@us...>]

Unfortunately, I don't know what the "test failure on Windows" actually is, could you provide the relevant test output, please?

> Is it possible to use the simpler unquote() here? 

If it works on Windows, using unquote() is a good idea. Could you test with
~~~
diff --git a/docutils/docutils/writers/_html_base.py b/docutils/docutils/writers/_html_base.py
index ead930e31..2eab36510 100644
--- a/docutils/docutils/writers/_html_base.py
+++ b/docutils/docutils/writers/_html_base.py
@@ -626,7 +626,7 @@ def uri2imagepath(self, uri):
         uri_parts = urllib.parse.urlparse(uri)
         if uri_parts.scheme not in ('', 'file'):
             raise ValueError('Can only read local images.')
-        imagepath = urllib.request.url2pathname(uri_parts.path)
+        imagepath = urllib.parse.unquote(uri_parts.path)
         if imagepath.startswith('/'):  # cf. config.html#root-prefix
             root_prefix = Path(self.settings.root_prefix)
             imagepath = (root_prefix/imagepath[1:]).as_posix()
~~~
and adapt the test script if required?

> Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

Per definitionem, there is no way to provide a system file-path as image ressource, both [rST](https://docutils.sourceforge.io/docs/ref/rst/directives.html#image) and the [Docutils Doctree](https://docutils.sourceforge.io/docs/ref/doctree.html#image)  explicitely expect an URI.

As URIs are capable of representing local file paths, I dont see the advantage.
Interpreting an image URI with undefined scheme as system path would be a complication of both, specification and implementation and a backwards-incompatible change. We would als have to think about images that are only read to get the size but kept external in HTML output.





---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 02:25 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #492 Git repo lacking tags

[Adam T. <aa-...@us...>]

> a new release is due once remaining issues are settled

At an appropriate time, I would be happy to help make the release, if useful -- an effort to increase our [bus factor](https://en.wikipedia.org/wiki/Bus_factor) perhaps! @grubert has written a very comprehensive set of notes in  `release.txt` & `releasing-log.txt` to follow!

A


---

**[bugs:#492] Git repo lacking tags**

**Status:** open
**Created:** Mon Aug 05, 2024 09:47 AM UTC by jfbu
**Last Updated:** Wed Aug 07, 2024 01:45 AM UTC
**Owner:** nobody


I updated my docutils git clone (from `url = git://repo.or.cz/docutils.git`), did `git pull --tags` which fetched tags such as `docutils-0.21.2`.  Then I started `git bisect` to research some issue but could not understand HEAD was seemingly thousands of commits ahead of  `docutils-0.21.2`.  In output of `git log --oneline` I indeed found much more recent releases, but none of these releases have git tags (which would normally show here).

~~~
4530fc43e FIX test no longer break on missing pil
f344d4daf releasing 0.21.2
323957acb Version 0.21.2
af87152e6 stale comment
aebceea6e Reconcile Docutils DTD and Document Tree documentation.
76ae9eec4 man utf8 output uses normal "-" char(45)
5aaa62b8e Uncomment classifiers for Georgian and Catalan (Valencian) languages.
fa86933d7 Small test speedup/simplification.
d10ff72f9 Make effect of centre-aligning figures visible in functional HTML text.
8218b2e47 Avoid dependency of functional tests on PIL/Pillow.
e6069cf76 Prevent test failure due no Pillow or Pillow version above 10.3.
fb9a9421d Remove duplicate test case.
d3fdd821f Do not exclude "test/functional/output/" from the source package.
e85d5159f Fix test failure if tests are started from docutils/docutils.
46db2819c Fix test failure with pygments >= 2.14.
6729e38e3 version 0.21.2b.dev
f83dac7cf release 0.21.1
64d4b0dd8 rough log of trying to install sdist from test.pypi
64733c201 Release 0.21.1 (sdist rerelease)
~~~

I am surely missing the obvious... (I dimly remember years ago having had already problems with tags on this project, which I think was simply due to me not having done `git pull --tags`...).


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #493 Test failure on Windows with embedded images

[Adam T. <aa-...@us...>]

---

**[bugs:#493] Test failure on Windows with embedded images**

**Status:** open
**Created:** Wed Aug 07, 2024 02:25 AM UTC by Adam  Turner
**Last Updated:** Wed Aug 07, 2024 02:25 AM UTC
**Owner:** nobody


xref [r9785], [r9853], [r9855]

Dear @milde,

Thank you for the fix to my recent patch. It seems neither my patch nor the fix addressed the root cause of the test failures, as tests have resumed failing on Windows.

I believe the following demonstrates the problem:

```pycon
>>> import sys; print(sys.platform)
win32
>>> import urllib.parse, urllib.request
>>> urllib.request.url2pathname('test/data/circle-broken.svg')
'test\\data\\circle-broken.svg'
>>> urllib.parse.unquote('test/data/circle-broken.svg')
'test/data/circle-broken.svg'
```

Currently, we use `imagepath = urllib.request.url2pathname(uri_parts.path)`, which converts path separators to their platform-native format. On UNIX, `url2pathname` simply calls `unquote`, but on Windows it handles UNC paths (``\\host\path\``) and escaped drive letters (``///C|/users/``).

I don't know what led to using `url2pathname()`, as it is quite specialised (the docstring notes "not recommended for general use"). Is it possible to use the simpler `unquote()` here?  

For local file paths (e.g. without a ``file:///`` scheme), should we even be using URI parsing? Perhaps we should use proper path handling if there is no URI scheme (i.e. the user has provided a file-path).

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.