docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:bugs] #446 Links with bracket references cause unexpected "Hyperlink target is not referenced" error

[Adam T. <aa-...@us...>]

It is technically correct output, although I agree surprising.

There are two targets in this document, `different name` and `a link`. 

`different name` is referenced by the alias reference syntax in line 1, but `a link` is never referenced.

The following document generates no errors:
```restructuredtext
`a link <different name_>`_

reference to `a link`_

.. _different name: https://www.example.com
```

Alternatively, you could use an anonymous reference:
```restructuredtext
`a link <different name_>`__

.. _different name: https://www.example.com
```

Perhaps we could improve the error message?

A


---

** [bugs:#446] Links with bracket references cause unexpected "Hyperlink target is not referenced" error**

**Status:** open
**Created:** Thu Apr 21, 2022 07:13 PM UTC by Jesse Brennan
**Last Updated:** Thu Apr 21, 2022 07:13 PM UTC
**Owner:** nobody


I'm getting a surprising error when running reporting on a minimal RST file.

Here's the minimal.rst:
~~~
`a link <different name_>`_

.. _different name: https//www.example.com
~~~

Now if I run
~~~
rst2html.py minimal.rst --report=1 --traceback > /dev/null
minimal.rst:3: (INFO/1) Hyperlink target "a link" is not referenced.
~~~

you see the error I get. As far as I know, this is perfectly valid RST, so I'm wondering if this error message is a bug.

For more context, I encountered this error originally on a different project: https://github.com/myint/rstcheck/issues/77 which makes use of docutils.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #447 docutils-cli.py not available when installing via pip

[Jesse B. <jes...@us...>]

---

** [bugs:#447] docutils-cli.py not available when installing via pip**

**Status:** open
**Created:** Thu Apr 21, 2022 07:19 PM UTC by Jesse Brennan
**Last Updated:** Thu Apr 21, 2022 07:19 PM UTC
**Owner:** nobody


This might not be an actual bug, but the behavior is surprising, and the documentation lead me to believe that docutils-cli.py should be available regardless of installation method.

I created a new virtual environment with Python 3.8.3. With the virtual environment active, I updated to the latest version of pip, then installed docutils.

I then tried to run docutils-cli.py but the command could not be found. Other tools were available, such as rst2html.py.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #446 Links with bracket references cause unexpected "Hyperlink target is not referenced" error

[Jesse B. <jes...@us...>]

---

** [bugs:#446] Links with bracket references cause unexpected "Hyperlink target is not referenced" error**

**Status:** open
**Created:** Thu Apr 21, 2022 07:13 PM UTC by Jesse Brennan
**Last Updated:** Thu Apr 21, 2022 07:13 PM UTC
**Owner:** nobody


I'm getting a surprising error when running reporting on a minimal RST file.

Here's the minimal.rst:
~~~
`a link <different name_>`_

.. _different name: https//www.example.com
~~~

Now if I run
~~~
rst2html.py minimal.rst --report=1 --traceback > /dev/null
minimal.rst:3: (INFO/1) Hyperlink target "a link" is not referenced.
~~~

you see the error I get. As far as I know, this is perfectly valid RST, so I'm wondering if this error message is a bug.

For more context, I encountered this error originally on a different project: https://github.com/myint/rstcheck/issues/77 which makes use of docutils.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] setup_option_parser slow

[Guenter M. <mi...@us...>]

Dear Jeremy,

On 2022-04-05, Jeremy Maitin-Shepard wrote in gmane.text.docutils.devel:

> Every construction of a Publisher object calls setup_option_parser which is
> rather expensive.

> When processing a small number of large documents the overhead may not be
> too significant, but when using Sphinx to process a large number of small
> documents (e.g. for API documentation where every symbol is a separate
> page), >10% of the total time is spent just in setup_option_parser.

> As a hack I was able to mitigate this problem by monkey patching
> Publisher.setup_option_parser to cache the option_parser based on
> `type(self.parser), type(self.reader), type(self.writer)` but that assumes
> that the options depend only the class (which is true in practice for my
> use case).

> It would be great to have a proper solution for this, though.

A possible solution would be to cache or pre-fetch the settings object
instead of the option_parser:

  Settings assigned to the settings parameter of the convenience
  functions or the Publisher.settings attribute are used instead of the
  above sources 

  -- https://docutils.sourceforge.io/docs/dev/runtime-settings-processing.html#settings-priority
 

If, for a Publisher instance `publisher`, ``publisher.settings`` is defined,
the call to ``publisher.setup_option_parser()`` is skipped.
https://docutils.sourceforge.io/docs/dev/runtime-settings-processing.html#runtime-settings-processing-for-other-applications


Maybe in Sphinx can use this when creating Publisher instances with
identical conditions for the settings (reader/parser/writer components
and docutils.conf files locations).

See docutils/tools/buildhtml.py for an example of settings pre-fetching.
Warning: the details there will change with the upcoming move from optparse
to argparse. https://sourceforge.net/p/docutils/bugs/441
Using ``Publisher.get_settings()`` should be safer.


Thank you for reporting your findings

Günter

[Docutils-develop] StateMachine.__init__ is slow

[Jeremy Maitin-S. <je...@je...>]

Each construction of a StateMachine object is fairly expensive as it calls
add_states which constructs all of the state objects and their transitions.

When using Sphinx to process a large number of small documents (though I
think the problem also applies to small documents), more than 5% of the
total time is spent just constructing the StateMachine objects --- e.g. 5
seconds out of 65 seconds total run time.

In general I think it would be better if the StateMachine definition were
decoupled from a particular execution of the state machine.  However, that
might require significant code changes.

There is already a minimal form of caching used in RSTState.nested_parse.
However, StateMachine objects are also created in nested_list_parse and
elsewhere.

As a hack I was able to mitigate this by adding a global cache of
StateMachine objects, keyed by the derived StateMachine class and by the
tuple of state classes, and modifying StateMachine.unlink to return the
object to the cache.  However i"m not sure if a global cache is an
appropriate solution to be added to docutils itself.

Please CC me in any replies as I'm not subscribed to the mailing list.

[Docutils-develop] setup_option_parser slow

[Jeremy Maitin-S. <je...@je...>]

Every construction of a Publisher object calls setup_option_parser which is
rather expensive.

When processing a small number of large documents the overhead may not be
too significant, but when using Sphinx to process a large number of small
documents (e.g. for API documentation where every symbol is a separate
page), >10% of the total time is spent just in setup_option_parser.

As a hack I was able to mitigate this problem by monkey patching
Publisher.setup_option_parser to cache the option_parser based on
`type(self.parser), type(self.reader), type(self.writer)` but that assumes
that the options depend only the class (which is true in practice for my
use case).

It would be great to have a proper solution for this, though.

Please CC me in any replies as I'm not subscribed to the mailing list.

Re: [Docutils-develop] Request to become a Project Developer

[Guenter M. <mi...@us...>]

Dear David, dear Docutils developers,

On 2022-02-22, Adam Turner wrote:

...
> I'm submitting this request to become a Project Developer.

I second Adam's application to become a Docutils developer.

Adam has brought forward many ideas, patches, and initatives. 
More importantly, he has shown patience and willingness to learn
and co-operate.


> I believe in absence of any other process that the decision over
> accepting me (or not!) would fall to David Goodger as "BDFN", but I
> welcome comment from anyone on my suitability.



Günter

[Docutils-develop] [docutils:bugs] #445 docutils generates links to redirecting PEP URLs

[Günter M. <mi...@us...>]

- **status**: open --> open-fixed
- **Comment**:

Thank you for reporting this recent URL change. The default PEP base URL is updated in [r9045].
As [the PEP base URL is configurable](https://docutils.sourceforge.io/docs/user/config.html#pep-base-url), you can silence the warnings also without updating Docutils.



---

** [bugs:#445] docutils generates links to redirecting PEP URLs**

**Status:** open-fixed
**Created:** Fri Mar 11, 2022 09:16 AM UTC by Arek_Fu
**Last Updated:** Fri Mar 11, 2022 09:16 AM UTC
**Owner:** nobody


The official URL for PEPs seems to have recently changed to the format [https://peps.python.org/pep-0008/](https://peps.python.org/pep-0008/). The links generated by docutils have the form [https://www.python.org/dev/peps/pep-0008](https://www.python.org/dev/peps/pep-0008); this redirects to the official URL, but it raises warnings in Sphinx's linkcheck builder. I initially thought this was an issue with Sphinx and opened a bug report [there](https://github.com/sphinx-doc/sphinx/issues/10253), but I think the links are actually generated by docutils.

Expected behaviour: docutils should generate links to the new PEP URLs.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #445 docutils generates links to redirecting PEP URLs

[Arek_Fu <ar...@us...>]

---

** [bugs:#445] docutils generates links to redirecting PEP URLs**

**Status:** open
**Created:** Fri Mar 11, 2022 09:16 AM UTC by Arek_Fu
**Last Updated:** Fri Mar 11, 2022 09:16 AM UTC
**Owner:** nobody


The official URL for PEPs seems to have recently changed to the format [https://peps.python.org/pep-0008/](https://peps.python.org/pep-0008/). The links generated by docutils have the form [https://www.python.org/dev/peps/pep-0008](https://www.python.org/dev/peps/pep-0008); this redirects to the official URL, but it raises warnings in Sphinx's linkcheck builder. I initially thought this was an issue with Sphinx and opened a bug report [there](https://github.com/sphinx-doc/sphinx/issues/10253), but I think the links are actually generated by docutils.

Expected behaviour: docutils should generate links to the new PEP URLs.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

On 2022-02-23, Adam Turner wrote:

>> Why `__public__` and not `__all__`?

> `__all__` has special meaning to the interpreter for `from module
> import *`. Perhaps it doesn't matter, but I didn't want to conflate the
> two usages.

IMO, the special meaning of `__all__` overlaps with "is a public object".
However, I don't think defining and maintaining `__all__` - lists for all
modules and classes in the docutils package is an undertaking not worth
the effort. 

The advice: "If you want to know whether an object is public, check the
docstring" is simple, easy to follow and makes it easy to implement.

...

> I'd prefer to adopt SemVer, but I dropped that from the proposal for
> later conversation. Would you be happy with adopting SemVer and
> updating the language in this section to "should" rather than "must"?

Semantic Versioning seems to emerge as the "consensus of least surprise".
I am fine with this. I prefer to keep it without additional constraints.
I updated the proposal in the sandbox accordingly.


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Wed Feb 23, 2022 09:53 PM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] Doctree validation

[Guenter M. <mi...@us...>]

On 2022-03-07, Brecht Machiels wrote:

> From time to time, I receive bug reports for rinohtype that are caused 
> by an invalid document tree. Sphinx plugins sometimes fail to respect 
> the separation between body and inline elements. For example, Breathe 
> adds a literal_block node as a child of a paragraph node [1]. While 
> most Sphinx builders happily render invalid document trees, I like to 
> be more strict about it.

> Of course, if Sphinx plugin authors do not get any feedback when 
> building an invalid document tree, it's easy to make this mistake. 
> Therefore, I would like to propose to add some basic document tree 
> checks to docutils. What do you think?

This sounds like a sensible idea to me.
Do you have some concrete ideas how to do this?

It should, IMO, not add to much overhead nor prevent the rendering of
"dirty" documents by writers that can handle this.

We currently have only an indirect test: the output of the docutils-xml
writer can be validated against the DTD e.g. with xmllint.
https://docutils.sourceforge.io/docs/dev/testing.html#footnote-3

Günter



> [1] https://github.com/brechtm/rinohtype/issues/312

[Docutils-develop] Doctree validation

[Brecht M. <br...@mo...>]

Hi,

>From time to time, I receive bug reports for rinohtype that are caused 
by an invalid document tree. Sphinx plugins sometimes fail to respect 
the separation between body and inline elements. For example, Breathe 
adds a literal_block node as a child of a paragraph node [1]. While 
most Sphinx builders happily render invalid document trees, I like to 
be more strict about it.

Of course, if Sphinx plugin authors do not get any feedback when 
building an invalid document tree, it's easy to make this mistake. 
Therefore, I would like to propose to add some basic document tree 
checks to docutils. What do you think?

P.S. Many thanks for the new styling of the documentation on 
docutils.sourceforge.io. Much appreciated!

Cheers,
Brecht

[1] https://github.com/brechtm/rinohtype/issues/312

Re: [Docutils-develop] Babel without Korean

[Guenter M. <mi...@us...>]

Dear Hoze,

thank you for reporting.

On 2022-03-04, Hoze wrote:

> In Babel class of the docutils.writers.latex2e module,
> the language_codes variable lacks 'ko': 'korean'.

> That causes a warning message, "no Babel option known for language
> 'ko'" every time I run sphinx or jupyter-book. It makes nearly no
> problem, yet annoying.

> It'll be appreciated if you add 'ko': 'korean'.

There are a number of problems:

1. Missing Babel support:

After adding 'ko': 'korean':: 

  diff --git a/docutils/docutils/writers/latex2e/__init__.py b/docutils/docutils/writers/latex2e/__init__.py
  index 71ac6bebe..6e62ac131 100644
  --- a/docutils/docutils/writers/latex2e/__init__.py
  +++ b/docutils/docutils/writers/latex2e/__init__.py
  @@ -356,6 +356,7 @@ class Babel:
           'is':           'icelandic',
           'it':           'italian',
           'ja':           'japanese',
  +        'ko':           'korean',
           'kk':           'kazakh',
           'la':           'latin',
           'lt':           'lithuanian',

I get a minimal LaTeX document::

  \documentclass[a4paper]{article}
  % generated by Docutils <https://docutils.sourceforge.io/>
  \usepackage{cmap} % fix search and cut-and-paste in Acrobat
  \usepackage{ifthen}
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage[korean,english]{babel}
  
  
  %%% Body
  \begin{document}
  
  한국어(韓國語)또는
  
  \end{document}

However, this does not compile. The error is ::

  ! Package babel Error: Unknown option `korean'. Either you misspelled it
  (babel)                or the language definition file korean.ldf was not found
  .

I cannot find a Babel language file "korean.ldf" on CTAN. So, IMO, the
currently reported error is correct: the LaTeX package Babel does not
support Korean, there is "no Babel option known for language 'ko'".

2. It works with "Polyglossia" (and `xelatex` or `lualatex`):

What I do find is "latex/polyglossia/gloss-korean.ldf", so it makes
sense, to add ``'ko': 'korean'`` to the polyglossia languages supported by
the "xetex" writer (for the UTF8-aware LaTeX engines `xelatex` and `lualatex`).

Testing the minimal example does not work out of the box, because the
default font "LatinModern" does not support the Korean script.
However, this can be soved by users via a custom LaTeX preamble or stylesheet,
e.g. ::

  \usepackage{fontspec}
  \setmainfont{Noto Sans CJK SC}[Language=Korean]

2. Sphinx uses a fork of the Docutils "latex" writer that is
   developed independently. Any fixes in Docutils will not
   help users of Sphinx -- unless ported over by Sphinx developers.
   
   I don't know about the situation with jupyter.
   
Günter   

[Docutils-develop] Babel without Korean

[Hoze <yi...@gm...>]

To whom it may concern,

In Babel class of the docutils.writers.latex2e module,
the language_codes variable lacks 'ko': 'korean'.

That causes a warning message, "no Babel option known for language 'ko'"
every time I run sphinx or jupyter-book.
It makes nearly no problem, yet annoying.

It'll be appreciated if you add 'ko': 'korean'.

Hoze Yi
>From South Korea

[Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Günter M. <mi...@us...>]

On 2022-03-02, Adam Turner wrote:

> I've combined your changes into the attached patch, they look good to
> me as enabling works.

Thanks for look at this. We are almost there.

`python3 -W all alltests.py` came up with a buch of DeprecationMessages which
could be solved with

~~~
--- a/docutils/test/test_settings.py
+++ b/docutils/test/test_settings.py
@@ -108,8 +108,7 @@ class ConfigFileTests(unittest.TestCase):
     def setUp(self):
         warnings.filterwarnings(action='ignore',
                                 category=frontend.ConfigDeprecationWarning)
-        warnings.filterwarnings(action='ignore', module='docutils.frontend',
-                                category=PendingDeprecationWarning)
+        warnings.filterwarnings(action='ignore', category=DeprecationWarning)
         self.option_parser = frontend.OptionParser(
             components=(pep_html.Writer, rst.Parser), read_config_files=None)
~~~

Note: testing with warnings switched on is helpful. (It would also be
      good to run one or several Sphinx builds with activated warnings,
      to see whether there are uses of deprecated functions/classes.)

Now, only the entries in RELEASE-NOTES (bacwards-incompatible changes,
deprecations) and maybe HISTORY (other important changes) are missing.

...
>> I retract the suggestion to make the move now

> OK. It would be good to talk more about the "settings api" proposals. 

Yes. But this may be done in parallel to (or after) releasing 0.19b, ..., 0.19,
so that the deprecation warnings are out.


>> [...] I am increasingly convinced that settings updated from
>> configuration files are not "default settings".

> Perhaps "initital" settings?

Thats an option. Best would be to find out whether there is an precedence
of projects using both, ConfigParser and ArgumentParser, how they combine
both, and how they name partially configured settings.)



---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Wed Mar 02, 2022 02:40 AM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Adam T. <aa-...@us...>]

I've combined your changes into the attached patch, they look good to me as enabling works.

> retract the suggestion to make the move now

OK. It would be good to talk more about the "settings api" proposals. 

> I don't have a compelling name suggsestion, unfortunately, but I am increasingly convinced that settings updated from configuration files are not "default settings".

> In programmatic use, one can easily differentiate "defaults" fallbacks, set by Docutils vs. "custom values", set in the configuration file (with the defaults as fallback).
> This is less clear in command line use (where config-file-customized values that are fed to the OptionParser/ArgumentParser as "default=...").

Perhaps "initital" settings?

A


Attachments:

- [0001-Preparations-for-argparse-migration.patch](https://sourceforge.net/p/docutils/bugs/_discuss/thread/c76f9a2618/45c6/3ffa/7fbd/0cd9/7a23/attachment/0001-Preparations-for-argparse-migration.patch) (27.0 kB; application/octet-stream)


---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Tue Mar 01, 2022 10:50 PM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Günter M. <mi...@us...>]

Thank you again, Adam for the work on this tricky problem.

I propose a change of plan:
My original naïve idea was:
> After raising the Python dependency to >=3.7, now may be the
> right time to make the move.

In the light of the complexity and extent of the required changes, I
retract the suggestion to make the move now. The danger of accidential
breaks for 3rd party code is IMO too high for this change to be done
without any advance warning. As there is no stated removal date/version
for "optparse" we have a chance to prepare and start
warnings/deprecations now and do the actual switch later.

This would also solve us from the headache of re-implementing OptionParser on
the basis of "ArgumentParser" and break the lock on other changes
(allow to continue with code cleanup and "low hanging fruit" and prepare 0.19b).

Back to the discussion:
>> We have 3 settings sources: components, config files, command line:
>> Are settings after updating from config files still "default"
>> settings?

> You could argue yes, the settings are the default for the particular
> environment. Happy to take better suggestions, though.

I don't have a compelling name suggsestion, unfortunately, but I am
increasingly convinced that settings updated from configuration files are
not "default settings". 

In programmatic use, one can easily differentiate "defaults" fallbacks,
set by Docutils vs. "custom values", set in the configuration file 
(with the defaults as fallback).
This is less clear in command line use (where config-file-customized
values that are fed to the OptionParser/ArgumentParser as "default=...").

>> I propose a more versatile variant that allows for
>> "settings_overrides" and customization of the config file names.

> I see the idea, but I would very strongly advise against -- this is
> meant to be a simple convinience function. If the more complex
> functionality is needed later, we can always either add it to the same
> function, or add a new one. 

I agree.
So, for now future-proof access to customized settings relies on
core.Publisher methods (cf. the patch for utils/__init__.py).

I'll attach a diff between the first commit in "pull/8" and my suggestion 
together with a review.





Attachments:

- [argparse-aa.review](https://sourceforge.net/p/docutils/bugs/_discuss/thread/c76f9a2618/45c6/3ffa/7fbd/0cd9/attachment/argparse-aa.review) (2.2 kB; application/octet-stream)
- [argparse-aa-gm.diff](https://sourceforge.net/p/docutils/bugs/_discuss/thread/c76f9a2618/45c6/3ffa/7fbd/0cd9/attachment/argparse-aa-gm.diff) (27.1 kB; text/x-patch)


---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Sun Feb 20, 2022 06:14 AM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Adam T. <aat...@ou...>]

> What is your intention with the GHA scripts?

This is the automatic tests I added to my fork on GitHub.

Due to a synchronisation error between master and the branch the "Add GHA scripts" commit was shown as part of the PR; I've fixed it now.

A

Re: [Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Guenter M. <mi...@us...>]

On 2022-02-20, Adam Turner via Docutils-develop wrote:

> I have pushed the changes from the review and Publisher.get_components
> into https://github.com/AA-Turner/docutils/pull/8 

What is your intention with the GHA scripts?

On my system, 

 cd repo/docutils
 flake8 .
 
works fine with the settings in `docutils/tox.ini`, so duplication as
`docutils.flake8` does not seem required.

Günter

Re: [Docutils-develop] Fix tests [r9010]

[Adam T. <aat...@ou...>]

Trunk is now passing on Windows & Unix [r9013], thanks Günter for committing the fix!

A

[Docutils-develop] Fix tests [r9010]

[Adam T. <aat...@ou...>]

Tests have been failing since r9010; the attached patch fixes one failure on Windows. The remaining two failures are because "docutils/test/data/config_syntax_error.txt" does not exist. I could a syntax error causing file if need be, I suspect more likely is it just wasn't committed accidentally.

A

-------------

>From 5afac77c70f72f4645962d4dc0a49d088e8488cd Mon Sep 17 00:00:00 2001
From: Adam Turner <908...@us...>
Date: Thu, 24 Feb 2022 01:22:23 +0000
Subject: [PATCH] Fix tests

---
 docutils/test/test_settings.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/docutils/test/test_settings.py b/docutils/test/test_settings.py
index 83213757e..c8cbbdd74 100755
--- a/docutils/test/test_settings.py
+++ b/docutils/test/test_settings.py
@@ -151,14 +151,14 @@ class ConfigFileTests(unittest.TestCase):

     def test_old(self):
         with self.assertWarnsRegex(FutureWarning,
-                                   'The "\[option\]" section is deprecated.'):
+                                   r'The "\[option\]" section is deprecated.'):
             self.files_settings('old')

     def test_syntax_error(self):
         with self.assertRaisesRegex(
                  ValueError,
                  'Error in config file ".*config_syntax_error.txt", '
-                 'section "\[general\]"'):
+                 r'section "\[general\]"'):
             self.files_settings('syntax_error')

     def test_one(self):
@@ -225,7 +225,9 @@ class ConfigEnvVarFileTests(ConfigFileTests):

     def test_old(self): pass # don't repreat this test

-    @unittest.skipUnless(os.name, 'posix')
+    @unittest.skipUnless(
+        os.name == 'posix',
+        'os.path.expanduser() no longer uses HOME on Windows (since 3.8)')
     def test_get_standard_config_files(self):
         os.environ['HOME'] = '/home/parrot'
         # os.path.expanduser() no longer uses HOME on Windows (since 3.8)
--
2.30.0.windows.2

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Adam T. <aa-...@us...>]

> I don't think we need to include the overview of docutils use(r)s in the
"Policies" document.

Removed

> Do you mean the provided output templates 

Yes, I updated to include the templates and stylesheets

> Why `__public__` and not `__all__`?

`__all__` has special meaning to the interpreter for `from module import *`. Perhaps it doesn't matter, but I didn't want to conflate the two usages.

> Do you want this ... in addition to [or] instead of SemVer

I'd prefer to adopt SemVer, but I dropped that from the proposal for later conversation. Would you be happy with adopting SemVer and updating the language in this section to "should" rather than "must"?

________

Fair enough, perhaps "DEP" is a reasonable way forwards.

A


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Wed Feb 16, 2022 02:32 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

> I've posted my redrafted suggestion at 
> https://github.com/AA-Turner/docutils/pull/14

Thanks for your proposal. Some points:

I don't think we need to include the overview of docutils use(r)s in the
"Policies" document.


>  Docutils public APIs are:
...
> * the Docutils writer templates

Do you mean the provided output templates [#]_?
How about provided style sheets [#]_ (unless marked as "provisional")?

.. [#] template.txt, default.tex, titlepage.tex, xelatex.tex
.. [#] html4css1.css, minimal.css, docutils.sty (LaTeX), styles.odt

> * behaviour and names of all items included in the ``__public__``
>   attribute of their parent objects.

Why `__public__` and not `__all__`?
Are there other packages using this variable name for this purpose?

> Deprecation periods  

Do you want this

- in addition to SemVer_ (i.e. also for changes in "major" releases,
  interdicting *all* incompatible changes in "minor" releases), or
- instead of SemVer_ (i.e. for "minor/medium" changes in "minor"
  releases, similar to :PEP:`387`)?
  
I don't think we need an additional restriction if we choose SemVer:
downstream users will be prepared for incompatible changes in a new
"major" version.

...

> I think the level of discussion we have on issues is reasonable to
> reach a good conclusion, and the work needed to write a "DEP" might
> just be better spent on the issue tracker.

The issue tracker is good for ongoing discussion, but not as reference

* for an informed final decision on the request/proposal,

* for the rationale and context in the "Policies" section on the public API,

* for the rationale and context when the policy/code will need adaption in,
  say, 10 years time. (I'll prefer to read a condensed/edited document
  instead of all of the comments in the issue tracker by then.)
  
Also, moving reStructuredText documents to a new home/host is far easier
than "saving" tracker issues.  


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Wed Feb 16, 2022 02:32 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] Request to become a Project Developer

[Adam T. <aat...@ou...>]

Hi,

As advised by Günter Milde off-list, I'm submitting this request to become a Project Developer.

The relevant text in the policies<https://docutils.sourceforge.io/docs/dev/policies.html#repository> notes:

> "If you would like to become a project developer, just ask!"

A brief overview of my (proposed) contributions to Docutils:

- Code cleanup for Python 3.7 & associated modernisations
- Removing 'prest' from the source tree
- Contributor to public API scope discussion
- Moving to argparse (in discussion)
- Enabling tests to be used with native unittest (in discussion)
- Declarative build system (in discussion)
- Using entry-points for front-end tools (in discussion)
- Adding a bcp-reference role (in discussion)

I have also contributed to the integrations between Docutils and Sphinx on the Sphinx bug-tracker.

Beyond this, I currently serve as a PEP editor for the Python project and in a new role as a "Python Documentation Team Member" - I would hope to use experiences from these roles to offer context to the Docutils project.

I believe in absence of any other process that the decision over accepting me (or not!) would fall to David Goodger as "BDFN", but I welcome comment from anyone on my suitability.

Thanks,
Adam

https://sourceforge.net/u/aa-turner/profile
https://github.com/AA-Turner/

[Docutils-develop] Substitute image options/path

[Urban A. A. <urb...@gm...>]

Is there an option to substitute image options/path in reStructuredText?

Something like:
```
.. image:: |picdir|/someImage.jpg
   :alt: SomeImage
   :width: |imagewidth|
   :target: |picdir|/someImage.jpg


.. |picdir| replace:: ./pic
.. |imagewidth| replace:: 500px
```