docutils-develop — For developer discussions of the implementation.

[Docutils-develop] [docutils:patches] #191 Update URLs

[Adam T. <aa-...@us...>]

---

** [patches:#191] Update URLs **

**Status:** open
**Group:** None
**Created:** Thu Jan 20, 2022 04:40 AM UTC by Adam  Turner
**Last Updated:** Thu Jan 20, 2022 04:40 AM UTC
**Owner:** nobody


Documentation / minor change. 

There are a lot of old URLs in the repo -- e.g. `docutils.sf.net`, `docutils.sourceforge.net` -- whilst these currently still work, we should update them all to the canonical `https://docutils.sourceforge.io` 

This patch does that, and also generally updates several other links to `https://` -- I tested each domain, so didn't update some old sourceforge domains that don't support `https://`. I also updated the `docutils.dtd` IDN in a single commit, so it can be dropped if wanted. 

I'd certainly reccomend applying the first 4 commits, and ideally the entire patch. 

https://github.com/AA-Turner/docutils/pull/10 // https://github.com/AA-Turner/docutils/pull/10.patch

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #441 Move from "optparse" to "argparse".

[Adam T. <aa-...@us...>]

> I do not see the benefit in the new helper functions outweighting the refactoring effort:

The functions are of most use to downstream developers, I'd argue. It provides a higher level abstraction which means you don't need to know about the internals of how `OptionParser` &c work. 

`get_settings_with_defaults` is tenuous, but I was trying to move to a point where the `OptionParser` and `ConfigParser` classes are only used in `docutils.frontend` -- providing a cleaner seperation of concerns and making any future refactorings easier.

> could be combined with the removal of the mod_python hack.

Done and pushed

> frontend.OptionParser is still present (is it also fully backwards compatible?)

I didn't remove it or change the name as many downstream consumers rely on it. It is fully backwards compatible, yes -- mostly due to the `settings_spec` <--> `arguments_spec` converters. It is deprecated, though (through the module level `__getattr__`).

> Would it be feasible to have a minimal change where only the Values, Option, and OptionParser classes were based on "argparse" instead of "optparse"

It would be very difficult. optparse makes fundamentally different assumptions than argparse, which took a while to sort out and convert. E.g. the parsing of config files, the "validator" behaviour -- there is a hack at the moment to intercept validators that downstream users have used and wrap them with an argparse `Action`. 

Broadly though, commit 2 ("Update to use argparse") can be made standalone if you want it to be -- I would be disapointed not to have the helper functions, but strictly they are not required. 

I think that is probably as minimal as it gets -- it was difficult enough to rewrite everything keeping 100% backwards compatability -- and I can't make any assumptions about which names are "safe" to remove -- e.g. I had to keep a no-op definition for `Option` as downstream users *might* use it.

A


---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Mon Jan 17, 2022 12:42 AM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Adam T. <aa-...@us...>]

> Docutils follows PEP 440

PEP 440 is more about how version numbers are parsed, it is not a versioning policy.

> in Docutils, the "major" part of the version identifier is incremented "if there is a major change in the design or API" while in Semantic Versioning the "major" part "MUST be incremented if any backwards incompatible changes are introduced to the public API".

I agree with your summary. The issue is that I think users, downstream developers will expect the latter -- having backwards incompatible changes in a minor version violates the principle of least surprise. 

> With your suggestion "adding the ability to have exceptions to the policy, with removal after one minor version" we are back to the current policy

I'd argue more a merger of the two than just back to the current policy. Pure semantic versioning allows you to make a backwards incompatible change with no warning or deprecation period, which is generally not how Python projects operate. What I proposed is the SemVer specification in terms of breaking changes (removals, changes in the DTD/default templates/etc), but codifying a deprecation period on top of that.

> In my understanding of the Docutils version policy, feature-complete means that all essential parts of the system are functional, stable, and documented.

I'd agree with you here - what I took issue to was the word "frozen" -- that e.g. wouldn't let us implement nested markup parsing after version 1.0.

_____

> ... keep it in sync with the code base and discoverability. This speaks in favour of defining the API in the code itself or in the docstrings

I agree with you. The problem is how best to do this. I would argue that type annotations and docstrings are orthoganal to the public API question -- for developers working on Docutils itself, both are helpful! (I would miss doctrings less than type hints, personally). 

I see your point on churn and names re undercores, but it is possible via module level `__getattr__` or similar to do a mass deprecation of names. On introducing underscores in some places might mean people assume everything else is public -- I would argue though that the people who are likely to do that, do it already -- as nothing has underscores.
This is one of the reasons I proposed the helpers in the `argparse` changeset by the way -- currently downstream developers have to assume a lot of knowledge of Docutils' internals -- I'd like to introduce higher level abstractions to make e.g. getting default settings or parsing a list of lines of reST easier -- ideally one function call. 

An alternative idea is an `@public` or `@private` no-op decorator that we could use to signal API status. This could then be used to generate the API list, at least in terms of code. 

```python 
def public(f):
    return f
```

> A massive name change would also complicate forensics with git blame.

Clearly Docutils still uses `svn`, but if this is a concern then `.git-blame-ignore-revs` can be used ([brief article](https://michaelheap.com/git-ignore-rev/))

--------

Concrete proposal RE public API in code: 

- Use `__all__` as you proposed for all global names (classes, module level functions, module level variables, etc)
- Adopt underscores for **new** private names
- *Consider* introducing underscores for existing names, through `__getattr__` or similar (with full backwards compatability and a deprecation period, and ideally helper functions to mean that downstream users don't need to use internal things as much)
- Don't use type annotations as an indication of status in the public API -- they are too helpful for that. Move to use them everywhere.
- Formalise the wording for docstrings for public vs private vs provisional (ideally this would be a single regex pattern).

I strongly think there is a benefit to moving more in line with the Python ecosystem as a whole in terms of how we name things and define the public API, but if this is entirely off the table, I think the proposals we've outlined would still be a net improvement.

A


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Mon Jan 17, 2022 12:16 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #88 Unify Docutils CLI tools into `docutils-cli`

[Adam T. <aa-...@us...>]

> Can you elaborate a bit on what would become a lot simpler?

Currently deep in Docutils' internals (everywhere that takes a `settings_spec` or uses `self.settings` sort of assumes working as a command line programme. However, a lot of usage (programmatic, through Sphinx or other methods) entirely use the default values for things. By moving to a single front end I would argue it is not only a cleaner user story, but it *might* enable refactoring to move the CLI usages of Docutils to a higher level. 

Currently we need to do awful things to subclass and patch either `optparse.OptionParser` or `argparse.ArgumentParser`. This is really unusual, and for developers coming from a more "normal" command line application, it can take a while to understand this part of the internals of Docutils.

I didn't go into detail intentionally so as not to spark a debate about these parts, but I do think (eventually) simplifying these interactions can lead to a cleaner codebase.

> the two-stage command line parsing

I don't think you can get away from this though without a combinatorial explosion of readers, writers, and parsers. Say we have two useful CLI readers (standalone/pep), three parsers (rst/recommonmark/myst), and 6 useful writers (html5/html4/latex/xetex/man/xml) that is 36 distinct front-end tools we should be providing. 

> a man page will always need to refer to external documentation

I will admit ignorance on how man pages work. `docutils-cli --writer xetex --help`, though, will always give the correct help output. This is also the version we should be promoting, not least as it works cross platform (if my patch with entrypoints is merged!).

> We need to care for "command line users" if their number is non-negligible 

Of course -- sorry if my post came across as callous in any way towards frontend tool users. I suppose what I don't want is to be in a situation where we are not making real improvements based on hypothetical situations. It might be useful to find ways of proxying for CLI usage -- bugs filed recently with us/redistributors, usages in public archives ( https://grep.app or similar ), etc.

> rather hard to find out how many non-Python projects uses "rst2html.py" in their Makefile or another form of build tool chain

True. However by the above methods we can get an estimate, surely? There are a lot of people who commit random things to GitHub / GitLab / whatever!

> finding out and approaching the right spot where to apply the change

This is why I proposed to go about it by emitting warnings during deprecation, before total removal. We also need to consider the support that this project offers -- if a downstream user has integrated Docutils into a complex tool chain and cannot maintain it, we shouldn't be responsible for that. 

> never underestimate the number of users/project managers that don't read the changelog ... yet depend on a stable Docutils for a stable system.

Fair enough -- though perhaps another route we could go down in the deprecation notices are to say "pin version XX". There is no best solution here -- all change will break someone's workflow ([XKCD 1172!](https://xkcd.com/1172/)), but we should be working to make the upgrade path as easy as possible.

> buildhtml

Ahh, I was under the impression that `buildhtml` was an internal tool for building the website. Would it be reasonable to formally retire it from public use, and reccommend Sphinx as an alternative? 

>  use `docutils` as front-end command // `python -m docutils`

+1

> Ease of discovery is important. TAB completion is a powerfull means here

Did you see my suggestion on using custom shell autocompletion functions? I believe that this would allow for tab completion with the reader/parser/writer flags.

> rst2 is established as the start of Docutils' front-end names for conversion from reStructuredText to something. I would like to keep this prefix as "ours". 

If we use what I propsed in one of my changesets to reimplement the `rst2` commands in terms of `docutils-cli`, it would be entirely possible to deprecate the `rst2` commands but just keep them forever. This would also mean that the simplifications I proposed at the top of this message wouldn't be blocked (I think).

-----

Concrete proposal:

- Promote `docutils` or `python -m docutils` where we currently reference `rst2`
- Reimplement the `rst2*` commands in terms of `docutils-cli` 
- Try to implement <TAB\> autocompletion for `docutils-cli`
- Use entrypoints for everything (but also keep `.py` aliases for a while)
- Deprecate `rst2*` commands, but with no removal date


A


---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 11:57 PM UTC
**Owner:** nobody


As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #190 Move to native `unittest`

[Adam T. <aa-...@us...>]

I have done all the work to get this working at https://github.com/AA-Turner/docutils/pull/9 with no noticable impact on time. However this is a massive changeset, so I need to think about how to break this into logical chunks/components.

A


---

** [patches:#190] Move to native `unittest`**

**Status:** open
**Group:** None
**Created:** Thu Jan 20, 2022 01:10 AM UTC by Adam  Turner
**Last Updated:** Thu Jan 20, 2022 01:10 AM UTC
**Owner:** nobody


From FR81 https://sourceforge.net/p/docutils/feature-requests/81/#fc55/3d8d by Günter Milde

> I had a quick look and the work looks promising, so yes, I would like to
> give it a try.
> 
> What is the effect on the test time and on the output if there are, e.g.,
> differences between output and expected in functional tests?
> 
> To avoid wasted work (and the sunken-cost fallacy), I propose to split
> the project into smaller changesets (of 1 to ca. 7 commits) which we can
> discuss and adapt before pushing to origin/master and then move to the
> next step.
> I expect some rounds of discussion per step, so there is no
> need to invest heavily in polishing in the first round.
> A link to a patch that I can apply with git am -3 to a local branch
> would be fine.
> The discussion could move to docutils-develop or a new ticket,
> whatever you prefer.

This is the proposed tracking issue for moving to native unittest

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #190 Move to native `unittest`

[Adam T. <aa-...@us...>]

---

** [patches:#190] Move to native `unittest`**

**Status:** open
**Group:** None
**Created:** Thu Jan 20, 2022 01:10 AM UTC by Adam  Turner
**Last Updated:** Thu Jan 20, 2022 01:10 AM UTC
**Owner:** nobody


From FR81 https://sourceforge.net/p/docutils/feature-requests/81/#fc55/3d8d by Günter Milde

> I had a quick look and the work looks promising, so yes, I would like to
> give it a try.
> 
> What is the effect on the test time and on the output if there are, e.g.,
> differences between output and expected in functional tests?
> 
> To avoid wasted work (and the sunken-cost fallacy), I propose to split
> the project into smaller changesets (of 1 to ca. 7 commits) which we can
> discuss and adapt before pushing to origin/master and then move to the
> next step.
> I expect some rounds of discussion per step, so there is no
> need to invest heavily in polishing in the first round.
> A link to a patch that I can apply with git am -3 to a local branch
> would be fine.
> The discussion could move to docutils-develop or a new ticket,
> whatever you prefer.

This is the proposed tracking issue for moving to native unittest

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] Re: #441 Move from "optparse" to "argparse".

[Günter M. <mi...@us...>]

Thank you for the reworked patch set!

> It will likely be easiest to review commit-by-commit.

Lets start with 1/4

I do not see the benefit in the new helper functions outweighting the
refactoring effort:

Usage (after all 4 commits): 

=========================================  ===  ===== =========
new function                               lib  tests dev-tools
=========================================  ===  ===== =========
frontend.read_config_files                  2
frontend.config_files_settings              1     1
frontend.get_default_settings               1     7        1
frontend.get_settings_with_defaults         1
frontend.parse_args                         1         
Publisher.config_section_to_settings_spec   3
=========================================  ===  ===== =========


Making OptionParser.get_standard_config_files() a class method could be
combined with the removal of the mod_python hack. 
The commit message or should note the reason 
(mod_python beeing no longer developed and too old)
with a link to, e.g., https://en.wikipedia.org/wiki/Mod_python.



I see that after all 4 commits, frontend.OptionParser is still present
(is it also fully backwards compatible?)

Would it be feasible to have a minimal change where only the Values,
Option, and OptionParser classes were based on "argparse" instead of
"optparse"

Then we can start moving to frontend.ArgumentParser and using "argparse"
features step by step.


---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Mon Jan 17, 2022 12:42 AM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] #188 Remove lingering references to Python 2

[Günter M. <mi...@us...>]

- **status**: open --> closed-fixed



---

** [patches:#188] Remove lingering references to Python 2**

**Status:** closed-fixed
**Group:** None
**Created:** Wed Jan 05, 2022 08:30 PM UTC by Adam  Turner
**Last Updated:** Sat Jan 15, 2022 08:49 PM UTC
**Owner:** nobody


@milde

The first commit removes textual references, the remaining do some clean-up in `nodes` and `error_reporting` -- I don't know if these are helpful. I think at least the first commit should be applied, though.

https://github.com/AA-Turner/docutils/pull/4 // https://github.com/AA-Turner/docutils/pull/4.patch


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

On 2022-01-16, Adam Turner wrote:

> The only text I can find on library versioning is at
> https://docutils.sourceforge.io/docs/dev/policies.html#version-identification,

Yes, this is the Docutils version policy (the implementation details
could eventually move to an API description).

...

> Firstly, I propose adopting a formal versioning "system"

Docutils follows [PEP 440](https://www.python.org/dev/peps/pep-0440/)

...

> Semantic versioning is nearly identical to the current versioning
> policy

the difference is, that in Docutils, changes to the "major" part of the
version identifier is incremented "if there is a **major** change in the
design or API" while in Semantic Versioning the "major" part "MUST be
incremented if **any** backwards incompatible changes are introduced to
the public API".

With your suggestion

> adding the ability to have exceptions to the policy, with removal after
> one minor version.

we are back to the current policy
(which I would not dare to label Semantic Versioning).

...

> What this does change is that the API is never considered complete, as
> in the original phrasing.

In my understanding of the Docutils version policy, *feature-complete* means
that all essential parts of the system are functional, stable, and documented.
Docutils reached this state long ago, except for the documentation and API
specification.

> The 1.0 release then does not need to be a "big deal",

Releasing 1.0 will not be a breaking change, rather a change in commitment.

******

> I also suggest enumerating all modules, classes, and functions that
> form the public API.
> ...
> At the very least I would suggest, `docutils.nodes`, the
> reader/writer/parser aliases, `docutils.core`, and the front end tools.

The abstract base classes for reader, writer, parser, and transforms come to mind.
Also `docutils.__init__` and the "plug-in API" for the components.
However, these modules and classes also contain auxiliary objects
(that should not become part of the "core API") and ambiguous
("non-core") cases (that may be useful for clients or may be used by
existing applications).

For the Docutils version number, the document model (docutils.dtd, doctree.txt)
and the rST specification are equally important.

> (I'm happy to write out the full list if you give me modules/etc that
> should be part of the public API).

For a first draft, the abovementioned objects may be a good starting point.
The modules and classes intended as part of the public API come with comprehensive docstrings
which are a good guide to differentiate core objects from auxiliary objects.

The [api/](https://docutils.sourceforge.io/docs/#api-api-reference-material-for-client-developers)
directory lists the existing API Reference Material.

The PEP template may serve as a scaffolding for the document structure
(I can mail or upload an adapted version).

Problems with an external list are the need to keep it in sync
with the code base and discoverability.
This speaks in favour of defining the API in the code itself or in the
docstrings. It should be possible to do a valid guess whether an object
is public by looking at the source code or using `pydoc`.

> It would also be useful to identify what parts of Docutils large
> downstream consumers use, and either create higher level abstractions
> or mark those as public API. (I would suggest Sphinx and MyST-parser).

Yes, the final document should be a collaborative effort and we may
improve/clean the code base in the process.
OTOH, Sphinx is so closely intertwined to Docutils that it, IMO,
deserves a special handling ensuring synchronised changes.
This may be extended to other projects that use the "non-core API" and are
ready to follow the development, test, and report back.

> It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles)
> to mark names as private by prefixing them with an underscore.

> I suggest adopting this, as it gives strong guidance to downstream
> library authors what Docutils considers private and public.

On the other hand, starting to use leading underscores would give the
impression that all other objects are public.
We would need to change a lot of names in one go and this would
break all applications that use "semi-public" names
(were we would change the name only as a precaution).
A massive name change would also complicate forensics with `git blame`.

> Making a private name public is far easier than going through a
> deprecation cycle for the reverse.

This is why I propose a different approach: instead of
"public unless indicated otherwise", the rule is
**all core objects MUST be type-annotated AND come with a docstring**:

* Objects without type annotations are not part of the core API
  (cf. [feature-request:#87]).

* Objects with comment instead of a docstring are not part of the core API.

* Objects where the docstring says "private" are not part of the core API.

* Objects where the docstring says "provisional" are exempt from the
  backwards compatibility promise.

In addition, `__all__` should list all public top-level objects of a module.

This leaves the ambiguous cases open for easy upgrading to public state.


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Mon Jan 17, 2022 12:16 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

>> docutils is a user application not a programmer tool

Docutils was designed from the start to be both.
Cf. [PEP 258](http://www.python.org/peps/pep-0258.html)
and https://docutils.sourceforge.io/docs/index.html#docutils-stakeholders

> ... I would outline three main use-cases for Docutils:

> 1. as an application, through one of the front-end tools
> 2. in direct dependencies, such as Sphinx or MyST-parser
> 3. when building plugins for tools in (2)

4. when building plugins for Docutils (e.g. myst-docutils and pycmark)

For backwards compatibility, we also need to think about indirect use:

5. as author of documents that will be processed by Docutils, Sphinx or
   similar
6. as maintainer of a project that uses Doctuils in its tool chain.

An important part of Docutils are also the specifications of the document
model (Docutils Document Tree) and reStructured Text!


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Mon Jan 17, 2022 12:16 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #88 Unify Docutils CLI tools into `docutils-cli`

[Günter M. <mi...@us...>]

While I am in favour of revising and updating Docutils' command line
entry points, I don't think we should drop the number down to one.

> I'll briefly re-outline my argument to (eventually) drop the `rst*`
> front-end tools, and only export `docutils-cli` (or `python -m
> docutils`).

>> I think a single front-end tool significantly simplifies a lot of
>> things

Can you elaborate a bit on what would become a lot simpler?

>> the docutils-cli wrapper is not complex, which gives it
>> significant points in favour in my book.

The generic front-end is one order of magnitude more complex because of
the two-stage command line parsing with the set of valid tags depending
on the components selected.
Even help output depends on the "component" tags. Due to the open nature
(allowing for plug-in components), a man page will always need to refer
to external documentation, while, e.g,  `man rst2html` lists all available
command line options (at least on Debian).

>> Most usage of Docutils today is programmatic, and not via the command
>> line tools

We need to care for "command line users" if their number is
non-negligible -- independent of the number of users depending on the
programmatic interface.

The number of users/projects using Docutils via the command line interface
cannot be estimated by looking at Python projects.
Unfortunately, it is rather hard to find out how many non-Python projects
uses "rst2html.py" in their `Makefile` or another form of build tool chain.

The first answer to [Explain Python entry points?](https://stackoverflow.com/questions/774824/explain-python-entry-points)
even cites Doctils as

  ... a great example of entry-point use: it will install something like a
  half-dozen useful commands for converting Python documentation to other
  formats.

(even if Docutils currently does not use the "console-scripts mechanism" to
provide cli entry points).

...

>>  We cannot know how many people would be affected with local random
>>  scripts, but it is a two-second change.

While the actual re-typing (or drag-and-drop) of the command may be that
fast, this is not the case for the complete task of finding out and
approaching the right spot where to apply the change in a complex build
chain.

>> Many users will also run with old or pinned versions of Docutils, and
>> part of updating is seeing the changelog.

A hard learned lesson from Docutils releases is to never underestimate
the number of users/project managers that don't read the changelog (nor
the announcements in the RELEASE-NOTES) yet depend on a stable Docutils
for a stable system.

********

I am pro change for instances where the current
[naming](https://clig.dev/#naming) is unfortunate or may stand in the way.

`buildhtml.py` is too generic, it may stand in the way.
Debian calls it `rst-buildhtml`. I could imagine `docutils-buildhtml` or
leaving it in the tools for individual installing.

`docutils-cli.py` is too long. This name was selected because a naming
the file for the generic front end tool "docutils.py" is misleading.
With "entry points" it is possible to use `docutils` as front-end command
without the need for a file "docutils.py".

`python3 -m docutils` currently results in the error:
`'docutils' is a package and cannot be directly executed`
It could be made more helpful, we know, a user typing `python -m ...` wants
to execute a command line tool (or just wants to know more about docutils).

`rst2` is established as the start of Docutils' front-end names for
conversion from reStructuredText to something. I would like to keep this
prefix as "ours". (After all, Docutils is the reference implementation of the
rST format.) 

Ease of discovery is important. TAB completion is a powerfull means here.
Additional parser or readers may add their own entry points, cf.
https://github.com/executablebooks/MyST-Parser/issues/347#issuecomment-1003717830


Rarely used and diagnostic tools may not need automatic installation into
the binary PATH. Here, it may help to diagnose which tools are installed by
`pip docutils` vs. OS-specific package managers.

Debian installs the following 13:

rst2html
rst2html4
rst2html5
rst2latex
rst2man
rst2odt
rst2odt_prepstyles
rst2pseudoxml
rst2s5
rst2xetex
rst2xml
rst-buildhtml
rstpep2html

Dropping the `.py` from `rst2*.py` commands may be considered.

+1 shorter and more command-like names
-1 backwards incompatible, an unknown number of users need to change their scripts.


---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 11:57 PM UTC
**Owner:** nobody


As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] Re: #186 Modernise packaging

[Günter M. <mi...@us...>]

> What does `python3 -c "import locale;
> print(locale.getlocale())"` print on your computer?

The following test script

    import locale, time
    print(locale.getlocale())
    print('before setlocale()', time.strftime('%c'))
    locale.setlocale(locale.LC_ALL, '')
    print('after setlocale()', time.strftime('%c'))

prints

    ('de_DE', 'UTF-8')
    before setlocale() Mon Jan 17 09:29:41 2022
    after setlocale() Mo 17 Jan 2022 09:29:41 CET

IMO, it is an intended Python feature that the `locale` module can be
imported without automatic "opt-in" to the localisation services.


---

** [patches:#186] Modernise packaging**

**Status:** open
**Group:** None
**Created:** Fri Dec 31, 2021 03:16 AM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 11:49 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-Use-flit-and-pyproject.toml.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0001-Use-flit-and-pyproject.toml.patch) (12.2 kB; application/octet-stream)
- [0002-Use-entry-points.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0002-Use-entry-points.patch) (20.7 kB; application/octet-stream)
- [0003-update-docs-etc-after-packaging-changes.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0003-update-docs-etc-after-packaging-changes.patch) (49.3 kB; application/octet-stream)


Hi,

I had a go at modernising the packaging stack. `setup.py` based invocations have been deprecated (https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html), and setuptools may remove them in the future. 

This takes the opportunity to move to a PEP 621 based declarative config, and also fixes a longstanding TODO item about providing script wappers for the frontend tools on windows, by migrating them to entry points.

I've updated install and development docs with the new guidance, and updated references to the frontend tools to remove `.py`, given they are now installed as proper scripts.

Hope this is appreciated -- happy to make revisions etc to help getting this merged.

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:bugs] #441 Move from "optparse" to "argparse".

[Adam T. <aa-...@us...>]

Hi, as promised please find the patch at https://github.com/AA-Turner/docutils/pull/8 // https://github.com/AA-Turner/docutils/pull/8.patch

It will likely be easiest to review commit-by-commit.

Note as discussed earlier, all code expecting a `settings_spec` tuple will can get one (see https://github.com/AA-Turner/docutils/pull/8/files#r785578670 ).

Would appreciate any feedback or reviews!

A


---

** [bugs:#441] Move from "optparse" to "argparse".**

**Status:** open
**Created:** Thu Jan 06, 2022 03:02 PM UTC by Günter Milde
**Last Updated:** Sat Jan 15, 2022 04:02 AM UTC
**Owner:** Günter Milde


The optparse documentation says:
> Deprecated since version 3.2: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

We are currently suppressing related deprecation warnings in the test suite.
After raising the Python dependency to >=3.7, now may be the right time to make the move.


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/bugs/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/bugs/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #89 Public API, versioning, and deprecation

[Adam T. <aa-...@us...>]

> docutils is a user application not a programmer tool

I think with the advent of Sphinx this is no longer the case *de facto*. I would outline three main use-cases for Docutils:

1. as an application, through one of the front-end tools
2. in direct dependencies, such as Sphinx or MyST-parser
3. when building plugins for tools in (2)

A cursory search for usage of docutils as a library (`(import|from) docutils`) finds some 2,600 results (https://grep.app/search?q=%28import%7Cfrom%29%20docutils®exp=true&filter[lang][0]=Python). 

It would be an option to entirely deprecate all uses of `docutils` as a library (this is what pip did for pip 10), but I would strongly advise against this -- I think it is far better to have a clear idea of what we view the public API to be (both in terms of use as an application and as a library), and go from there. This might involve creating high level functions that mean downstream users don't have to reach into the internals of Docutils as often -- I have had to do this myself, and would prefer not to (part of the optparse -> argparse work I'm doing introduces some such abstractions). 

Perhaps @chrisjsewell or @tk0miya might have views from MyST and Sphinx?

A


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 04:23 PM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #88 Unify Docutils CLI tools into `docutils-cli`

[Adam T. <aa-...@us...>]

> of course new readers wont show up for rstTABTAB

I think this partly speaks to the issue -- the tab completion functionality only works "by accident", and doesn't support readers/parsers. 

It seems it might be possible to add custom bash autocomplete rules (https://caliban.org/bash/#completion) -- would this be an acceptable workaround?

A


---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 04:16 PM UTC
**Owner:** nobody


As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:patches] Re: #186 Modernise packaging

[Adam T. <aa-...@us...>]

That's really interesting. What does `python3 -c "import locale; print(locale.getlocale())"` print on your computer? 

It seemed from the various things that locale issues had been largely improved, it would be sad if they haven't

A


---

** [patches:#186] Modernise packaging**

**Status:** open
**Group:** None
**Created:** Fri Dec 31, 2021 03:16 AM UTC by Adam  Turner
**Last Updated:** Sat Jan 15, 2022 09:57 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-Use-flit-and-pyproject.toml.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0001-Use-flit-and-pyproject.toml.patch) (12.2 kB; application/octet-stream)
- [0002-Use-entry-points.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0002-Use-entry-points.patch) (20.7 kB; application/octet-stream)
- [0003-update-docs-etc-after-packaging-changes.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0003-update-docs-etc-after-packaging-changes.patch) (49.3 kB; application/octet-stream)


Hi,

I had a go at modernising the packaging stack. `setup.py` based invocations have been deprecated (https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html), and setuptools may remove them in the future. 

This takes the opportunity to move to a PEP 621 based declarative config, and also fixes a longstanding TODO item about providing script wappers for the frontend tools on windows, by migrating them to entry points.

I've updated install and development docs with the new guidance, and updated references to the frontend tools to remove `.py`, given they are now installed as proper scripts.

Hope this is appreciated -- happy to make revisions etc to help getting this merged.

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

Re: [Docutils-develop] [docutils:bugs] #441 Move from "optparse" to "argparse".

[Guenter M. <mi...@us...>]

On 2022-01-15, Adam Turner via Docutils-develop wrote:

>> We need to keep compatibility the current "settings spec" specification

> Yes of course. I should have maybe explained a little -- I added a
> transparent conversion function from `arguments_spec` to
> `settings_spec` and vice versa. 

Yes, this is important. My point was that we cannot remove this little gem
even if all settings specs in the Docutils code were changed to use a dict.

Thanks for the clarification and thanks for your work.
Günter

[Docutils-develop] [docutils:patches] Re: #186 Modernise packaging

[Günter M. <mi...@us...>]

On 2022-01-15, Adam Turner wrote:

>> The current locale docs still say

> I went through the history, and it appears that snippet has remained
> almost unchanged for ~24 years (
> https://github.com/python/cpython/blob/bc12f78bb3d774260444c76bc22507e9a5844bde/Doc/lib/liblocale.tex#L31-L36
> ). 

This is most probably due to this part of the code beeing stable.

The Docutils front-end tools and Docutils have locale-dependent
behaviour:

Minimal example:
~~~
.. |heute| date:: %A

Heute ist |heute|.
~~~

With my system set to German locale (de_DE.UTF-8), rst2html5 writes
 
    Heute ist Sonntag.
    
Without the "setlocale()" call, this becomes

    Heute ist Sunday.
    
which is not what I want.

This is a backwards incompatible change. 
I propose to keep the locale-dependent behaviour also in a future
front-end-tool setup.


---

** [patches:#186] Modernise packaging**

**Status:** open
**Group:** None
**Created:** Fri Dec 31, 2021 03:16 AM UTC by Adam  Turner
**Last Updated:** Sat Jan 15, 2022 09:57 PM UTC
**Owner:** nobody
**Attachments:**

- [0001-Use-flit-and-pyproject.toml.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0001-Use-flit-and-pyproject.toml.patch) (12.2 kB; application/octet-stream)
- [0002-Use-entry-points.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0002-Use-entry-points.patch) (20.7 kB; application/octet-stream)
- [0003-update-docs-etc-after-packaging-changes.patch](https://sourceforge.net/p/docutils/patches/186/attachment/0003-update-docs-etc-after-packaging-changes.patch) (49.3 kB; application/octet-stream)


Hi,

I had a go at modernising the packaging stack. `setup.py` based invocations have been deprecated (https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html), and setuptools may remove them in the future. 

This takes the opportunity to move to a PEP 621 based declarative config, and also fixes a longstanding TODO item about providing script wappers for the frontend tools on windows, by migrating them to entry points.

I've updated install and development docs with the new guidance, and updated references to the frontend tools to remove `.py`, given they are now installed as proper scripts.

Hope this is appreciated -- happy to make revisions etc to help getting this merged.

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/patches/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/patches/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #58 Migration Docutils from SourceForge to Github

[Günter M. <mi...@us...>]

The issue not forgotten. You may have a look at 
https://docutils.sourceforge.io/docs/dev/todo.html#repository
for some thoughts (and at the version history at
https://sourceforge.net/p/docutils/code/HEAD/log/?path=/trunk/docutils/docs/dev/todo.txt
to monitor the (admittedly slow) progress).


---

** [feature-requests:#58] Migration Docutils from SourceForge to Github**

**Status:** open
**Group:** Default
**Created:** Fri Feb 16, 2018 03:23 PM UTC by Yves Chevallier
**Last Updated:** Sun Jan 16, 2022 04:05 PM UTC
**Owner:** nobody


Sourceforge is not really user friendly to report issues, propose pull-request and contribute to the project. I would like to know if it is possible to migrate Docutils to GitHub. 


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #89 Public API, versioning, and deprecation

[engelbert g. <gr...@us...>]

I am in favor of semver, i think there are too much users relying on certain things

docutils is a user application not a programmer tool. means 

* any script.py
*  any css/dom-construct
*   any tex-macro

might be API


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 01:39 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #88 Unify Docutils CLI tools into `docutils-cli`

[engelbert g. <gr...@us...>]

working on the commandline means tipping less by using completion

if i type rstTABTAB the list of all writers shows up 
if there is only docutils-cli i have to read the documentation.
if there happens to be a new writer with rst.... i will be notified by the completion result,
if there is only docutils-cli i have to read the documentation.

of course new readers wont show up for rstTABTAB


---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sat Jan 15, 2022 10:11 PM UTC
**Owner:** nobody


As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #58 Migration Docutils from SourceForge to Github

[engelbert g. <gr...@us...>]

hei

Adam cloned the live-clone on github and started contributing .

Günter is busy being cautious not to break docutils compatibility and still incorporate changes.

I am slow 



---

** [feature-requests:#58] Migration Docutils from SourceForge to Github**

**Status:** open
**Group:** Default
**Created:** Fri Feb 16, 2018 03:23 PM UTC by Yves Chevallier
**Last Updated:** Sun Jan 16, 2022 06:32 AM UTC
**Owner:** nobody


Sourceforge is not really user friendly to report issues, propose pull-request and contribute to the project. I would like to know if it is possible to migrate Docutils to GitHub. 


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] Re: #58 Migration Docutils from SourceForge to Github

[Pradyun G. <pr...@us...>]

Hiya folks! Any updates on this? It's been two months without activity on this.


---

** [feature-requests:#58] Migration Docutils from SourceForge to Github**

**Status:** open
**Group:** Default
**Created:** Fri Feb 16, 2018 03:23 PM UTC by Yves Chevallier
**Last Updated:** Mon Nov 15, 2021 10:50 PM UTC
**Owner:** nobody


Sourceforge is not really user friendly to report issues, propose pull-request and contribute to the project. I would like to know if it is possible to migrate Docutils to GitHub. 


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #89 Public API, versioning, and deprecation

[Adam T. <aa-...@us...>]

---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 01:39 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Docutils-develop] [docutils:feature-requests] #88 Unify Docutils CLI tools into `docutils-cli`

[Adam T. <aa-...@us...>]

---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sat Jan 15, 2022 10:11 PM UTC
**Owner:** nobody


As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A


---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.