[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

On 2022-01-16, Adam Turner wrote:

> The only text I can find on library versioning is at
> https://docutils.sourceforge.io/docs/dev/policies.html#version-identification,

Yes, this is the Docutils version policy (the implementation details
could eventually move to an API description).

...

> Firstly, I propose adopting a formal versioning "system"

Docutils follows [PEP 440](https://www.python.org/dev/peps/pep-0440/)

...

> Semantic versioning is nearly identical to the current versioning
> policy

the difference is, that in Docutils, changes to the "major" part of the
version identifier is incremented "if there is a **major** change in the
design or API" while in Semantic Versioning the "major" part "MUST be
incremented if **any** backwards incompatible changes are introduced to
the public API".

With your suggestion

> adding the ability to have exceptions to the policy, with removal after
> one minor version.

we are back to the current policy
(which I would not dare to label Semantic Versioning).

...

> What this does change is that the API is never considered complete, as
> in the original phrasing.

In my understanding of the Docutils version policy, *feature-complete* means
that all essential parts of the system are functional, stable, and documented.
Docutils reached this state long ago, except for the documentation and API
specification.

> The 1.0 release then does not need to be a "big deal",

Releasing 1.0 will not be a breaking change, rather a change in commitment.

******

> I also suggest enumerating all modules, classes, and functions that
> form the public API.
> ...
> At the very least I would suggest, `docutils.nodes`, the
> reader/writer/parser aliases, `docutils.core`, and the front end tools.

The abstract base classes for reader, writer, parser, and transforms come to mind.
Also `docutils.__init__` and the "plug-in API" for the components.
However, these modules and classes also contain auxiliary objects
(that should not become part of the "core API") and ambiguous
("non-core") cases (that may be useful for clients or may be used by
existing applications).

For the Docutils version number, the document model (docutils.dtd, doctree.txt)
and the rST specification are equally important.

> (I'm happy to write out the full list if you give me modules/etc that
> should be part of the public API).

For a first draft, the abovementioned objects may be a good starting point.
The modules and classes intended as part of the public API come with comprehensive docstrings
which are a good guide to differentiate core objects from auxiliary objects.

The [api/](https://docutils.sourceforge.io/docs/#api-api-reference-material-for-client-developers)
directory lists the existing API Reference Material.

The PEP template may serve as a scaffolding for the document structure
(I can mail or upload an adapted version).

Problems with an external list are the need to keep it in sync
with the code base and discoverability.
This speaks in favour of defining the API in the code itself or in the
docstrings. It should be possible to do a valid guess whether an object
is public by looking at the source code or using `pydoc`.

> It would also be useful to identify what parts of Docutils large
> downstream consumers use, and either create higher level abstractions
> or mark those as public API. (I would suggest Sphinx and MyST-parser).

Yes, the final document should be a collaborative effort and we may
improve/clean the code base in the process.
OTOH, Sphinx is so closely intertwined to Docutils that it, IMO,
deserves a special handling ensuring synchronised changes.
This may be extended to other projects that use the "non-core API" and are
ready to follow the development, test, and report back.

> It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles)
> to mark names as private by prefixing them with an underscore.

> I suggest adopting this, as it gives strong guidance to downstream
> library authors what Docutils considers private and public.

On the other hand, starting to use leading underscores would give the
impression that all other objects are public.
We would need to change a lot of names in one go and this would
break all applications that use "semi-public" names
(were we would change the name only as a precaution).
A massive name change would also complicate forensics with `git blame`.

> Making a private name public is far easier than going through a
> deprecation cycle for the reverse.

This is why I propose a different approach: instead of
"public unless indicated otherwise", the rule is
**all core objects MUST be type-annotated AND come with a docstring**:

* Objects without type annotations are not part of the core API
  (cf. [feature-request:#87]).

* Objects with comment instead of a docstring are not part of the core API.

* Objects where the docstring says "private" are not part of the core API.

* Objects where the docstring says "provisional" are exempt from the
  backwards compatibility promise.

In addition, `__all__` should list all public top-level objects of a module.

This leaves the ambiguous cases open for easy upgrading to public state.


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Mon Jan 17, 2022 12:16 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.