[Docutils-develop] [docutils:feature-requests] Re: #89 Public API, versioning, and deprecation

[Günter M. <mi...@us...>]

>> in Docutils, the "major" part of the version identifier is incremented
>> "if there is a major change in the design or API" while in Semantic
>> Versioning the "major" part "MUST be incremented if any backwards
>> incompatible changes are introduced to the public API".

> I agree with your summary. The issue is that I think users, downstream
> developers will expect the latter -- having backwards incompatible
> changes in a minor version violates the principle of least surprise. 

> What I proposed is the
> SemVer specification in terms of breaking changes (removals, changes in
> the DTD/default templates/etc), but codifying a deprecation period on
> top of that.

> _____

> I would argue that type annotations and docstrings are orthogonal to
> the public API question -- for developers working on Docutils itself,
> both are helpful!

Docstrings are (at least indirectly) related to public API in PEP 8:

> Documented interfaces are considered public, unless the documentation
> explicitly declares them to be provisional or internal interfaces
> exempt from the usual backwards compatibility guarantees.
> All undocumented interfaces should be assumed to be internal.

----

> Concrete proposal RE public API in code: 

> - Use `__all__` as you proposed for all global names (classes, module level functions, module level variables, etc)

> - Adopt underscores for **new** private names
> - *Consider* introducing underscores for existing names, through `__getattr__` or similar (with full backwards compatability and a deprecation period, and ideally helper functions to mean that downstream users don't need to use internal things as much)

I'd rather not.

> - Don't use type annotations as an indication of status in the public API -- they are too helpful for that. Move to use them everywhere.

I don't insist on type annotations as indicator.
However, adding type annotations to existing code should concentrate on the
public API.

> - Formalise the wording for docstrings for public vs private vs provisional (ideally this would be a single regex pattern).

Agreed.


I'll attach an draft document that tries to sum up what we have reached so
far (work in progress).


---

** [feature-requests:#89] Public API, versioning, and deprecation**

**Status:** open
**Group:** Default
**Created:** Sun Jan 16, 2022 01:39 AM UTC by Adam  Turner
**Last Updated:** Thu Jan 20, 2022 02:27 AM UTC
**Owner:** nobody


As requested by @milde in https://sourceforge.net/p/docutils/bugs/441/#7043/cdb8/8742/6e7f I'm opening this issue to allow for discussion on Docutils' public API, versioning policy, and deprecation.

This also relates to FR 87 on type annotations. 

From Günter, 

> The idea is to reconcile the reality (Docutils is used as if it were mature) and the version number (<1) once the API sufficiently defined a deprecation policy is agreed.

The only text I can find on library versioning is at https://docutils.sourceforge.io/docs/dev/policies.html#version-identification, excerpt below:

> **Major releases** (x.0, e.g. 1.0) will be rare, and will
represent major changes in API, functionality, or commitment.  The
major number will be bumped to 1 when the project is
feature-complete, and may be incremented later if there is a major
change in the design or API.  When Docutils reaches version 1.0,
the major APIs will be considered frozen.
For details, see the `backwards compatibility policy`_.

> Releases that change the minor number (x.y, e.g. 0.5) will be
**feature releases**; new features from the `Docutils core`_ will
be included.

> Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
**bug-fix releases**.  No new features will be introduced in these
releases; only bug fixes will be included.

The proposed backwards compatability policy reads:

> Docutils' backwards compatibility policy follows the rules for Python in
PEP 387. ... The scope of the public API is laid out at the start of the backwards
  compatibility rules

I propose two modifications to the policies, which will make future changes to Docutils easier to review and reason about, for project members as well as outside contributors. 

Firstly, I propose adopting a formal versioning "system" such as Sematic Versioning ([SemVer](https://semver.org/)) or Calendar Versioning ([CalVer](https://calver.org/)). 

Semantic versioning is nearly identical to the current versioning policy, and has the benefit of being a known quantity, reducing misunderstandings. A potential phrasing would be:

**"Docutils follows SemVer. All changes must also follow the backwards compatability policy."**

What this does change is that the API is never considered complete, as in the original phrasing. Docutils [isn't slated for inclusion](https://www.python.org/dev/peps/pep-0258/#rejection-notice) in the standard library any more, and there will always be potential improvements and changes -- new parsers, new node types, changes in the HTML specification, et cetera. 

The 1.0 release then does not need to be a "big deal", and future breaking changes can go to 2.0, 3.0, etc -- setuptools is now on `60.5.0`!

Semantic versioning does have some drawbacks (https://snarky.ca/why-i-dont-like-semver/), so projects like pip have adopted calendar based versioning -- the major version is the year (22), and the minor version is either the current month or an increasing number.  This relies on strong documentation and changelogs, so that when users upgrade it is obvious what changed (the idea being that every change breaks someone, so there is no contract that version numbers within a certain range mean no breakage). Luckily, Docutils has a good culture around changelogs and histories etc, so this would be easy to adopt.

______


I also suggest enumerating all modules, classes, and functions that form the public API. The current backwards compatability policy references PEP 387, which is specifically for the Python project itself. Checking through the documentation on every change to identify if a name is public or private is error prone (we are all human, and can miss things with no malintent) and time consuming.

At the very least I would suggest, `docutils.nodes`, the reader/writer/parser aliases, `docutils.core`,  and the front end tools. (I'm happy to write out the full list if you give me modules/etc that should be part of the public API). It would also be useful to identify what parts of Docutils large downstream consumers use, and either create higher level abstractions or mark those as public API. (I would suggest Sphinx and MyST-parser). 

It is also the [established practice](https://www.python.org/dev/peps/pep-0008/#descriptive-naming-styles) to mark names as private by prefixing them with an underscore. I suggest adopting this, as it gives strong guidance to downstream library authors what Docutils considers private and public. Making a private name public is far easier than going through a deprecation cycle for the reverse.

I'd suggest adding the ability to have exceptions to the policy, with removal after one minor version. My full suggested text is:

**"Removal or significant alteration of any members of the public API will only take place after the behaviour has been marked as deprecated for two minor releases. Certain changes may have a shorter deprecation period of one minor release. This requires at least two project members to be in favour of doing so, and no members against.**

**Changes that may affect end-users (e.g. by requiring changes to the configuration file or potentially breaking custom style sheets) should be announced with a FutureWarning."**

This post is intentionally opinionated so as to provide something to talk over / edit -- I'm not massively attached to any one thing!

A

----
for reference, amongst others, I took inspiration from:
https://setuptools.pypa.io/en/latest/development/releases.html
https://pip.pypa.io/en/latest/development/release-process/
https://numpy.org/neps/nep-0023-backwards-compatibility.html



---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.