#186 Modernise packaging

[Adam Turner]

Hi,

I had a go at modernising the packaging stack. setup.py based invocations have been deprecated (https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html), and setuptools may remove them in the future.

This takes the opportunity to move to a PEP 621 based declarative config, and also fixes a longstanding TODO item about providing script wappers for the frontend tools on windows, by migrating them to entry points.

I've updated install and development docs with the new guidance, and updated references to the frontend tools to remove .py, given they are now installed as proper scripts.

Hope this is appreciated -- happy to make revisions etc to help getting this merged.

A

[Adam Turner]

The patches merged together as a full diff is also attached on this comment

[Adam Turner]

A further change could be to deprecate/remove the rst2* tools in favour of the unified docutils-cli tool that was added a couple of versions ago -- this would simplifiy the command line usage space and perhaps enable further simplifications in the logic for using doctuils as a library.

If you'd like me to do this more than happy to -- I think it would be pretty simple.

A

[engelbert gruber]

contributions very welcome

please bear with us, currently we are still supporting python2.7. trying to avoid breakage needs some checking.
nag me if you think i am to slow which i am usually

thanks again

[Adam Turner]

we are still supporting python2.7

I checked and the build system proposed generates universal wheels still -- you can tell by the -py2.py3- tags in the filename.

A

[Adam Turner]

Meant to add as well -- I noticed you intend to drop support for 2.7 in a future release (https://sourceforge.net/p/docutils/code/8921/tree/trunk/docutils/RELEASE-NOTES.txt#l23).

Is there a tracking issue for this? I'd love to help in any way I can if possible @grubert -- please do let me know!

A

[Adam Turner]

(I can't find any way to update my patches/edit the message above similar to how one would on a pull request in GitHub or similar, so apologies for posting an entirely new message.)

I've updated the patch into what I think is a cleaner implementation, somewhat segregated into three parts.

Patches 1 and 2 now set up the minimum for a standards compliant packaging solution, and solve the issue of automatically creating executables on both unix and windows through entry points.

Patches 3 and 4 move the rst* frontends into the new entry points based setup (in the new _cli module, and update documentation to reflect both that the frontends should no longer be called with .py, and that the commands for building/installing Docutils have changed.

Patches 5 and 6 subsequentley deprecate the rst* based frontends (printing a message to the console on invocation noting what the user should run instead) and update documentation to prefer docutils-cli throughout as the canonical front-end tool.

I have intentionally kept 3/4 and 5/6 distinct, as I am not sure of your opinions on deprecating the other front-end tools.

As before, all tests pass & all commands etc are usable.

(Please let me know if there's anything else that would be useful to help on -- I proposed a patch for FR#61, but I don't know what your priorities are as maintainers. Thanks!)

A

[engelbert gruber]

Good morning Adam,

You have the same problem as i have, unsure about future command line tools.
I am waiting (a little) on Günter's comment, he made the new cli interface.
and am waiting (a more) on me slow thinking

many thanks to you
P.S. if you send me a pull request I think it is a diff in the end so I should be able to decipher

[Adam Turner]

No worries!

How do I make a pull request on sourceforge? or are you talking about via one of the git mirrors? Sorry for the confusion!

A

[engelbert gruber]

yes I would like to try it from a git mirror, what could go wrong :-)

[Adam Turner]

OK I have created a PR at https://github.com/AA-Turner/docutils/pull/1

You can see the unified diff by putting .diff or .patch on the end of the PR url, e.g.
https://github.com/AA-Turner/docutils/pull/1.diff

I pinged @grubert on GitHub which seems to be you!

A

[engelbert gruber]

and we have to think who might get in trouble if setup.py and install.py is missing.

[Adam Turner]

who might get in trouble if setup.py and install.py is missing

install.py is noted in the docs as just a hack for windows -- I'm writing this on a windows PC & have done all my testing on one, so no issues there!

As noted in Paul Ganssle's blog[1], python -m build is the recommended method for the future. Distribution packagers are working to these standards now -- for example typing_extensionshas had ~100mm downloads in the last month, and uses flit, as I'm proposing here. (python-rsa is another such example also at around 100mm per month).

The maintainers of setuptools (see the blog post) have indicated that they want to (intend to) deprecate usage of setup.py install, setup.py build, et cetera. I updated all the user documentation to make it clear how to build and install Docutils in the proposed build system, which hopefully eases the transition.

A

[1] https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary

[engelbert gruber]

I just want to ponder and assume your patch will be applied. But as docutils is older than setuptools we might have to consider things they do not ... somewhere in the blogpost linux distribution packagers are mentioned and then there is the sentence

note: this does not mean removing setup.py entirely or switching away from setuptools, it just means that you change all your scripts and documentation to tell people not to do this

I like it :-)

[Adam Turner]

I just want to ponder and assume your patch will be applied.

No worries, I appreciate python packaging isn't the easiest thing (or the most fun!).

You're right that removing setup.py is not mandatory. However it only makes sense to keep it if the "build backend" is kept as setuptools. One of the advantages of flit is that it does a lot less (only pacakges pure python projects like Docutils, no C complier integration etc), so there's less to go wrong. It is also an official Python Packaging Authority project (https://github.com/pypa/flit/), and has been recommended by python core developers.

A

[Günter Milde]

Currently, "setuptools" is also used for editable installs.
Will (editable) installs also work without setup.py when using pip 20.3.4 (the version in current Debian/stable) ?

[Adam Turner]

Will (editable) installs also work without setup.py when using pip 20.3.4 (the version in current Debian/stable) ?

Editable installs through the new standardised mechanism (PEP 660) require pip 21.3 or newer. This can be installed on any system with python -m install --upgdade pip. If 20.3.4 is in Debian, that is 2 years old at this point, but yes it would require updating.

Upgrading pip is always recommeded (see the warning every time you run with an old version of pip) -- some redistributors include rather old versions.

A

[Günter Milde]

If a current, stable pip does editable installs, we can drop the "install in development mode with setuptools" route. Requiring a recent pip should be acceptable for developers (and other users prefering an editable install), given that "manual" install is still possible for the "diehards".
A standard install should remain possible from, e.g., Debian without further non-system dependencies after any change.
If pip 20.3.4 suffices for a standard install without "setup.py" and if we can install the front-end tools under their current names, I am fine with a change of the build system.

[Adam Turner]

we can drop the "install in development mode with setuptools" route

Yep, I already applied that in my patch.

pip has done editable installs via setup.py basically forever. The new thing was editable installs with build backends that are not setuptools -- that is what was added in pip 21.3.

The current patch (moving to declarative metadata but keeping setuptools) therefore meets all of these critera, I think.

A

[engelbert gruber]

strictly speaking : this patch changes the command line API, removing commands, so it can only be applied in a major version change, really ?
Removing commands will break a lot of scripts.
I understand https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html sees the same problem

When you move away from direct setup.py invocations (note: this does not mean removing setup.py entirely or switching away from setuptools, it just means that you change all your scripts and documentation to tell people not to do this)

So my plan is to apply this piecewise, leave commands (the work to keep setup.* and pyproject.toml is mine anyway) AND keep the discussion assume to be nudged now and then.

[Adam Turner]

to crossreference my reply at https://github.com/AA-Turner/docutils/pull/1#issuecomment-1003845954

A

[Günter Milde]

Thank you for the elaborated patch set. Please be patient with our hesitation to apply it as-is.

Although officially still in Beta stage (version number <1.0), Docutils is widely used as if it were stable and we try to minimise the impact of changes on users.

Installation should be possible without dependencies other than stock Python and its standard library. There is an exception with setup.py requiring "setuptools", but currently it's also possible to get Docutils running by placing a *.pth file in the Python path and symlinks for the front-ends in the binary path (/usr/local/bin/rst2html5 -> /usr/local/src/Docutils/docutils/tools/rst2html5.py, say).

Switching to static metadata should be possible without violating this condition. Renaming setup.py to pyproject.toml before applying the changes to make it a toml file would allow keeping the classifiers in the version history.

The "longstanding TODO item about providing script wappers for the frontend tools on windows, by migrating them to entry points" comes as a suprise to me.
I see the potential benefit for Windows users, OTOH, I'd like to keep the front-ends as executable files for "hand installation" as well as templates for custom, home made front-ends.
I wonder if we could eat the cake and have it by moving the tools/ directory inside the package (maybe renaming it to cli/) and modifying the front-end tools to provide an entry function that is executed if name == '__main__.

A name change for the frontend tools is an API change that requires an advance warning in the RELEASE-NOTES (note that, e.g., Debian already drops the *.py extension from the front-end tools). If implemented via an "entry point", the command name docutils-cli may also be simplified to docutils.
Once we are at it, it may be worth discussing which front-end tools should be provided. The generic docutils-cli could obsolete some or all of the rst2*.py scripts, but it is new and not yet reviewed/endorsed by Docutils main author David Goodger.
Also, regular Docutils users and developers benefit from dedicated tools (and TAB-expansion after typing rst2) while users installing Docutils only because it is required by some other package may prefer less "noise" in the binary path.

[Adam Turner]

Installation should be possible without dependencies other than stock Python and its standard library.

There's an intention in python packaging to become more standards based (PEPs 517, 518, 621, 660, etc) and less reliant on a single blessed tool (setuptools).

There have been recent discussions of removing setuptools from the default python binaries (https://mail.python.org/archives/list/python-dev@python.org/thread/3BVAUIQOEOXAULHVYQNLLQIZQQETX2EV/) -- the only constraint seems to be when pip no longer needs it.

My argument in short is that I don't believe installing Docutils from source using just the standard library will be possible for much longer -- disutils is being removed from the stdlib & will be removed in Python 3.12, and after that there are no bundled building mechanisms.

Flit (the system I proposed) has a strong focus / emphasis on boostraping -- see https://flit.readthedocs.io/en/latest/bootstrap.html. If you have the sources of flit and Docutils, you can end up with a functioning installation of Docutils using only the standard library -- would that be good enough reassurance?

Renaming setup.py to pyproject.toml before applying the changes to make it a toml file would allow keeping the classifiers in the version history.

Will do

I wonder if we could eat the cake and have it by moving the tools/ directory

I sort of did this in _cli.py -- see the section that starts DEPRECATED ENTRY POINTS. I switched from using the publish_command_line functions to routing through docutils-cli as it shows how they're all connected.

If you wanted to keep the .py files in the tools directory then we could add aliases -- the contents of rst2html5.py would simply be:

from docutils import _cli
_cli.publish_html5()

templates for custom, home made front-ends

We could also add explicit templates in an examples folder or similar -- this would allow demonstrating potential usages of the API without needing to use it ourselves.

A name change for the frontend tools is an API change that requires an advance warning in the RELEASE-NOTES

Of course, I can prepare a patch that only updates RELEASE-NOTES

If implemented via an "entry point", the command name docutils-cli may also be simplified to docutils.

OK, will do. We can also keep the legacy docutils-cli as an alias and emit deprecation warnings until removal in the next major version.

Once we are at it, it may be worth discussing which front-end tools should be provided. The generic docutils-cli could obsolete some or all of the rst2*.py scripts, but it is new and not yet reviewed/endorsed by Docutils main author David Goodger.
Also, regular Docutils users and developers benefit from dedicated tools (and TAB-expansion after typing rst2) while users installing Docutils only because it is required by some other package may prefer less "noise" in the binary path.

(note the below is entirely personal opinions, I am not David!)

I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

A converstation I wanted to have at a later date is starting to separate the application and library usages of Docutils (whilst keeping full backwards compatability). Having a single front-end command line interface would make that (theoretical) work significantly easier.

I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

My suggested timeline is:
- 0.19.0: announce upcoming changes. Just the drop of support of <3.7 may be big enough to release 0.19 -- if so could the future changes notice be included before you release it please?
- 0.20.0: add entrypoints, retain rst2* files in tools/ (although no longer automatically copied into PATH), deprecate rst2* and docutils-cli commands at runtime (the deprecation message contains the replacement invocation needed)
- 0.21.0: Remove the deprecated rst2* and docutils-cli aliases, leaving the single docutils command line tool.

I'd be interested in your thoughts on the proposed timeline -- is it workable? Is there anything I missed? (it is quite late here!)

A

---

package name	downloads	pin	type
awscli	105,842,835	>=0.10,<0.16	pure docutils
apache-airflow	4,023,459	<0.17	with Sphinx
sphinx	3,843,715	<0.18,>=0.14	with Sphinx
readme-renderer	2,353,477	>=0.13.1	pure docutils
sphinx-rtd-theme	1,740,846	<0.18	with Sphinx
c7n	945,488	==0.17.1	with Sphinx
c7n-org	837,789	==0.17.1	with Sphinx
recommonmark	611,332	>=0.11	with Sphinx

[Günter Milde]

On 2022-01-04, Adam Turner via Docutils-develop wrote:

... I don't believe installing Docutils from source using just the
standard library will be possible for much longer

What I have in mind is a "manual install" (point 3 in https://docutils.sourceforge.io/docs/dev/repository.html#editable-installs).
You may also call it "using Docutils without installing".
This can be done without any dependencies and offers most flexibility
(e.g. select which front-ends to make available as commands and under which name).

[engelbert gruber]

I usually do this , set a symlink and work on.
And I very much appreciate not having to install docker and hub and whatever for development.
But this is only me and I can change

[Adam Turner]

The symlink approach will absolutely still work :)

I don't think anyone has suggested Docker -- that's far too complex!

Do you think you could merge the patch as currently proposed? If not, what are the blockers?

A