[Docutils-develop] [docutils:feature-requests] Re: #88 Unify Docutils CLI tools into `docutils-cli`

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

> Can you elaborate a bit on what would become a lot simpler?

Currently deep in Docutils' internals (everywhere that takes a `settings_spec` or uses `self.settings` sort of assumes working as a command line programme. However, a lot of usage (programmatic, through Sphinx or other methods) entirely use the default values for things. By moving to a single front end I would argue it is not only a cleaner user story, but it *might* enable refactoring to move the CLI usages of Docutils to a higher level. 

Currently we need to do awful things to subclass and patch either `optparse.OptionParser` or `argparse.ArgumentParser`. This is really unusual, and for developers coming from a more "normal" command line application, it can take a while to understand this part of the internals of Docutils.

I didn't go into detail intentionally so as not to spark a debate about these parts, but I do think (eventually) simplifying these interactions can lead to a cleaner codebase.

> the two-stage command line parsing

I don't think you can get away from this though without a combinatorial explosion of readers, writers, and parsers. Say we have two useful CLI readers (standalone/pep), three parsers (rst/recommonmark/myst), and 6 useful writers (html5/html4/latex/xetex/man/xml) that is 36 distinct front-end tools we should be providing. 

> a man page will always need to refer to external documentation

I will admit ignorance on how man pages work. `docutils-cli --writer xetex --help`, though, will always give the correct help output. This is also the version we should be promoting, not least as it works cross platform (if my patch with entrypoints is merged!).

> We need to care for "command line users" if their number is non-negligible 

Of course -- sorry if my post came across as callous in any way towards frontend tool users. I suppose what I don't want is to be in a situation where we are not making real improvements based on hypothetical situations. It might be useful to find ways of proxying for CLI usage -- bugs filed recently with us/redistributors, usages in public archives ( https://grep.app or similar ), etc.

> rather hard to find out how many non-Python projects uses "rst2html.py" in their Makefile or another form of build tool chain

True. However by the above methods we can get an estimate, surely? There are a lot of people who commit random things to GitHub / GitLab / whatever!

> finding out and approaching the right spot where to apply the change

This is why I proposed to go about it by emitting warnings during deprecation, before total removal. We also need to consider the support that this project offers -- if a downstream user has integrated Docutils into a complex tool chain and cannot maintain it, we shouldn't be responsible for that. 

> never underestimate the number of users/project managers that don't read the changelog ... yet depend on a stable Docutils for a stable system.

Fair enough -- though perhaps another route we could go down in the deprecation notices are to say "pin version XX". There is no best solution here -- all change will break someone's workflow ([XKCD 1172!](https://xkcd.com/1172/)), but we should be working to make the upgrade path as easy as possible.

> buildhtml

Ahh, I was under the impression that `buildhtml` was an internal tool for building the website. Would it be reasonable to formally retire it from public use, and reccommend Sphinx as an alternative? 

>  use `docutils` as front-end command // `python -m docutils`

+1

> Ease of discovery is important. TAB completion is a powerfull means here

Did you see my suggestion on using custom shell autocompletion functions? I believe that this would allow for tab completion with the reader/parser/writer flags.

> rst2 is established as the start of Docutils' front-end names for conversion from reStructuredText to something. I would like to keep this prefix as "ours". 

If we use what I propsed in one of my changesets to reimplement the `rst2` commands in terms of `docutils-cli`, it would be entirely possible to deprecate the `rst2` commands but just keep them forever. This would also mean that the simplifications I proposed at the top of this message wouldn't be blocked (I think).

-----

Concrete proposal:

- Promote `docutils` or `python -m docutils` where we currently reference `rst2`
- Reimplement the `rst2*` commands in terms of `docutils-cli` 
- Try to implement <TAB\> autocompletion for `docutils-cli`
- Use entrypoints for everything (but also keep `.py` aliases for a while)
- Deprecate `rst2*` commands, but with no removal date

A

---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Sun Jan 16, 2022 11:57 PM UTC
**Owner:** nobody

As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A

---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.