[Docutils-develop] [docutils:feature-requests] Re: #88 Unify Docutils CLI tools into `docutils-cli`

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

> Currently deep in Docutils' internals (everywhere that takes a
> `settings_spec` or uses `self.settings` sort of assumes working as a
> command line programme. However, a lot of usage (programmatic, through
> Sphinx or other methods) entirely use the default values for things.

Even with Sphinx, some features can only be customised from a
docutils.conf configuration file.

The `sttings_spec` and `document.settings` are Docutils abstraction from
the different configuration ways (config-files/command line/programmatic).
Using `document.settings` should be possible without too much thinking
about the actual source of the setting value.

An overview for programmatic use of the "settings" framework is given in
https://docutils.sourceforge.io/docs/api/runtime-settings.html#runtime-settings-processing-from-applications
(best read alongside `pydoc3 -b` output for the mentioned functions/classes).

> By moving to a single front end I would argue it is not only a cleaner
> user story, but it *might* enable refactoring to move the CLI usages of
> Docutils to a higher level.

We already have the `docutils.core.publish_*` "convenience functions" as
a high-level API for custom front-ends (both command-line and programmatic).

"docutils-cli" is more complex because here we want the components to be
configurable from the command line or config file. I am working on moving
the complexity to a library function that can be re-used by other
"script" entry-points in need of configurable components. This will become
an extension or addition to `docutils.core.publish_cmdline()`.
(It may also become simpler once "optparse" is replaced with "argparse".)

> Currently we need to do awful things to subclass and patch either
> `optparse.OptionParser` or `argparse.ArgumentParser`. This is really
> unusual, and for developers coming from a more "normal" command line
> application, it can take a while to understand this part of the
> internals of Docutils.

Yes, indeed. Docutils has an elaborated configuration framework which
actually predates the "optparse" module. Later development of "Optik"
into "optparse" and then "argparse" implemented some of the abstractions and
enhancements offered by Docutils in a different way.

But (in contrast to developers working on the optparse->argparse
transition :) "normal" developers using the "docutils" package don't need
to care about the details here. They can use the high-level API offered
by SettingsSpec / settings and get the command line and config file
processing for free (`docutils.frontend` is only the "workhorse", 
`docutils.core` is the high-level interface).
I agree that there is room for improvement in this API, but I don't think
getting rid of the simple front-ends in favour of one complex front end
will be of much help in this quest.

> I didn't go into detail intentionally so as not to spark a debate about
> these parts, but I do think (eventually) simplifying these interactions
> can lead to a cleaner codebase.

I suggest moving this thread of the discussion over to [bug:#441].

>> the two-stage command line parsing

> I don't think you can get away from this though without a combinatorial
> explosion of readers, writers, and parsers.
> Say we have two useful CLI readers (standalone/pep), three parsers
> (rst/recommonmark/myst), and 6 useful writers
> (html5/html4/latex/xetex/man/xml) that is 36 distinct front-end tools
> we should be providing.

However, only some of the combinations will be of common interest.
We should try to find the right balance -- IMO, both extremes are
sub-optimal.

* Docutils will not include dedicated front-end tools for 3rd-party
  parsers/writers/... ("pycmark2..." shall be provided by "pycmark" etc).

* One idea is to have two packages at pypi: "docutils-core", say,
  without dedicated front-end tools (but supporting `python -m docutils`)
  and "docutils" providing a sensible set of front-end tools.

* Another idea is to auto-install a small default set (rst2html,
  rst2latex, ...) and keep a rich set in `/tools` so that every user may
  install (copy, symlink or write alias commands in ~/.bashrc or
  ~/.profile) the tools the want "by hand".

* `rst2odt_prepstyles.py` is a rarely used auxiliary script.
  I propose to move it to `docutils/writers/odtwriter/`
  (alongside the stylefile(s) it prepares).

...

>> Ease of discovery is important. TAB completion is a powerful means here

> Did you see my suggestion on using custom shell autocompletion
> functions? I believe that this would allow for tab completion with the
> reader/parser/writer flags.

That is a possibility. However, it only works with some shells (bash) so it
is not for all users.

>> rst2 is established as the start of Docutils' front-end names for
>> conversion from reStructuredText to something. I would like to keep
>> this prefix as "ours".

> If we use what I propsed in one of my changesets to reimplement the
> `rst2` commands in terms of `docutils-cli`, it would be entirely
> possible to deprecate the `rst2` commands but just keep them forever.
> This would also mean that the simplifications I proposed at the top of
> this message wouldn't be blocked (I think).

I am just working on a way to disentangle frontend.OptionParser and
frontend.ConfigParser but this is a topic for [bug:#441].

---

** [feature-requests:#88] Unify Docutils CLI tools into `docutils-cli`**

**Status:** open
**Group:** Default
**Created:** Sat Jan 15, 2022 10:11 PM UTC by Adam  Turner
**Last Updated:** Thu Jan 20, 2022 01:52 AM UTC
**Owner:** nobody

As noted at https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e/ef2d by @milde, 
> we should open a new ticket for the command line tool review

This is a tracker issue for this, and to allow discussion. 

I'll briefly re-outline my argument to (eventually) drop the `rst*` front-end tools, and only export `docutils-cli` (or `python -m docutils`). 

> I think a single front-end tool significantly simplifies a lot of things -- the docutils-cli wrapper is not complex, which gives it significant points in favour in my book.

> Most usage of Docutils today is programmatic, and not via the command line tools (see the table at the bottom of this post - it shows all the projects that have a full dependency on Docutils with over 500k downloads in the last month. Of those 8, none use the command line tools)

> I also suspect (although the data does not exist) that most command line uses of the Docutils tools will be rst2html(5). This is already the default in docutils-cli, so it is a drop-in replacement.

> ...

>  My proposal isn't to remove them *\[the rst2 front-end tools\]* with no recourse, but to deprecate over a period of time, clearly marking identical drop-in commands at runtime to affected users. ... We cannot know how many people would be affected with local random scripts, but it is a two-second change.

> Many users will also run with old or pinned versions of Docutils, and part of updating is seeing the changelog. If Debian or other redistributors already make changes, they could decide to keep shell aliases from rst2* to the new docutils-cli based invocations. 

(quotes taken from https://sourceforge.net/p/docutils/patches/186/#897a and https://sourceforge.net/p/docutils/patches/186/?page=1#897a/547e )

A

---

Sent from sourceforge.net because doc...@li... is subscribed to https://sourceforge.net/p/docutils/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/docutils/admin/feature-requests/options.  Or, if this is a mailing list, you can unsubscribe from the mailing list.