#86 Replace recommonmark with myst-docutils
Hey guys,
As discussed in https://github.com/readthedocs/recommonmark/issues/221, recommonmark has been officially deprecated in favour of https://github.com/executablebooks/MyST-Parser/.
In myst-parser v0.16, I added first-class support for direct docutils use (as opposed to via sphinx) and introduced the parallel https://pypi.org/project/myst-docutils/ distribution, which is myst-parser but without explicit install requirements on docutils or sphinx, i.e. you can install myst-docutils with any version of docutils and it will work without sphinx installed.
receive(see https://github.com/executablebooks/MyST-Parser/blob/master/CHANGELOG.md#0160---2021-12-06, https://github.com/executablebooks/MyST-Parser/issues/347, and https://myst-parser.readthedocs.io/en/latest/docutils.html)
receive
receiveIn the master branch (and soon to be myst-docutils v0.17) I have also been working to improve the rigour of the behaviour/testing against this parser.
(as an example, I believe your current recommonmark solution does not actually work properly with code fences, because it does not syntax highlight the code, see: https://github.com/executablebooks/MyST-Parser/pull/478)
The fixture test sets, essentially provide a full specification of the Markdown -> docutils AST conversion (pre-docutils-transforms):
- https://raw.githubusercontent.com/executablebooks/MyST-Parser/master/tests/test_renderers/fixtures/docutil_syntax_elements.md
- https://raw.githubusercontent.com/executablebooks/MyST-Parser/master/tests/test_renderers/fixtures/docutil_roles.md
- https://raw.githubusercontent.com/executablebooks/MyST-Parser/master/tests/test_renderers/fixtures/docutil_directives.md
and are currently tested in the CI for myst-docutils against docutils v0.16, v0.17 and v0.18: https://github.com/executablebooks/MyST-Parser/runs/4670995311?check_suite_focus=true
I'm also happy to recieve any feedback / improvement suggestions to facilitate this, and have opened a parallel discussion topic here: https://github.com/executablebooks/MyST-Parser/discussions/487
Cheer,
Chris
As an aside, I am also working on propagating this first-class docutis support to https://github.com/executablebooks/MyST-NB, so you can do e.g.
mystnb-docutils-pseudoxml --report="info" --nb-execution-mode=cache tests/notebooks/basic_unrun.ipynb, but more on that at a later date :)Since [r8916], "myst" is a parser alias that points to the module
myst_parser.docutils_. You can use, e.g.,docutils-cli.py --parser=mystto parse MyST Markdown input.Docutils' config documentation links from the section
https://docutils.sourceforge.io/docs/user/config.html#myst-parser
to the upstream documentation.
I am not in favour of removing "recommonmark" support as long as
"recommonmark" is widely available (e.g. as Debian package).
We may consider changing the "commonmark" and "markdown" parser aliases to
a wrapper module that selects whichever of "myst" or "recommonmark" is
installed at the user's system.
Is there anything else that would be better done on the Docutils side?
Related
Commit: [r8916]
That great cheers, indeed the aliases are the main thing here:
I would prefer docutils itself did not override the
myst_parser.docutils_.Parser(as it does for the recommonmark parser) and instead any required changes/improvements were implemented directly in myst-parser.Obviously, I will leave it up to you guys, as to any testing you want to include directly in docutils.
For the "markdown"/"commonmark" alias, I would certainly be for it using myst-parser, but note that this would currently be equivalent to:
docutils-cli.py --parser=myst --myst-commonmark-only=yes.This "turns off" any syntax that is not strictly part of https://spec.commonmark.org/0.30/, such as tables and footnotes.
(you can see the implementation logic at: https://github.com/executablebooks/MyST-Parser/blob/0f2902ab5eb6236805ed24dc387b34c00fe41744/myst_parser/main.py#L226)
There could perhaps be a slight override for this.
Similarly, it might be interesting to note that the Github Flavoured Markdown spec: https://github.github.com/gfm/, is roughly equivalent to:
docutils-cli.py --parse=myst --myst-enable-extensions=linkify,tasklist --myst-disable_syntax=footnote_ref,footnote_def.The only thing that cannot be replicated from that spec currently is strikethrough: it is available in the markdown-it parser, but docutils does not have any native node class to support it :(
Agreed. So we may close the ticket as "open-fixed"?
This would require a (light) "parser wrapper" for commonmark that sets the
"myst_commonmark_only" setting default to
True.@chrisjsewell, @createsoftware, @tk0miya, and all interested: How about the attached patch.
Last edit: Günter Milde 2022-01-20
Just a note to say congratulations on this, folks. I'm excited to see MyST become a first-class citizen of the Docutils ecosystem :)
Well thank you for your help Clement!
Sounds good to me thanks, and don't hesitate to get in touch for any other related issues :)
I finished my work. It has been fully compatible with CommonMark.
The remaining task is creating an extension for sphinx using pycmark
and pycmark-gfm.
I thought to release v1.0 after that, but I lost my interest for it
because myst-parser has appeared.
Last edit: Günter Milde 2022-01-13
Marking this as fixed, will close once the next release is out.
BTW: what is the state of https://pypi.org/project/pycmark/ by @ tk0miya?
It looks promising as a lightweight alternative for pure CommonMark
as it only depends on Docutils.
Thank you all for the good work.
Release 0.19 (2022-07-05) supports parsing "Markdown" input with 3rd party parsers
myst, pycmark, or recommonmark.
Examples:
Master documents
Included documents:
Thank you for your contributions