[Docutils-users] Dealing with exception-based control flow in sections

[Arne S. <arn...@gm...>]

Dear all,
I am working on an application of docutils/rST, but I'm running into a
snag with how sections find their place in the tree using exceptions
for control flow.

The use case I'm working with is making a solution for non-technical
users to edit templates that will be used to produce the final
documents. So basically a templating solution, but domain-specific and
couched in the terms of the domain rather than explicit logic. This is
probably somewhat counter to how rST is intended to be used, but I
still think it could be a good solution.

Mostly things are working out quite well, but the way sections are
implemented in the rST parsing of docutils interacts poorly with my
construct that has a looping logic. Consider, the following fragment
(obviously simplified, but it should communicate how I intend it to
work):

.. loop-construct:

   This text is repeated, but with a varying value: :getfield:`fieldname`.

The way it's implemented is that the loop-construct directive loops
over the underlying list, setting the value in a
contextvars.ContextVar and calling self.state.nested_parse on the
content each time, producing the elements that get inserted into the
output. This works quite well, until section headers get involved.
What happens is that when a section is encountered that is a sibling
of the current section level or higher in the section tree, the
parsing state gets scrolled back to just before the offending section
header, and EOFError is raised.[0] This error percolates up the stack,
unwinding the section state as we go, until we get to a state where we
can continue.

0: docutils.parsers.rst.states.py, line 361 in
RSTState.check_subsection in my copy of docutils-0.21.2

The practical result of this is that the contents of my directive get
truncated just before the section header each time around the loop,
which is obviously not ideal. I've tried a couple of different
approaches to getting this right, but nothing that quite works. The
closest I've gotten involves looking at the return value from the
nested parse, and when there's material left over taking the remaining
lines, injecting the directive in front of those lines and indenting
them before reinserting them into the input stream with
self.state_machine.insert_input(). That sort of work, but I'm pretty
sure it won't propagate the loop state correctly outside of the tiny
example I've been using to debug this issue.

Does anyone have suggestions on how to best handle this? I'm open to
alternate ways of implementing the looping logic, but it really does
need to loop I'm afraid and I think all my alternate ideas have so far
run into some issue with threading the state correctly while still
also getting everything into the output.

Arne
:wq