#450 0.18: New HTML markup for footnotes is difficult to stylise

[Pradyun Gedam]

Docutils 0.18 changed the HTML markup for footnotes/citations generated for rst documents. This changed markup is significantly more difficult to stylise with CSS, in ways that was possible with Docutils 0.17. This has negatively affected HTML themes for Sphinx, since the Sphinx 5 release allows using docutils 0.18, which is preferentially installed by pip.

The reasons for this boils down to an inconsistent number of elements inside each aside.

1. It's not possible use CSS grid layouts sanely, with this setup.
2. Even if you added multiple rules based on number of elements in the section, it's not possible to know what the 2nd element might be -- which balloons the complexity+size of the stylesheet, if it tries to accomodate for this.

It is theoretically possible to stylise this but it would be on the order of 100s of lines of CSS to get this right, compared to ~20 with 0.17.

Would it be possible to change this markup, to wrap the label & backrefs in a div and to wrap the content paragraphs in a separate div as well? This would make it possible to stylise this content in ways that were feasible with 0.17, with significantly less complexity in the stylesheets.

---

Sources:

[some content that references these footnotes]

.. [1] A footnote contains body elements, consistently indented by at
   least 3 spaces.

   This is the footnote's second paragraph.

.. [#label] Footnotes may be numbered, either manually (as in [1]_) or
   automatically using a "#"-prefixed label.  This footnote has a
   label so it can be referred to from multiple places, both as a
   footnote reference ([#label]_) and as a hyperlink reference
   (label_).

0.17 output HTML:

<dl class="footnote brackets">
  <dt class="label" id="id6"><span class="brackets">1</span><span class="fn-backref">(<a href="#id1">1</a>,<a
        href="#id7">2</a>)</span></dt>
  <dd>
    <p>A footnote contains body elements, consistently indented by at
      least 3 spaces.</p>
    <p>This is the footnote’s second paragraph.</p>
  </dd>
  <dt class="label" id="label"><span class="brackets">2</span><span class="fn-backref">(<a href="#id3">1</a>,<a
        href="#id8">2</a>)</span></dt>
  <dd>
    <p>Footnotes may be numbered, either manually (as in <a class="footnote-reference brackets" href="#id6"
        id="id7">1</a>) or
      automatically using a “#”-prefixed label. This footnote has a
      label so it can be referred to from multiple places, both as a
      footnote reference (<a class="footnote-reference brackets" href="#label" id="id8">2</a>) and as a hyperlink
      reference
      (<a class="reference internal" href="#label">label</a>).</p>
  </dd>
</dl>

0.18 output HTML:

<aside class="footnote brackets" id="id6" role="note">
  <span class="label"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></span>
  <span class="backrefs">(<a href="#id1" role="doc-backlink">1</a>,<a href="#id7" role="doc-backlink">2</a>)</span>
  <p>A footnote contains body elements, consistently indented by at
    least 3 spaces.</p>
  <p>This is the footnote’s second paragraph.</p>
</aside>
<aside class="footnote brackets" id="label" role="note">
  <span class="label"><span class="fn-bracket">[</span>2<span class="fn-bracket">]</span></span>
  <span class="backrefs">(<a href="#id3" role="doc-backlink">1</a>,<a href="#id8" role="doc-backlink">2</a>)</span>
  <p>Footnotes may be numbered, either manually (as in <a class="footnote-reference brackets" href="#id6" id="id7"
      role="doc-noteref"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></a>) or
    automatically using a “#”-prefixed label. This footnote has a
    label so it can be referred to from multiple places, both as a
    footnote reference (<a class="footnote-reference brackets" href="#label" id="id8" role="doc-noteref"><span
        class="fn-bracket">[</span>2<span class="fn-bracket">]</span></a>) and as a hyperlink reference
    (<a class="reference internal" href="#label">label</a>).</p>
</aside>

The suggested change to the markup is to generate something like (using only the first aside for this demo):

<aside class="footnote brackets" id="id6" role="note">
  <div class="footnote-label">
    <span class="label"><span class="fn-bracket">[</span>1<span class="fn-bracket">]</span></span>
    <span class="backrefs">(<a href="#id1" role="doc-backlink">1</a>,<a href="#id7" role="doc-backlink">2</a>)</span>
  </div>
  <div class="footnote-content">
    <p>A footnote contains body elements, consistently indented by at
      least 3 spaces.</p>
    <p>This is the footnote’s second paragraph.</p>
  </div>
</aside>

[Adam Turner]

This is actually pretty easy to fix -- I agree it is currently not ideal.

Patch and diff attached. @milde -- can this go into 0.19 please? I don't know if you heard back fron Engelbert on when he's going to make the release.

diff --git a/docutils/docutils/writers/_html_base.py b/docutils/docutils/writers/_html_base.py
index 19e394657..d0d2d58e8 100644
--- a/docutils/docutils/writers/_html_base.py
+++ b/docutils/docutils/writers/_html_base.py
@@ -951,7 +951,6 @@ class HTMLTranslator(nodes.NodeVisitor):
                                        role="note"))

     def depart_footnote(self, node):
@@ -1104,8 +1102,6 @@ class HTMLTranslator(nodes.NodeVisitor):
                          for (i, ref) in enumerate(backrefs, 1)]
             self.body.append('<span class="backrefs">(%s)</span>\n'
                              % ','.join(backlinks))
+        self.body.append('</div>\n')  # closer for <div class="fn-text">
         self.body.append('</aside>\n')

     def visit_footnote_reference(self, node):
@@ -1081,6 +1082,7 @@ class HTMLTranslator(nodes.NodeVisitor):

     # footnote and citation labels:
     def visit_label(self, node):
+        self.body.append('<div class="fn-label">\n')
         self.body.append('<span class="label">')
         self.body.append('<span class="fn-bracket">[</span>')
         # footnote/citation backrefs:
@@ -1102,6 +1104,8 @@ class HTMLTranslator(nodes.NodeVisitor):
                          for (i, ref) in enumerate(backrefs, 1)]
             self.body.append('<span class="backrefs">(%s)</span>\n'
                              % ','.join(backlinks))
+        self.body.append('</div>\n')
+        self.body.append('<div class="fn-text">\n')

     def visit_legend(self, node):
         self.body.append(self.starttag(node, 'div', CLASS='legend'))

A

[Adam Turner]

Updated patch, I fixed missing line breaks.

A

[Adam Turner]

A third update, I've now updated tests and CSS to follow the new structure.

A

[Günter Milde]

Would it be possible to change this markup, to wrap the label &
backrefs in a div and to wrap the content paragraphs in a separate
div as well?

Unfortunately, this would break the default footnote styling ("minimal.css")
as well as the styles building on it ("plain.css" and "responsive.css").
Try the "standalone_rst_html5.html" test sample
to see the intended footnote rendering.

If you gave more details of what you want to achieve and/or examples of
the CSS we may find a solution.

[Adam Turner]

@milde -- checking my patch in FireFox 101, I see no visual difference between standalone_rst_html5.html#footnotes before and after applying the patch.

I believe I was fairly thorough in updating the CSS, although I might've gotten something wrong -- I would appreciate if you had a quick look / check please?

A

[Pradyun Gedam]

https://pradyunsg.me/furo/kitchen-sink/generic/#footnotes has the 0.17 rendering.

https://github.com/pradyunsg/furo/blob/d955850a89310a932ecab4e84cba0c20371a6a33/src/furo/assets/styles/content/footnotes.sass#L12 has the CSS, basically using the CSS Grid model to stylise the content to be aligned correctly (written in Sass for an indentation based syntax).

FYI: https://docutils.sourceforge.io/test/functional/expected/standalone_rst_html5.html#footnotes seems to be unstylised, due to a broken link to minimal.css.

[Adam Turner]

Attached is a fourth patch that fixes two remaining inconsistencies I noticed. Günter -- sorry, the third patch I uploaded is not visually identical, but this one should be.

Pradyun -- your footnotes.sass link doesn't seem to work.

due to a broken link to minimal.css.

This is one of the problems I had during the reposurgeon conversion -- these four CSS files in input/ seem to have broken symlinks in the SVN index or some other oddity going on.

A

[Pradyun Gedam]

The markup rendering for Sourceforge seems to have eaten an _.

https://github.com/pradyunsg/furo/blob/d955850a89310a932ecab4e84cba0c20371a6a33/src/furo/assets/styles/content/_footnotes.sass#L12

[Günter Milde]

can this go into 0.19

I am hesitant to introduce an incompatible API change unannounced and in the last minute .
It has the potential to break custom style sheets.

We don't know whether the patch solves the OP's problem. The rendering in https://pradyunsg.me/furo/kitchen-sink/generic/#footnotes cannot be reproduced easily even with the patch, as every footnote has its own grid (as opposed to one grid for the list of footnotes in the example).

The varying number of elements in every footnote <aside> element can easily be rendered on a 3-column grid if you assign them to grid columns. See attachment. (Tested with rst2html5 --stylesheet-path=minimal.css,gridfoot.css footnotes.txt.)

OTOH, a layout with one column for footnote label and backlinks and a second for the footnote text becomes problematic if there are many backlinks
(as, e.g. in https://docutils.sourceforge.io/docs/user/config.html#override).
This may be the reason, why html4css1 puts the backlinks in the column with the footnote text.

[Pradyun Gedam]

On Tue, Jun 7, 2022 at 7:21 PM "Günter Milde" milde@users.sourceforge.net
wrote:

can this go into 0.19

I am hesitant to introduce an incompatible API change unannounced and in
the last minute .
It has the potential to break custom style sheets.

Could you elaborate on this? There was an incompatible API change in the
0.18 release -- why can't the same be done in 0.19?

[Günter Milde]

There was an incompatible API change in the 0.18 release -- why can't the same be done in 0.19?

There is no outright ban on such a change in the Docutils policies or in semantic versioning (as 0.19 < 1.0).
OTOH, HTML structure is part of the API and incompatible changes should be avoided if there is another way to solve the problem.

To ask it the other way: How can we be sure an incompatible change in 0.19 is a net benefit, when an incompatible change in 0.18 introduced a problem?

[Pradyun Gedam]

Fun, SourceForge does not allow users to reply via email.

We don't know whether the patch solves the OP's problem. The rendering in https://pradyunsg.me/furo/kitchen-sink/generic/#footnotes cannot be reproduced easily even with the patch, as every footnote has its own grid (as opposed to one grid for the list of footnotes in the example).

That's correct. It does not.

In my example, <div class="footnote-label"> functionally replaces the dt and the content div replaces the dd from 0.17. It changes the footnote to being a wrapper around the divs, which come in pairs. In effect, mirroring the structure from before (with an aside rather than dl, and divs rather than dt and dd).

I am hesitant to introduce an incompatible API change unannounced and in the last minute .
It has the potential to break custom style sheets.

"incompatible API change" seems like a weak reason for hesitancy for accepting a patch, since an incompatible API change in 0.18 (compared to 0.17) is literally what caused this issue.

Does docutils announce upcoming releases and their changes somewhere before the release is made? If not, I can tell you that end users will not care if the change was made 1 hour, 1 week, 18 weeks before the release. :)

[Adam Turner]

Does docutils announce upcoming releases and their changes somewhere before the release is made?

We try to: https://docutils.sourceforge.io/RELEASE-NOTES.html#future-changes and https://docutils.sourceforge.io/HISTORY.html#changes-since-0-18-1

A

[Adam Turner]

@pradyun -- sorry, for clarity does my patch no. 4 allow you to solve the issue in Furo or is more needed?

@milde -- from a CSS styling perspective, I will say I found it challenging working with the footnotes structure in v0.18, and I believe the patch makes it easier for end users. (Note many end users haven't yet updated their themes and CSS styles to v0.18 as we just released Sphinx 5 with support -- this is probably why the issue was raised now rather than earlier).

A

[Günter Milde]

Fun, SourceForge does not allow users to reply via email.

It does. This is an email reply.

I don't know what kind of pre-conditions and spam-checks apply, though.

[Pradyun Gedam]

You're right. I'm not sure why, but looks like it came in ~4 hours after I sent it. :)

https://sourceforge.net/p/docutils/bugs/450/#6bdd/ddc7

[Pradyun Gedam]

@aa-turner -- the patch would be an improvement over status quo. See image for what can be achieved with the new markup, which is an improvement over 0.18 but not at parity with 0.17.

In other words, for preserving the style in Furo, more is needed.

As @milde pointed out, Furo uses a grid layout on dl, which means it is able to keep all the footnotes aligned even when there's different width among various labels+references. This would require moving away from each footnote living in its own aside to having a single aside containing multiple footnotes (which is what 0.17 had).

[Günter Milde]

The result shown in the image can easily be achieved with the current
"html5" footnote markup and a 3-column grid layout
(see gridfoot.css for a prove of concept).

How can we improve?
"html5_polyglot" aims for clear, semantic HTML markup that is easily
stylable and "degrades gracefully" (e.g. with missing CSS).

Traditionally, groups of footnotes are rendered either as
sequence of paragraphs (with standard paragraph separation, hanging indent
or no separation) preceded by the reference mark or as a
list with the reference mark as list label.

The list style is common for endnotes.
rST and the Docutils document model don't differentiate footnotes from
endnotes - it is up to the placement in the source.
(Endnote support is an old TODO item, cf. below.)

There is no equivalent for the optional footnote backrefs element
in printed documents. Wikipedia references
and the "html4css writer" use only the reference mark as list label
and place backrefs in front of the footnote text.

Reference mark, optional backlinks and actual footnote content should be
wrapped in a "footnote" element for accessibility and clear semantics.
I don't know whether this constitutes a major obstacle for styling as a list
by way of a CSS "grid".
The HTML standard allows div wrappers around dt –dd groups
in dl elements.
Hence, there might be a precedence for styling such an element hierarchy as grid.

If the additional level constitutes a major obstacle for styling
by way of a CSS "grid", we may compromise for the case of HTML5 output
treating all footnotes as endnotes:
wrapping groups of footnotes in an element with role doc-endnotes
containing a list with the actual notes.
This would match nicely the use case of rST footnotes placed "at the
end of a work or a section within it" at the cost of a mis-match for
footnotes placed within a section, near their reference point in the text.
I'd prefer to spare this for a possible "endnotes" directive to collect footnotes.

[Adam Turner]

Immediate response on the CSS styling issue -- patch version 5 wraps all adjacent footnotes into one aside so that CSS grid can be used (because of the wrapping, you need to use aside.footnote > div {display: contents;} to be able to assign grid column positions to the fn-label and fn-text classes -- ideally we'd use subgrids but there's no support beyond FireFox. @pradyun would you be able to test if this works for Furo? (I haven't updated tests or HISTORY)

On the issue of semantics -- I think grouping adjacent footnotes into a single container makes sense, as it allows for multiple blocks of footnotes in one document (say one per major section). For the HTML writer, I think we should aim to allow the end user to style as he prefers -- with sensible defaults in minimal, plain, and responsive.

We would ideally be able to lay this out as we saw fit, but the CSS specification, and the implementation of the CSS specification by browser engines are limiting factors that mean we have to make unfortunate compromises.

An alternate patch is also attached, which only changes the outer wrapper and doesn't use the .fn-(label|text) elements. This allows using the CSS grid by e.g.:

aside.footnote * {
    grid-column: 2;
}
aside.footnote .label,
aside.footnote .backrefs {
    grid-column: 1;
}

I think these two approaches are the most pragmatic that balance functionality in CSS against semantics of the HTML -- I would welcome thoughts on either implementation. Once (if) we have a winner, I will do the final polishing steps to prepare it as a proper patch.

A

[Günter Milde]

I don't believe that the "backrefs" should be grouped as part of the footnote label:
it complicates the structure and handicaps styles similar to the Wikipedia references. Therefore I prefer the simpler, "alternate-patch".

The "ids" and the ARIA role must be kept on the individual footnotes. (Did you test the footnote links?)
When also keeping the "footnote" and "brackets"/"superscript" class values on the individual notes, minimal.css keeps working. Only tuftig.css needs adaption.
The attached patch is a proof of concept.

[Adam Turner]

@milde your approach makes a lot more sense! The attached builds on your patch -- I renamed footnotes to footnote-list for consistency with citation-list and as I think the risk for single-character typos would be high.

I also added a CHANGES entry.

A

[Günter Milde]

Updated patch based on r9070

• keep using <aside> for footnotes. Both, a group of footnotes (footnotes list) and the individual footnotes "could be considered separate from the content [...] around the aside element".
• Adapt tuftig.css, make use of the wrapper to simplify plain.css.
• Document change in HISTORY and RELEASE_NOTES.
• Use ARIA role "doc-footnote" instead of "note". This could also be done later, in a separate fix.

[Adam Turner]

+1 on the attached patch.

My only thought would be using <section class="footnote-list"> for the outer container, but I'm not sure it is a good fit:

The section element represents a generic section of a document or application. A section, in this context, is a thematic grouping of content, typically with a heading.

So the aside.footnote-list > aside.footnotes.(brackets|superscript) seems the best option. If we commit today, we may be able to make a beta release tomorrow even.

A

[Günter Milde]

One more variation: add the bracket/superscript class value on the wrapper, too.

[Adam Turner]

+1, makes sense. The CSS Selectors Level 4 standard introduces a :has selector [1]_ that would allow doing .footnote-list:has(> .brackets) but it is only supported by Safari according to MDN -- so the approach in your patch is needed.

_[1]: https://developer.mozilla.org/en-US/docs/Web/CSS/:has

A