#5411 Improve whiteout behaviour with indifferent layering

[Simon Albrecht]

The current drawing process doesn’t always work well with regard to grobs that are on the same layer, sometimes requiring extensive manual tweaks to layering.

Original report by Malte Meyn:

In the following example the NoteHead.whiteout doesn’t only cover the tie but also one NoteHead whites out the other:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\version "2.19.82"

\relative <<
  {
    \override NoteHead.whiteout = 3
    \override NoteHead.layer = -1
    r2 <f' a>
  } \\ {
    \override Tie.layer = -2
    <a d,>1~ q4
  }
>>
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Of course, it would be possible to use a \tweak here:

r2 <\tweak whiteout 3 \tweak layer -1 f' a>

But that makes the code less readable and if more notes are affected it’s a pain.

IMO it would be nice if a grob’s white box on one layer wouldn’t cover grobs on the same layer; or at least not grobs of the same type on the same layer.

Would it be possible to put the whiteout part half a layer below so that \override NoteHead.layer = -1 puts the notehead at layer -1 and the surrounding white box at layer -1.5? (Or, if you want only integers: -1 puts the notehead at layer -2 and the white box at layer -3.)

David Kastrup says:

I think that's a reasonable request but it would likely require
significant restructuring of the drawing process. It would also mean
that whiteout would never work without assigning different layers. In
practice, you cannot rely on its operation in other circumstances.

https://lists.gnu.org/archive/html/bug-lilypond/2018-08/msg00050.html

[Simon Albrecht]

The problem with putting the whiteout itself 0.5 or 1 layers below is that it may well mess with use cases that already use that layer for other grobs, so I think it would rather be a matter of the drawing process intelligently handling grobs and whiteout on one layer without changing layer values. That might be somewhat difficult…

[Malte Meyn]

LilyPond allows integers only for the 'layer property, therefore the user cannot mess with layers that are 0.5 off (I probably wouldn’t use that value but “simply” draw (all) whiteouts of layer x before the rest of layer x). The following code warns about that and shows that ignoring the warning makes LilyPond ignore the override:

\version "2.19.82"

{
  \override Staff.StaffSymbol.layer = -1
  \override NoteHead.layer = -2.5
  \override NoteHead.whiteout = 5
  g'
}