#517 content_offset in directives inside grid/simple tables are off

[Felix Fontein]

When parsing a directive in a grid table or simple table, the directive's content_offset is off by one. (When nesting such a table in another such a table, it's off by one for every nesting level.)

For example, for the following code block directives, content_offset is always off from the real content line where the code block's content is in:

content_offset is off by 1:

+----------------------+
| .. code-block:: yaml |
|                      |
|    - foo             |
+----------------------+

content_offset is off by 2:

+------------------------+
|+----------------------+|
|| .. code-block:: yaml ||
||                      ||
||    - foo             ||
|+----------------------+|
+------------------------+

content_offset is off by 3:

+--------------------------+
|+------------------------+|
||+----------------------+||
||| .. code-block:: yaml |||
|||                      |||
|||    - foo             |||
||+----------------------+||
|+------------------------+|
+--------------------------+

content_offset is off by 1:

=====  =====
col 1  col 2
=====  =====
1      .. code-block:: yaml

          - foo
=====  =====

content_offset is off by 2:

=====  =====
col 1  col 2
=====  =====
1      =====  =====
       col 1  col 2
       =====  =====
       1      .. code-block:: yaml

                 - foo
       =====  =====
=====  =====

(I've noticed this when registering an own code block directive to find the actual location - row and column - of the code block's contents, to be able to give more precise error messages when linting.)

[Günter Milde]

Thank you for the report.
Could you attach your test-directive, so that I can experiment and try a fix?

[Felix Fontein]

I've attached a small script which shows the problem. It shows the line indicated by self.content_offset surrounded by some source lines for every code block.

Like here you can see that content_offset is correct outside a table:

     1 ''
     2 '.. code-block:: yaml'
     3 ''
 ->  4 '   - foo'
     5 ''
     6 'content_offset is off by 1:'
     7 ''

And here it is off by one when inside a grid table:

     9 '| .. code-block:: yaml |'
     10 '|                      |'
     11 '|    - foo             |'
 ->  12 '+----------------------+'
     13 ''
     14 'content_offset is off by 2:'
     15 ''

[Günter Milde]

Thanks for the test script. The attached variant was used to explore the problem and find the
spot that needs to be changed.
The upcoming commit [r10280] will fix the problem (and also wrong line numbers in some system messages for markup errors inside table cells.

[Felix Fontein]

Thanks for looking into this! Unfortunately with your patch, the result of my test script seems unchanged.

This change fixes the issue for me, but I'm not sure whether it is correct:

--- a/docutils/docutils/parsers/rst/states.py
+++ b/docutils/docutils/parsers/rst/states.py
@@ -1950,7 +1950,7 @@ def build_table_row(self, rowdata, tableline):
             entry = nodes.entry(**attributes)
             row += entry
             if ''.join(cellblock):
-                self.nested_parse(cellblock, input_offset=tableline+offset,
+                self.nested_parse(cellblock, input_offset=tableline+offset-1,
                                   node=entry)
         return row

[Felix Fontein]

Oh sorry, I missed the part on "upcoming commit"... I thought the commit you added earlier today (r10279) already fixed the issue...

[Günter Milde]

The fix is out.
Sorry for the confusion: I intended to publish 2 commits with one git svn dcommit
but there was a timeout after the first :(

[Günter Milde]

Line numbers are still wrong in the nested parsing of cell content in "CSV tables".
Generally, source line numbers are only intended for ease of debugging and should not be relied upon too heavily.

[Felix Fontein]

Thanks for fixing this, I can confirm that with the latest commit it works fine (except tables lists, as you mentioned - but these are not very common, at least the ones with code blocks in them :) )!

I'm using the line numbers to try to identify the exact position of the code block (including its column offset), mainly for two use-cases:
1. Being able to show more precise error locations when linting code in code blocks;
2. To be able to replace code block content's with output (extracted from running the code in other code blocks or somewhere else) to make sure that the output shown in documentation is correct. (Obviously this only works if the exact position can be located and replacing the content is easy, which is usually the case for code blocks in list tables or outside of tables.)

If you're interested in it, the location code is here: https://github.com/felixfontein/antsibull-docutils/blob/main/src/antsibull_docutils/rst_code_finder.py (it's a bit more involved than the example code for this issue).
A use (of that code) for calling yamllint in code blocks is here: https://github.com/ansible/ansible-documentation/blob/devel/tests/checkers/rst-yamllint.py
And another use of that code for replacing code block contents is here: https://github.com/ansible-community/antsibull-docs/tree/main/src/antsibull_docs/ansible_output (documentation with examples: https://docs.ansible.com/projects/antsibull-docs/ansible-output/)