#67 latex2e: recorddeps to images rel to CWD (+ lots of fixes)

open
nobody
None
5
2010-03-01
2010-03-01
Kirill Smelkov
No

Subject: [PATCH 1/8] Node.copy: don't forget to also setup .document/.source/.line in duplicate

Spot this by debugging Substitute transformation which does .deepcopy()
to substitute definitions.

This will be needed in the next patch, where I'll be teaching Inliner to
propagate .line for `inline' elements.
---
docutils/docutils/nodes.py | 23 +++++++++++++++++++----
1 files changed, 19 insertions(+), 4 deletions(-)

Discussion

1 2 > >> (Page 1 of 2)
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    • summary: Node.copy: don't forget to also setup .document/.source/.lin --> latex2e: recorddeps to images rel to CWD (+ lots of fixes)
     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 2/8] rst: teach Inliner to propagate .source & .line for inline elements

    In particular this will be needed for substitution_reference in
    combination with image substitution_definition, latex2e backend and
    --record-dependencies. e.g. for

    some/file.txt:
    .. |up| image:: pics/up.png

    text text text |up| ...

    when generating dependencies to pics/up.png, writer will want to prepend
    document source path to pics/up.png so that the prerequisite becomes
    some/pics/up.png , and for this it needs .source attribute in image
    node.

    This changes one test in test_substitutions.py:

    input:
    .. |indirect1| replace:: |indirect2|
    .. |indirect2| replace:: |Sub|
    .. |sub| replace:: |indirect1|

    Use |sub| and |indirect1| and |sub| again (and |sub| one more time).

    -: expected
    +: output

    [...]

    <system_message backrefs="id6" ids="id5" level="3" line="5" source="test data" type="ERROR">
    <paragraph>
    Circular substitution definition referenced: "sub".
    - <system_message backrefs="id8" ids="id7" level="3" source="test data" type="ERROR">
    + <system_message backrefs="id8" ids="id7" level="3" line="2" source="test data" type="ERROR">
    ? +++++++++

    Look, previsouly the ERROR has not line=... at all!

    But that line is incorrect - it should be 5. The problem here is with
    Substitution transform incorrectly reporing it -- I'll fix it in the
    next patch.
    ---
    docutils/docutils/parsers/rst/states.py | 6 +++++-
    .../test/test_transforms/test_substitutions.py | 2 +-
    2 files changed, 6 insertions(+), 2 deletions(-)

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 3/8] Substitutions: unroll problematic refs to their origin in document when reporting errors

    This is needed to correctly report erroneous line for substitution
    references. For example for

    .. |indirect1| replace:: |indirect2|
    .. |indirect2| replace:: |Sub|
    .. |sub| replace:: |indirect1|

    Use |sub| and |indirect1| and |sub| again (and |sub| one more time).

    we now correctly report line info for 3rd substitutrion reference to |sub|
    being at line 5 instead of 2.

    The problem with previous approach was that we were iteratively expanding refs
    to defs, and then again defs to refs and so on, but during that process,
    source/line info for nodes were changing accordingly, e.g. when first |sub|
    gets replaced with |inderect1|, resulting node line is 3, then |indirect1| is
    replaced with |indirect2| and node line is 1, and then |indirect2| is replaced
    to |sub| (node line is 2) and boom, we hit the cycle (this is right), but the
    line number (2) was wrong.

    In order to avoid that, let's just store back refs to original substitution
    reference, so that when there is a cycle, we can unroll it to the original
    problematic node and report right info.
    ---
    docutils/docutils/transforms/references.py | 8 +++++++-
    .../test/test_transforms/test_substitutions.py | 2 +-
    2 files changed, 8 insertions(+), 2 deletions(-)

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 4/8] Node.setup_child: teach it to perform recursive setup of document/source/line in grandchildren

    neccessary e.g. for `node` in cases like:
    1. create node
    2. create container
    3. container += node
    4. document += container

    because in such cases, node would be left without document/source/line
    info attached.

    And also, in parser/rst/states.py, we manually set .line in some places,
    but .source is not initalized explicitely anywhere, and this for e.g.

    ---- 8< q.txt ----
    .. include:: s.txt
    ---- 8< ----

    ---- 8< s.txt ----

    reStructuredText Test Document

    Footnotes
    ---------

    .. [4] Here's an unreferenced footnote, with a reference to a
    nonexistent footnote: [5]_.

    text
    ---- 8< ----

    results in a very weird behaviour -- it says there is a reference to
    unknown target "5", on line 8, but file is q.txt insetead of s.txt!

    So we have to ("option 1") either manually patch every place where .line
    is setup to also setup .source (and .document), or better provide some
    way for nodes to automatically do it. One way would be to use something
    like document.new_node(node_type, *args), like we create document not
    through document.__init__, but through utils.new_document(). And that
    new_node() would also setup current node context (document/source/line/
    etc...) so we'll avoid such problems at all.

    Or, try to restore source/line context for grandchildren in
    Node.setup_child().

    Right now I prefer second change -- it's less intrusive, but in the
    longer run, option 1 is more viable...

    This is a really important change. Unfortunatelly it also makes a lot of
    tests fail.

    Some of the test failures are for the better - one example of this is
    e.g. standalone_rst_html4css1.py where in ERROR we were reporting
    *incorrect* file (functional/input/standalone_rst_html4css1.txt, line
    363 instead of functional/input/data/standard.txt)!

    However, where source changed from standalone_rst_html4css1.txt to
    standard.txt, there is no line info now... I'll restore it later though.

    Other failures look like this:

    functional/input/data/standard.txt:

    ---- 8< ----
    367 .. [CIT2002] Citations are text-labeled footnotes. They may be
    368 rendered separately and differently from footnotes.
    369
    370 Here's a reference to the above, [CIT2002]_, and a [nonexistent]_
    371 citation.
    372
    373 .. _Another Target:

    - <system_message backrefs="id82" ids="id81" level="3" line="372" source="functional/input/data/standard.txt" type="ERROR">
    + <system_message backrefs="id82" ids="id81" level="3" line="371" source="functional/input/data/standard.txt" type="ERROR">
    <paragraph>
    Unknown target name: "nonexistent".
    ---- 8< ----

    ---- 8< ----
    input:
    Duplicate external target_'s (different URIs):

    .. _target: first

    .. _target: second

    -: expected
    +: output
    <document source="test data">
    <paragraph>
    Duplicate external
    <problematic ids="id3" refid="id2">
    target_
    's (different URIs):
    <target dupnames="target" ids="target" refuri="first">
    <system_message backrefs="id1" level="2" line="5" source="test data" type="WARNING">
    <paragraph>
    Duplicate explicit target name: "target".
    <target dupnames="target" ids="id1" refuri="second">
    - <system_message backrefs="id3" ids="id2" level="3" line="1" source="test data" type="ERROR">
    ? ^
    + <system_message backrefs="id3" ids="id2" level="3" line="2" source="test data" type="ERROR">
    ? ^
    ---- 8< ----

    or

    ---- 8< ----
    input:
    Unknown reference_.

    -: expected
    +: output
    <document source="test data">
    <paragraph>
    Unknown
    <problematic ids="id2" refid="id1">
    reference_
    .
    - <system_message backrefs="id2" ids="id1" level="3" line="1" source="test data" type="ERROR">
    ? ---------
    + <system_message backrefs="id2" ids="id1" level="3" source="test data" type="ERROR">
    <paragraph>
    Unknown target name: "reference".
    ---- 8< ----

    etc...

    But I still insist that modulo "option 1" this change is *right*. It's
    just that there are lots of other bugs still present in docutils.

    We now have 11 test failures. I'll address them all in the next patches.
    ---
    docutils/docutils/nodes.py | 11 +++++++++++
    .../expected/standalone_rst_html4css1.html | 2 +-
    .../functional/expected/standalone_rst_latex.tex | 2 +-
    .../expected/standalone_rst_pseudoxml.txt | 2 +-
    4 files changed, 14 insertions(+), 3 deletions(-)

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 5/8] rst: teach Inliner to propagate .line for reference elements

    This fixes 5 of 11 failing testcases, e.g. this

    input:
    Duplicate external target_'s (different URIs):

    .. _target: first

    .. _target: second

    -: expected
    +: output
    <document source="test data">
    <paragraph>
    Duplicate external
    <problematic ids="id3" refid="id2">
    target_
    's (different URIs):
    <target dupnames="target" ids="target" refuri="first">
    <system_message backrefs="id1" level="2" line="5" source="test data" type="WARNING">
    <paragraph>
    Duplicate explicit target name: "target".
    <target dupnames="target" ids="id1" refuri="second">
    - <system_message backrefs="id3" ids="id2" level="3" line="1" source="test data" type="ERROR">
    ? ^
    + <system_message backrefs="id3" ids="id2" level="3" line="2" source="test data" type="ERROR">
    ? ^
    <paragraph>
    Duplicate target name, cannot be used as a unique reference: "target".

    and this

    input:
    Unknown reference_.

    -: expected
    +: output
    <document source="test data">
    <paragraph>
    Unknown
    <problematic ids="id2" refid="id1">
    reference_
    .
    - <system_message backrefs="id2" ids="id1" level="3" line="1" source="test data" type="ERROR">
    ? ---------
    + <system_message backrefs="id2" ids="id1" level="3" source="test data" type="ERROR">
    <paragraph>
    Unknown target name: "reference".

    and the like...

    Still 6 failures...
    ---
    docutils/docutils/parsers/rst/states.py | 1 +
    1 files changed, 1 insertions(+), 0 deletions(-)

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 6/8] rst: teach Inliner to propagate .line for footnote_reference elements

    This fixes 3 failures like

    input:
    Duplicate manual footnote labels, with reference ([1]_):

    .. [1] Footnote.

    .. [1] Footnote.

    -: expected
    +: output
    <document source="test data">
    <paragraph>
    Duplicate manual footnote labels, with reference (
    <problematic ids="id5 id1" refid="id4">
    [1]_
    ):
    <footnote dupnames="1" ids="id2">
    <label>
    1
    <paragraph>
    Footnote.
    <footnote dupnames="1" ids="id3">
    <label>
    1
    <system_message backrefs="id3" level="2" line="5" source="test data" type="WARNING">
    <paragraph>
    Duplicate explicit target name: "1".
    <paragraph>
    Footnote.
    - <system_message backrefs="id5" ids="id4" level="3" line="1" source="test data" type="ERROR">
    ? ^
    + <system_message backrefs="id5" ids="id4" level="3" line="2" source="test data" type="ERROR">
    ? ^
    <paragraph>
    Duplicate target name, cannot be used as a unique reference: "1".

    And also, as I promised, this restores line info in ERROR where file was
    incorrect.

    Still 3 failures...

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 7/8] rst: teach Inliner to propagate .line in interpreted_or_phrase_ref()

    This fixes 3 last failures like

    test_functional.py: functional/tests/standalone_rst_pseudoxml.py; test (test_functional.FunctionalTestCase):
    --- functional/expected/standalone_rst_pseudoxml.txt
    +++ functional/output/standalone_rst_pseudoxml.txt
    @@ -1884,7 +1884,7 @@
    <system_message backrefs="id82" ids="id81" level="3" line="372" source="functional/input/data/standard.txt" type="ERROR">
    <paragraph>
    Unknown target name: "nonexistent".
    - <system_message backrefs="id84" ids="id83" level="3" line="399" source="functional/input/data/standard.txt" type="ERROR">
    + <system_message backrefs="id84" ids="id83" level="3" line="398" source="functional/input/data/standard.txt" type="ERROR">

    i.e. we've got all that line numbers destabilized by Nodes.setup_child()
    change back.
    ---
    docutils/docutils/parsers/rst/states.py | 7 +++++--
    1 files changed, 5 insertions(+), 2 deletions(-)

     
  • Kirill Smelkov
    Kirill Smelkov
    2010-03-01

    Subject: [PATCH 8/8] latex2e: record_dependencies to images relative to CWD, not document source

    So that e.g. for

    --- path/to/1.txt ---
    .. image:: pics/start.png
    ---------------------

    rst2latex --record-dependencies path/to/1.txt

    dependencies are

    path/to/pics/start.png

    not
    pics/start.png

    This is the same behaviour we have in Include directive and probably
    others, e.g. for

    --- path/to/3.txt ---
    .. include:: 4.txt
    ---------------------

    rst2<whatever> --record-dependencies path/to/3.txt

    dependencies are

    path/to/4.txt

    This is the only consistent way which is meaningful for make(1), because
    if otherwise if we treat paths and "urls" in record-dependencies
    differently, the result is not suitable for project builds.

    --------

    Please pay attention, that this patch won't work without all ther other
    patches I posted before. The problem was there was no good accounting
    for source/line info in Docutils -- lot's of nodes ended up being
    created and processed without .source / .line attributes, and this
    creates problem at ``node.source``
    ---
    docutils/docutils/writers/latex2e/__init__.py | 5 ++++-
    docutils/test/test_dependencies.py | 3 +--
    2 files changed, 5 insertions(+), 3 deletions(-)

     
1 2 > >> (Page 1 of 2)