Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#202 Docutils 0.10 snapshot doesn't allow user created writers

closed
nobody
None
5
2012-08-22
2012-08-20
Toshio Kuratomi
No

Working on a snapshot since we're going to need to support python-3.3 for the next version of Fedora. A mercurial maintainer noticed that the behaviour of docutils has changed. mercurial has a locally created writer to take in rst and output manpages. With the snapshot they are unable to get docutils to load this:
https://bugzilla.redhat.com/show_bug.cgi?id=849744

The change that made this is:
http://docutils.svn.sourceforge.net/viewvc/docutils/trunk/docutils/docutils/writers/__init__.py?r1=7317&r2=7486

Which is to fix this bug on python3.3:
https://sourceforge.net/tracker/?func=detail&aid=3541369&group_id=38414&atid=422030

It looks like the python-3.3 change is that python3.3 no longer allows __import__ to take a level argument of -1. That argument allowed __import__ to search both absolute and relative paths for the module. The current code fixed that by changing level to "1" which only searches a relative path for the module. This works for the modules inside of docutils because they are located in the parent package of the code that loads them. Out-of-tree modules break with this scheme because the module is not a child of the docutils package.

To fix this the code can do something like this:

try:
module = __import__(writer_name, globals(), locals(), level=0)
except ImportError:
module = __import__(writer_name, globals(), locals(), level=1)

What this does is first look for the module as an absolute import. This will be triggered if it's a user defined module. If no module is found as an absolute import it then looks in the parent of the module that called it. This is where all current modules shipped with docutils are called.

This code is what is closest to level=-1 which is present in python2.x and python3 up to python3.2.

It might make sense to revisit hardcoded's patch to issue 3541369 to implement this as the try: except is boilerplate in all the places we need to use __import__.

Discussion

  • 1. have to get a test
    2. have to get py3.3
    3. test from 2.4 to 3.3/trunk

     
  • Do you want a test case suitable for a unittest or an end-to-end test?

    This is the simplest way I found to test:
    mkdir cleantest
    cd cleantest
    cp /usr/lib/python2.7/site-packages/docutils/writers/manpage.py testwriter.py
    python
    >>> from docutils.writers import get_writer_class
    >>> get_writer_class('manpage')
    <class docutils.writers.manpage.Writer at 0x7f94ee0aba78>
    >>> get_writer_class('testwriter')
    <class testwriter.Writer at 0x7f94eea0bc18>

    Under the current snapshot, get_writer_class('testwriter') will fail with a traceback.

    For end to end testing, this is how mercurial is using it:

    They have a script runrst and a custom writer hgmanpage.py which is a slightly modified version of the docutils builtin manpage.py. The runrst script is invoked twice. Once to create man pages using their custom hgmanpage.py writer and the second time to generate html using docutil's builtin html writer (docutils/writers/html4css1.py).

    The runrst script uses docutils.core.publish_cmdline() to make use of the writers.

    Under the current snapshot the hgmanpage.py module is not found by docutils and a traceback is raised.
    To see how all this works:

    wget http://www.selenic.com/mercurial/release/mercurial-2.3.tar.gz
    tar -xzvf mercurial-2.3.tar.gz
    cd mercurial-2.3/doc
    make all

    Note that I haven't tested this with python3.3 yet -- because of this bug: https://sourceforge.net/tracker/?func=detail&aid=3555164&group_id=38414&atid=422030 we don't yet have a docutils package built for python3.3. Reading the python3.3 documentation, the approach in my iniital comment shouldn't be using functionality that was removed from __import__() though.

     
  • a unittest that fits into docutils tests would be the ultimate
    but this might be too much hassle

    ::

    cd docutils-trunk/docutils
    ../../cpython-HEAD/python.exe setup.py build
    ../../cpython-HEAD/python.exe test3/alltest.py

    works (4 errors in odt-writer and alot of warnings ::

    ResourceWarning: unclosed file <_io.TextIOWrapper name='functional/tests/_default.py' mode='r' encoding='UTF-8'>
    defaultpy = open(join_path(datadir, 'tests', '_default.py')).read()

    but docutils seams to work

     
  •  
    Attachments
  • 1. cd docutils
    2. cp test_get_writer_class.py test
    3. passes::

    python2.6 test/test_get_writer_class.py

    4. for py3 set PYTHONPATH to build/lib::

    export PYTHONPATH=`pwd`/build/lib/:
    cp test_get_writer_class.py test3
    python3.2 test3/test_get_writer_class.py

    Traceback (most recent call last):
    File "test3/test_get_writer_class.py", line 7, in <module>
    wr = get_writer_class(t[0])
    File "/Users/engelbert/projects/docutils/trunk/docutils/build/lib/docutils/writers/__init__.py", line 136, in get_writer_class
    module = __import__(writer_name, globals(), locals(), level=1)
    ImportError: No module named dummy-writer

    python3.3 is the same

     
  • If you commit your test_get_writer_class.py I can take a look. Also note -- this is going to affect all the code touched by revision 7486, not just get_writer_class().

     
  • Ah sorry -- I missed that the test_writer and dummy-writer are attached to this report. /me downloads them now.

     
  • The test case fails in the same way as on python3 for me. Using python2.7 but I'm wondering if the difference is that you didn't set PYTHONPATH when running your python2 test? The system docutils is likely the last release so it would pass the test (using level=-1). You would need to make sure that you're getting the docutils module from the snapshot to get the traceback.

     
  • $ export PYTHONPATH=/srv/git/python-docutils/docutils
    $ python2.7 test_get_writer_class.py (08:56:22):1225
    Traceback (most recent call last):
    File "test_get_writer_class.py", line 7, in <module>
    wr = get_writer_class(t[0])
    File "/srv/git/python-docutils/docutils/docutils/writers/__init__.py", line 136, in get_writer_class
    module = __import__(writer_name, globals(), locals(), level=1)
    ImportError: No module named dummy-writer

    Also note: For the actual unittest, probably want to add an else: to the try: except and assert if t[1] == 0 in the else.

     
  • you are correct, 2.7 fails the same
    but 2.3 to 2.6 passes

     
  • Happens for me on python2.6:

    $ PYTHONPATH=$(pwd)
    $ export PYTHONPATH
    $ python2.6 test_get_writer_class.py
    Traceback (most recent call last):
    File "test_get_writer_class.py", line 7, in <module>
    wr = get_writer_class(t[0])
    File "/home/fedora/toshio/docutils-0.10/docutils/writers/__init__.py", line 136, in get_writer_class
    module = __import__(writer_name, globals(), locals(), level=1)
    ImportError: No module named dummy-writer
    $ ls dummy-writer* test_get_writer*
    dummy-writer.py dummy-writer.pyc test_get_writer_class.py
    $ pwd
    /home/fedora/toshio/docutils-0.10/test

    python2.4 and below should definitely work because there's no level argument to __import__() there [So we end up using the docutils._compat.__import__ code that ignores level]

     
  • hei toshio

    you wrote ---------
    > Happens for me on python2.6:
    >
    > $ PYTHONPATH=$(pwd)
    > $ export PYTHONPATH
    > $ python2.6 test_get_writer_class.py
    > Traceback (most recent call last):
    > File "test_get_writer_class.py", line 7, in <module>
    > wr = get_writer_class(t[0])
    > File "/home/fedora/toshio/docutils-0.10/docutils/writers/__init__.py",
    > line 136, in get_writer_class
    > module = __import__(writer_name, globals(), locals(), level=1)
    > ImportError: No module named dummy-writer
    > $ ls dummy-writer* test_get_writer*
    > dummy-writer.py dummy-writer.pyc test_get_writer_class.py
    > $ pwd
    > /home/fedora/toshio/docutils-0.10/test

    you are in "test" and set PYTHONPATH=$(pwd) ?

    next step for me is to get a unittest into test/test_writers (if no one objects)

    > python2.4 and below should definitely work because there's no level
    > argument to __import__() there [So we end up using the
    > docutils._compat.__import__ code that ignores level]

    i tried on linux 2.2 to 2.6 worked
    on mac 2.5 and 2.6 work

     
  • Hmmm... Copy and paste error on my part. I am in the test directory but it was actually:

    $ PYTHONPATH=$(pwd)/..
    $ export PYTHONPATH
    $ python2.6 test_get_writer_class.py

     
    • status: open --> closed