python-markdown-discuss

[Python-markdown-discuss] Python 3.0

[Yuri T. <qar...@gm...>]

I tried running python markdown the recently released py3k and after
two commits it only requires 2to3 and one manual fix. The recommended
method for maintaining a library that should work as both 2.x and 3.x
is to maintain it in 2.x and rely on automatic conversion with 2to3
with no further manual changes. So, we should think of how to fix up
our code so that it works with 2.x and then converts into
3.0-compliant code when we run 2to3.

Get the newest version from git, then run 2to3:

2to3 -w markdown
2to3 -w markdown.py
2to3 -w test-markdown.py

After that you need to make one change manually, in markdown/inlinepatterns.py:

-import htmlentitydefs
+from html import entities as htmlentitydefs

After that, the script seems to work for all test cases except
"para-with-hr". The test-markdown.py itself is broken in 3.0, though,
when it comes to saving the results to a file. As a quick fix, you can
disable this:

test-markdown.py:
-    htmlDiffFile.write(diffsBuffer.decode("utf8"))
+    #htmlDiffFile.write(diffsBuffer.decode("utf8"))

Some of the extensions fail too, though those problems might actually
be in test-markdown.py

Let's not worry about the extensions working with 3.0 for our 2.0
release, though. I'll try to fix up the test script, though it perhaps
would be ok to have a different version of it for 3.0.

I am not planning on committing the output of 2to3 to avoid confusion.
At least for now, anyone who wants to use markdown with python 3.0
should get the latest version from git and run 2to3 on it themselves.

 - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Python 3.0

[Artem Y. <se...@sp...>]

On Thu, Dec 4, 2008 at 11:42 PM, Yuri Takhteyev <qar...@gm...> wrote:

>
> After that you need to make one change manually, in
> markdown/inlinepatterns.py:
>
> -import htmlentitydefs
> +from html import entities as htmlentitydefs
>
> After that, the script seems to work for all test cases except
> "para-with-hr". The test-markdown.py itself is broken in 3.0, though,
> when it comes to saving the results to a file. As a quick fix, you can
> disable this:
>
> test-markdown.py:
> -    htmlDiffFile.write(diffsBuffer.decode("utf8"))
> +    #htmlDiffFile.write(diffsBuffer.decode("utf8"))
>
>
Nice :) What about adding version check ? Something like this:

    if sys.version >= "3.0":
        from html import entities as htmlentitydefs
    else:
        import htmlentitydefs

Re: [Python-markdown-discuss] Python 3.0

[Yuri T. <qar...@gm...>]

> Nice :) What about adding version check ? Something like this:
>
>     if sys.version >= "3.0":
>         from html import entities as htmlentitydefs
>     else:
>         import htmlentitydefs

We shouldn't _have_ to do that, but works for now. I also realized
that "para-with-hr" already fails, so the new problem is just limited
to test-markdown.py. (Test-markdown.py fails in python 3.0 when it
tries to save diff for a failed test.)

So, with the most recent commit (cd73ef7), all you have to do is run
2to3 on "markdown", "markdown.py" and (optionally) "test-markdown.py".

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Thu, Dec 4, 2008 at 5:50 PM, Yuri Takhteyev <qar...@gm...> wrote:
>> Nice :) What about adding version check ? Something like this:
>>
>>     if sys.version >= "3.0":
>>         from html import entities as htmlentitydefs
>>     else:
>>         import htmlentitydefs

Seeing the code won't run as-is (the user still needs to run 2to3) in
3.0, I do *not* think we should be adding this version check.
Obviously, if we are supporting 3.0 (through 2to3) that should be
documented and a note about this could be made there. I suppose a
comment in the source could help as well.

    import htmlentitydefs
    # For Python 3.0+ comment above line and uncomment line below
    # from html import entities as htmlentitydefs

>
> We shouldn't _have_ to do that, but works for now. I also realized
> that "para-with-hr" already fails,

Hmm, it's not failing for me in 2.5. I haven't installed 2.6 yet. My
understanding is that that is where we should be testing before
running 2to3. I'll have to investigate further.

> so the new problem is just limited
> to test-markdown.py. (Test-markdown.py fails in python 3.0 when it
> tries to save diff for a failed test.)
>
> So, with the most recent commit (cd73ef7), all you have to do is run
> 2to3 on "markdown", "markdown.py" and (optionally) "test-markdown.py".
>

Cool!



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[Yuri T. <qar...@gm...>]

> Seeing the code won't run as-is (the user still needs to run 2to3) in
> 3.0, I do *not* think we should be adding this version check.
> Obviously, if we are supporting 3.0 (through 2to3) that should be
> documented and a note about this could be made there. I suppose a
> comment in the source could help as well.

When we make a release, the user should be able to download a py3k
version and just install it.  We just shouldn't keep the py3k version
in our git tree. Our git tree should have 3.0-friendly code that runs
as is with python 2.3, 2.4, 2.5 and 2.6. (Again, we can drop 2.3 at
some point, if there is a good reason.) "3.0-friendly" means that if
you run it through 2to3, it would work with py3k without further
changes. People who want to contribute patches should always do so
against the 2.x version. When we make a release, however, we can
publish two versions simultaneously, from the same code base:

git -> zip -> "markdown-2.0.zip"
git -> 2to3 -> zip -> "markdown-2.0-py3k.zip"

We don't _have_ to do this for 2.0, since at this point people who
want to run py3k will know how to run 2to3, but we'll have to get to
doing this eventually and we might as well start now. (It doesn't look
like we have to do much additional work for this.)

> Hmm, it's not failing for me in 2.5. I haven't installed 2.6 yet. My
> understanding is that that is where we should be testing before
> running 2to3. I'll have to investigate further.

It fails for me in 2.5.  BTW, it looks like we (actually, probably I)
broke 2.4 compatibility at some point.

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Thu, Dec 4, 2008 at 11:21 PM, Yuri Takhteyev <qar...@gm...> wrote:
[snip]
> (Again, we can drop 2.3 at some point, if there is a good reason.)
[snip]

Fyi, Django is currently talking about dropping 2.3 support [1]. It
hasn't been decided conclusively yet with some devs on either side of
the fence, but they're looking at it. I'll report back on this when I
know more. Obviously, we should not do so just because some other
package does, but the Django team is very attune to the community at
large and a nice way for a small project like ours to gauge these
sorts of things. There's some interesting points both for and against
in that discussion.

I have to wonder if the jump to 2.0 is enough for us to warrant
dropping 2.3 support given other, larger projects are looking at doing
the same. After all, having a major release (x.0) is a strange place
to start issuing depreciation warnings (as some suggest for Django)
for 2.3 specific code that will be removed in x.1. That seems silly to
me. If we're going to do it, now is as good a time as any. I suppose
we'll have to see how things play out for others.

[1]: http://groups.google.com/group/django-developers/browse_thread/thread/dcf70c736c204594

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Thu, Dec 4, 2008 at 2:42 PM, Yuri Takhteyev <qar...@gm...> wrote:
>
> After that you need to make one change manually, in markdown/inlinepatterns.py:
>
> -import htmlentitydefs
> +from html import entities as htmlentitydefs
>

Hmm, on further thought and a little research, this is a known change
[1], so shouldn't 2to3 be making this change for us? If not, is that a
bug in 2to3? I did a search and haven't found it reported anywhere.

[1]: http://www.python.org/dev/peps/pep-3108/#id62

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Thu, Dec 4, 2008 at 11:21 PM, Yuri Takhteyev <qar...@gm...> wrote:
>> Hmm, it's not failing for me in 2.5. I haven't installed 2.6 yet. My
>> understanding is that that is where we should be testing before
>> running 2to3. I'll have to investigate further.
>
> It fails for me in 2.5.  BTW, it looks like we (actually, probably I)
> broke 2.4 compatibility at some point.
>

Ok, I got 2.6 and 3.0 installed today. All tests are passing for me in
2.5 & 2.6 (including extensions) and all non-extension tests are
passing in 3.0. I should also note that the doctests in extensions
(run regression-tests.py) that have them *are* passing in 3.0 (after
removing the u from string output in the tests), so I suspect it's a
problem with test-markdown.py.

I am getting one small anomaly in 2.3 & 2.4 with the "uche" test,
although it appears to only be insignificant whitespace. The
regression tests run fine except for doctests for the 'headerid' and
'wikilinks' extensions in 2.3 - although the syntax tests are passing
for both.

Strangely, I don't see a "para-with-hr" test anywhere. Is this perhaps
some old test you never committed and it isn't passing after some
recent changes?


-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

> I am getting one small anomaly in 2.3 & 2.4 with the "uche" test,
> although it appears to only be insignificant whitespace

Figured this one out. I appears that the brand new versions of
ElementTree (newer than what ships with 2.5, 2.6 & 3.0) I installed
for 2.3 & 2.4 is encoding the line break in that test (as ``
``)
while older versions left it as-is (``\n``).

As those linebreaks are insignificant, I am just removing them.

> Hmm, perhaps we should wrap that test
> in a conditional and only run it when those conditions (Pygments and
> version > 3.0) are met.

I've also committed a somewhat hacky patch for this. It occurs to me
that perhaps the tester should be able to do this, but the API is not
so clear. We could pass a list of dependant packages to the
test_directory method. Something like:

    tester.test_directory("tests/extensions-x-codehilite", depends=["pygments"])

Then the tester would only run the test if it can successfully import
each package in the list. Perhaps a message should be printed that the
test was skipped and why.

Any thoughts?

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[John S. <jo...@sz...>]

On Sat, Dec 6, 2008 at 10:38 PM, Waylan Limberg <wa...@gm...> wrote:
[snip]
> Strangely, I don't see a "para-with-hr" test anywhere. Is this perhaps
> some old test you never committed and it isn't passing after some
> recent changes?

It's in tests/misc.  That's one I submitted to you guys a couple months ago.

-John

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Sun, Dec 7, 2008 at 3:52 AM, John Szakmeister <jo...@sz...> wrote:
> On Sat, Dec 6, 2008 at 10:38 PM, Waylan Limberg <wa...@gm...> wrote:
> [snip]
>> Strangely, I don't see a "para-with-hr" test anywhere. Is this perhaps
>> some old test you never committed and it isn't passing after some
>> recent changes?
>
> It's in tests/misc.  That's one I submitted to you guys a couple months ago.
>

Thanks John, and Yuri for commiting it. However, the test was failing
because the *test* was wrong, not the code. That was not valid html. A
<hr> is a block level element and no block level elements are allowed
in <p>s. I have updated the test and it passes. In other words the
code was correct. I thought I had tested for that when refactoring the
core - which explains my confusion about it failing.

Fyi Yuri, I also fixed your missing imports in markdown.py after you
rearranged the logger.



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Python 3.0

[John S. <jo...@sz...>]

On Sun, Dec 7, 2008 at 12:34 PM, Waylan Limberg <wa...@gm...> wrote:
[snip]
> Thanks John, and Yuri for commiting it. However, the test was failing
> because the *test* was wrong, not the code. That was not valid html. A
> <hr> is a block level element and no block level elements are allowed
> in <p>s. I have updated the test and it passes. In other words the
> code was correct. I thought I had tested for that when refactoring the
> core - which explains my confusion about it failing.

Heh.  I was certain I looked that up... especially since I found
confusing documentation on whether it was a block-level or span-level
tag.  Sorry for the broken test.

-John

Re: [Python-markdown-discuss] Python 3.0

[Yuri T. <qar...@gm...>]

Thanks. What about the extensions-x-codehilite/code.txt test. I am
getting a diff on this one. Do you?

Does anyone have any other tests to contribute?

- yuri

On Sun, Dec 7, 2008 at 9:34 AM, Waylan Limberg <wa...@gm...> wrote:
> On Sun, Dec 7, 2008 at 3:52 AM, John Szakmeister <jo...@sz...> wrote:
>> On Sat, Dec 6, 2008 at 10:38 PM, Waylan Limberg <wa...@gm...> wrote:
>> [snip]
>>> Strangely, I don't see a "para-with-hr" test anywhere. Is this perhaps
>>> some old test you never committed and it isn't passing after some
>>> recent changes?
>>
>> It's in tests/misc.  That's one I submitted to you guys a couple months ago.
>>
>
> Thanks John, and Yuri for commiting it. However, the test was failing
> because the *test* was wrong, not the code. That was not valid html. A
> <hr> is a block level element and no block level elements are allowed
> in <p>s. I have updated the test and it passes. In other words the
> code was correct. I thought I had tested for that when refactoring the
> core - which explains my confusion about it failing.
>
> Fyi Yuri, I also fixed your missing imports in markdown.py after you
> rearranged the logger.
>
>
>
> --
> ----
> Waylan Limberg
> wa...@gm...
>



-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Python 3.0

[Waylan L. <wa...@gm...>]

On Mon, Dec 8, 2008 at 4:52 AM, Yuri Takhteyev <qar...@gm...> wrote:
> Thanks. What about the extensions-x-codehilite/code.txt test. I am
> getting a diff on this one. Do you?

I did until I installed Pygments for each version of Python, which
that extension depends on. Note that the extension does have a
fallback if Pygments is not available, but, obviously, the output is
different (not highlighted) so the test fails. Pygments has not yet
been released for 3.0 and I'm not about to try running 2to3 on it, so
it will continue to fail there. Hmm, perhaps we should wrap that test
in a conditional and only run it when those conditions (Pygments and
version > 3.0) are met.


-- 
----
Waylan Limberg
wa...@gm...