python-markdown-discuss — A general purpose discussion list

Re: [Python-markdown-discuss] GSoC progress

[Artem <ne...@gm...>]

Waylan Limberg wrote:
> Now to find the time to play with it...
>
>   

Committed it to repository. Markdown._processPlaceholders method looks 
not very clear, I tried to implement it using iterator object or using 
re.finditer, but those approaches was quite slow, so I decided to leave 
current implementation.

Re: [Python-markdown-discuss] GSoC progress

[Yuri T. <qar...@gm...>]

> Just one question though. Why did you split convert into two methods
> (convert & markdownToTree) when all you needed to do was add one
> additional line to convert for the extra step to run inline patterns?
> We have enough methods as it is. In fact, I've considered combining
> convert and _transform as they are never called any different way.

This was my suggestion.   We'll need to decide whether to consider the
new markdownToTree() public or private, but I do think it will be
useful to clearly separate the conversion into those two steps.

> Which, makes me think - if applyInlinePatterns operates on a DOM of
> the entire document, why not make it a postprocessor? That may be a
> little drastic, but it would open up a few more possibilities for
> overriding behavior. Just a thought.

Interesting.  Not sure what exactly we would gain from it, but it
should be possible.  Let's not worry about this for now, though.  I
think the next step should be dropping all the current inline patterns
and replacing them with simple regular expression substitutions,
perhaps borrowed from Trent's code.  In fact, we should see if
applyInlinePatterns could just use a chunk of Trent's code "as is".

  - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] GSoC progress

[Waylan L. <wa...@gm...>]

Cool.

Just one question though. Why did you split convert into two methods
(convert & markdownToTree) when all you needed to do was add one
additional line to convert for the extra step to run inline patterns?
We have enough methods as it is. In fact, I've considered combining
convert and _transform as they are never called any different way.

Which, makes me think - if applyInlinePatterns operates on a DOM of
the entire document, why not make it a postprocessor? That may be a
little drastic, but it would open up a few more possibilities for
overriding behavior. Just a thought.

Now to find the time to play with it...

On Mon, Jun 30, 2008 at 7:47 PM, Artem <ne...@gm...> wrote:
> Hi all,
>
> Now Markdown.convert method uses two-step approach. 1 - it converts
> markdown to DOM tree, without applying inline patterns, 2 - it parses
> DOM tree and applying inline patterns.
>
> The main problem with inline patterns is solved. I re-implemented Inline
> patterns mechanism, using string with placeholders for DOM elements. Now
> all of those examples:
>
>    [*test*](http://example.com)
>    *[test](http://example.com)*
>    **[*test*](http://example.com)**
>    __*[test](http://example.com)*__
>
> works fine:
>
>    <p><a href="http://example.com"><em>test</em></a>
>       <em><a href="http://example.com">test</a></em>
>       <strong><a href="http://example.com"><em>test</em></a></strong>
>       <strong><em><a href="http://example.com">test</a></em></strong>
>    </p>
>
> The new version is 10-20% faster. Not all tests from the test suite is
> working now, but I'm working on it.
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

[Python-markdown-discuss] GSoC progress

[Artem <ne...@gm...>]

Hi all,

Now Markdown.convert method uses two-step approach. 1 - it converts 
markdown to DOM tree, without applying inline patterns, 2 - it parses 
DOM tree and applying inline patterns.

The main problem with inline patterns is solved. I re-implemented Inline 
patterns mechanism, using string with placeholders for DOM elements. Now 
all of those examples:

    [*test*](http://example.com)
    *[test](http://example.com)*
    **[*test*](http://example.com)**
    __*[test](http://example.com)*__

works fine:

    <p><a href="http://example.com"><em>test</em></a>
       <em><a href="http://example.com">test</a></em>
       <strong><a href="http://example.com"><em>test</em></a></strong>
       <strong><em><a href="http://example.com">test</a></em></strong>
    </p>

The new version is 10-20% faster. Not all tests from the test suite is 
working now, but I'm working on it.

Re: [Python-markdown-discuss] 2 questions: whitelisting && HTML4

[Waylan L. <wa...@gm...>]

Hello.

On Mon, Jun 30, 2008 at 2:25 PM, h s <lon...@go...> wrote:
>
> 1. I would like to disallow elements being used in comment forms such
> as headings and images, is there a way to white-/blacklist elements?

You can disallow any raw html in the input by setting `safe_mode` to
one of "replace", "remove", "escape". However, if you mean you want to
disallow markdown markup that represents certain elements, then, no,
that is not currently implemented, nor do I expect it to be. However,
with Python-Markdown's extension API it shouldn't be to hard to write
your own extension that removes the appropriate inline-patterns from
the class instance.

For more information on safe_mode scroll to the bottom of this page:
http://www.freewisdom.org/projects/python-markdown/Using_as_a_Module

For information on writing your own extension see the source as well
as this page:
http://www.freewisdom.org/projects/python-markdown/Writing_Extensions
>
> 2. I couldn´t find how to output a different format than XHTML. Is it
> possible to output HTML4?
>

Unfortunately, this is currently not possible. Python-Markdown uses
it's own DOM implementation which only outputs xhtml. Having the
option to chose difference output formats is on my wishlist, but I
haven't had time to work on a solution yet. There should be some xhtml
-> html tools out there that will do fine though. I don't recall if
ElementTree (in Python2.5 standard library) will do that or not. If
not, BeautifulSoup would be my next choice.

Hope that helps.

-- 
----
Waylan Limberg
wa...@gm...

[Python-markdown-discuss] 2 questions: whitelisting && HTML4

[h s <lon...@go...>]

Hey there!

My two questions:

1. I would like to disallow elements being used in comment forms such
as headings and images, is there a way to white-/blacklist elements?

2. I couldn´t find how to output a different format than XHTML. Is it
possible to output HTML4?

Re: [Python-markdown-discuss] Test suite extension

[Trent M. <tr...@gm...>]

>> Can you explain what all the .tags files are?
> I don't know exactly, suppose it's for some additional information, it's
> useless for us, I'll remove them in the next commit.

They are used by the test harness I use in python-markdown2. Yes, they
are useless for you.


Trent

-- 
Trent Mick
tr...@gm...

Re: [Python-markdown-discuss] Test suite extension

[Artem <ne...@gm...>]

Yuri Takhteyev wrote:
> Yes, this looks much better!
>   
Good :)
> Can you explain what all the .tags files are?  
I don't know exactly, suppose it's for some additional information, it's 
useless for us, I'll remove them in the next commit.
> Also, are those all
> just Trent's tests "as is" or did you have to adjust them in anyway?
>   
Yes, I wrote a script that reformatted Trent's tests, because there are 
some differences in line breaks between markdown and markdown2 output. 
It's in tests2/formater.py
> Anyway, the thing to do with those now is to decide which of them we
> actually want to get working and which we don't.  Also, can you add a
> file somewhere saying that those are Trent's tests and are under a
> different license from python markdown itself?
>   
Ok.

Re: [Python-markdown-discuss] Test suite extension

[Yuri T. <qar...@gm...>]

> Yes, first time I merged clone with my previous repository.
> I re-created repository. Everything seems right now.

Yes, this looks much better!

Can you explain what all the .tags files are?  Also, are those all
just Trent's tests "as is" or did you have to adjust them in anyway?
Anyway, the thing to do with those now is to decide which of them we
actually want to get working and which we don't.  Also, can you add a
file somewhere saying that those are Trent's tests and are under a
different license from python markdown itself?

  - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Test suite extension

[Artem <ne...@gm...>]

Yuri Takhteyev wrote:
> Hmm, something is seriously not right about this repository.  For some
> reason, your "first commit" shows all files as newly created.  Did you
> merge those changes in from somewhere?  Can you try to re-do this?
>   
Yes, first time I merged clone with my previous repository.
I re-created repository. Everything seems right now.

Re: [Python-markdown-discuss] Test suite extension

[Yuri T. <qar...@gm...>]

Glad to see you are starting to make commits!

> http://gitorious.org/projects/python-markdown/repos/gsoc2008

Hmm, something is seriously not right about this repository.  For some
reason, your "first commit" shows all files as newly created.  Did you
merge those changes in from somewhere?  Can you try to re-do this?
Basically, we need a commit that shows a clear diff from the previous
revision, both for the sake of maintaining history and also to allow
me to cherry-pick the commits.

Consider the difference between:

http://gitorious.org/projects/python-markdown/repos/gsoc2008/commits/356cd58e06e38ace4a4ea65ac0cbbdbe910e5dfc

http://gitorious.org/projects/python-markdown/repos/gsoc2008/commits/448c3397c0dc53f4aa8207d8a61f300768647524

In the first case we know exactly what changed.  In the second case,
it looks like everything is new.

I would suggest doing the following: copy your files somewhere else,
start with a brand new clone from mainline, then copy files into it in
groups, making incremental commits.  The smaller commits, the better.
E.g., if you fixed a bug in markdown.py, then commit that change by
itself, or together with the test cases that specifically relate to
this change.  Something like this:

    cp ../artems-modified-files/markdown.py .
    git add markdown.py
    git commit # enter a message explaining the change
    cp ../artems-modified-files/tests/
    cp -r ../artems-modified-files/tests/foo tests/
    git add ... # add specific files that need to be added
    git commit # enter a message


  - yuri



>
> tm-cases:
> ======
>
> **blockquote**
> ">" transforms to ">"
>
>    [Trent wrote]
>    > no way
>
>    [Jeff wrote]
>    > way
>
> other implementations consider ">" in this test as blockquote
>
>
> **escapes**
> problem with hash characters escaping at the end of header
>
>
> **lists2**
> ul followed by ol transforms to one ul
>
>
> **mising_link_defn**
> works fine in markdown 1.7, but don't work in markdown from repository.
>
>
> **parens_in_url_4**
> this input:
>
>    [Inline link 4 with non-escaped parens](</url(test)> "title").
>
> gives invalid output:
>
>    <p><a href="&lt;/url(test">Inline link 4 with non-escaped
> parens</a>&gt; "title").
>    </p>
>
>
> **raw_html**
> HTML comments escaping
>
>    test  <!-- *blah* -->
>
>
> **sublist_para**
> test fails, sublist with 2-spaces indentation
>
>
> **tricky_anchors**
> fails on:
>
>    [only open [bracket(/in/) text](/url/   'a title')
>
> but all the other markdown implementations pass this test.
>
>
> php-markdown-cases:
> ==============
>
>
> **Blackslash escapes**
> Seems right but differ from other markdown implementations
>
>
> **Empasis**
> inline patterns problem
>
>
> **Images (Untitled)**
> Behavior differs from others implementations on:
>
>    ![alt text](</url/>)
>
> expected:
>
>    <p><img src="/url/" alt="alt text"/>
>    </p>
>
> but gives that:
>
>    <p><img src="&lt;/url/&gt;" alt="alt text"/>
>    </p>
>
>
> **Inline HTML (Simple)**
> skips markdown paragraph
>
>
> **Inline HTML (Span)**
> don't recognize html blocks
>
>
> **Links, inline style**
>
>
> **Nesting**
> Inline patterns problem
>
>
> **Parens in URL**
>
>
> **PHP-Specific Bugs**
>
>
> **Tight blocks**
> blockquote problem
>
>
> tm-cases excluded list:
> ===============
>
> basic_safe_mode
> basic_safe_mode_escape
> auto_link_safe_mode
> code_safe_emphasis
> emacs_head_vars
> emacs_tail_vars
> footnotes
> footnotes_letters
> footnotes_markup
> footnotes_safe_mode_escape
> nested_list_safe_mode
> issue2_safe_mode_borks_markup
> issue3_bad_code_color_hack
> link_defn_spaces_in_url
> link_patterns
> link_patterns_double_hit
> link_patterns_edge_cases
> mismatched_footnotes
> nested_lists_safe_mode
> pyshell
> syntax_color
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
http://sputnik.freewisdom.org/

[Python-markdown-discuss] Test suite extension

[Artem <ne...@gm...>]

Hi,
I extended test suite with some markdown2 tests, failed tests listed below.
I didn't include some tests from tm-cases, because they use option files.
I committed version with extended test suite here: 
http://gitorious.org/projects/python-markdown/repos/gsoc2008

tm-cases:
======

**blockquote**
">" transforms to ">"

    [Trent wrote]
    > no way

    [Jeff wrote]
    > way

other implementations consider ">" in this test as blockquote


**escapes**
problem with hash characters escaping at the end of header


**lists2**
ul followed by ol transforms to one ul


**mising_link_defn**
works fine in markdown 1.7, but don't work in markdown from repository.


**parens_in_url_4**
this input:

    [Inline link 4 with non-escaped parens](</url(test)> "title").

gives invalid output:

    <p><a href="&lt;/url(test">Inline link 4 with non-escaped 
parens</a>&gt; "title").
    </p>


**raw_html**
HTML comments escaping

    test  <!-- *blah* -->


**sublist_para**
test fails, sublist with 2-spaces indentation


**tricky_anchors**
fails on:

    [only open [bracket(/in/) text](/url/   'a title')

but all the other markdown implementations pass this test.


php-markdown-cases:
==============


**Blackslash escapes**
Seems right but differ from other markdown implementations


**Empasis**
inline patterns problem


**Images (Untitled)**
Behavior differs from others implementations on:

    ![alt text](</url/>)

expected:

    <p><img src="/url/" alt="alt text"/>
    </p>

but gives that:

    <p><img src="&lt;/url/&gt;" alt="alt text"/>
    </p>


**Inline HTML (Simple)**
skips markdown paragraph


**Inline HTML (Span)**
don't recognize html blocks


**Links, inline style**


**Nesting**
Inline patterns problem


**Parens in URL**


**PHP-Specific Bugs**


**Tight blocks**
blockquote problem


tm-cases excluded list:
===============

basic_safe_mode
basic_safe_mode_escape
auto_link_safe_mode
code_safe_emphasis
emacs_head_vars
emacs_tail_vars
footnotes
footnotes_letters
footnotes_markup
footnotes_safe_mode_escape
nested_list_safe_mode
issue2_safe_mode_borks_markup
issue3_bad_code_color_hack
link_defn_spaces_in_url
link_patterns
link_patterns_double_hit
link_patterns_edge_cases
mismatched_footnotes
nested_lists_safe_mode
pyshell
syntax_color

Re: [Python-markdown-discuss] [PATCH] unwrapping urlparse result tuple manually for python 2.4 compatibility.

[Waylan L. <wa...@gm...>]

Sorry, for some reason those commits are not listed on the Gitorious
front page (or the repo's overview page) but they are listed on the
repo's commit page. Weird.

On Fri, Jun 6, 2008 at 8:28 AM, Waylan Limberg <wa...@gm...> wrote:
> Hey Yuri, I don't see where this was pushed to Gitorious. What am I missing?
>
> On Fri, May 30, 2008 at 6:18 PM, Yuri Takhteyev <qar...@gm...> wrote:
>> Thanks.  I pushed this commit.  Sorry about this, not sure when this
>> has slipped in.  My goal has been to keep compatibility not only with
>> 2.4 but also with 2.3.  I also noticed that while markdown.py works
>> with 2.3 (well used to, and now again with your patch), the
>> test-markdown.py script did not.  So, I changed test-markdown.py to
>> work with 2.3.
>>
>> So, let's all agree that test-markdown.py should be ran with
>> python2.3, python2.4 and python2.5 after each change.  If you cannot
>> check it with all three, then make a commit in a branch and email it
>> to me so that I could test before merging.
>>
>> Also, Waylan and David, can you check in test cases for the fixes you
>> made in the recent months?
>>
>> BTW, I also noticed that we have a regression on one of the tests.  I
>> suspect this might be the price we are paying for some of the other
>> issues that Waylan fixed.  Hopefully we'll resolve this with Artem's
>> help over the next few months.
>>
>>  - yuri
>>
>> On Fri, May 30, 2008 at 2:22 PM, Jeff Balogh <its...@gm...> wrote:
>>> In 2.5 the result is wrapped in a ParseResult class so that fields can be
>>> accessed by name, but a plain tuple is returned in 2.4.
>>> ---
>>>
>>> I don't think I've seen any patches on the list yet, but you can apply it with
>>> git-am.
>>>
>>>  markdown.py |    8 ++++----
>>>  1 files changed, 4 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/markdown.py b/markdown.py
>>> index 3348acc..422cde4 100644
>>> --- a/markdown.py
>>> +++ b/markdown.py
>>> @@ -34,7 +34,7 @@ __revision__ = "$Rev$"
>>>
>>>
>>>  import re, sys, codecs
>>> -from urlparse import urlparse
>>> +from urlparse import urlparse, urlunparse
>>>
>>>  from logging import getLogger, StreamHandler, Formatter, \
>>>                     DEBUG, INFO, WARN, ERROR, CRITICAL
>>> @@ -930,9 +930,9 @@ class LinkPattern (Pattern):
>>>
>>>         """
>>>         locless_schemes = ['', 'mailto', 'news']
>>> -        url = urlparse(url)
>>> +        scheme, netloc, path, params, query, fragment = url = urlparse(url)
>>>         safe_url = False
>>> -        if url.netloc != '' or url.scheme in locless_schemes:
>>> +        if netloc != '' or scheme in locless_schemes:
>>>             safe_url = True
>>>
>>>         for part in url[2:]:
>>> @@ -942,7 +942,7 @@ class LinkPattern (Pattern):
>>>         if self.safe_mode and not safe_url:
>>>             return ''
>>>         else:
>>> -            return url.geturl()
>>> +            return urlunparse(url)
>>>
>>>  class ImagePattern(LinkPattern):
>>>     """ Return a NanoDom img Element from the given match. """
>>> --
>>> 1.5.5.1
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by: Microsoft
>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>> _______________________________________________
>>> Python-markdown-discuss mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>>
>>
>>
>>
>> --
>> http://sputnik.freewisdom.org/
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Python-markdown-discuss mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>
>
>
>
> --
> ----
> Waylan Limberg
> wa...@gm...
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] [PATCH] create the test output directory if it doesn't already exist.

[Waylan L. <wa...@gm...>]

I don't see where this one was pushed either.

On Fri, May 30, 2008 at 10:33 PM, Yuri Takhteyev <qar...@gm...> wrote:
> Thanks.  I merged and pushed it.
>
> BTW, it looks like something went very wrong with the "To:" field in
> your message.
>
>  - yuri
>
> On Fri, May 30, 2008 at 4:52 PM, Jeff Balogh <its...@gm...> wrote:
>> codecs.open was failing without it.
>> ---
>>  test-markdown.py |    3 +++
>>  1 files changed, 3 insertions(+), 0 deletions(-)
>>
>> diff --git a/test-markdown.py b/test-markdown.py
>> index e3997a4..bc2b891 100644
>> --- a/test-markdown.py
>> +++ b/test-markdown.py
>> @@ -214,6 +214,9 @@ def testDirectory(dir, measure_time=False, safe_mode=False) :
>>     except:
>>         hd = None
>>
>> +    if not os.path.exists(TMP_DIR):
>> +        os.mkdir(TMP_DIR)
>> +
>>     htmlDiffFilePath = os.path.join(TMP_DIR, os.path.split(dir)[-1]) + ".html"
>>     htmlDiffFile = codecs.open(htmlDiffFilePath, "w", encoding=encoding)
>>     htmlDiffFile.write(DIFF_FILE_TEMPLATE)
>> --
>> 1.5.5.1
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Python-markdown-discuss mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>
>
>
>
> --
> http://sputnik.freewisdom.org/
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] [PATCH] unwrapping urlparse result tuple manually for python 2.4 compatibility.

[Waylan L. <wa...@gm...>]

Hey Yuri, I don't see where this was pushed to Gitorious. What am I missing?

On Fri, May 30, 2008 at 6:18 PM, Yuri Takhteyev <qar...@gm...> wrote:
> Thanks.  I pushed this commit.  Sorry about this, not sure when this
> has slipped in.  My goal has been to keep compatibility not only with
> 2.4 but also with 2.3.  I also noticed that while markdown.py works
> with 2.3 (well used to, and now again with your patch), the
> test-markdown.py script did not.  So, I changed test-markdown.py to
> work with 2.3.
>
> So, let's all agree that test-markdown.py should be ran with
> python2.3, python2.4 and python2.5 after each change.  If you cannot
> check it with all three, then make a commit in a branch and email it
> to me so that I could test before merging.
>
> Also, Waylan and David, can you check in test cases for the fixes you
> made in the recent months?
>
> BTW, I also noticed that we have a regression on one of the tests.  I
> suspect this might be the price we are paying for some of the other
> issues that Waylan fixed.  Hopefully we'll resolve this with Artem's
> help over the next few months.
>
>  - yuri
>
> On Fri, May 30, 2008 at 2:22 PM, Jeff Balogh <its...@gm...> wrote:
>> In 2.5 the result is wrapped in a ParseResult class so that fields can be
>> accessed by name, but a plain tuple is returned in 2.4.
>> ---
>>
>> I don't think I've seen any patches on the list yet, but you can apply it with
>> git-am.
>>
>>  markdown.py |    8 ++++----
>>  1 files changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/markdown.py b/markdown.py
>> index 3348acc..422cde4 100644
>> --- a/markdown.py
>> +++ b/markdown.py
>> @@ -34,7 +34,7 @@ __revision__ = "$Rev$"
>>
>>
>>  import re, sys, codecs
>> -from urlparse import urlparse
>> +from urlparse import urlparse, urlunparse
>>
>>  from logging import getLogger, StreamHandler, Formatter, \
>>                     DEBUG, INFO, WARN, ERROR, CRITICAL
>> @@ -930,9 +930,9 @@ class LinkPattern (Pattern):
>>
>>         """
>>         locless_schemes = ['', 'mailto', 'news']
>> -        url = urlparse(url)
>> +        scheme, netloc, path, params, query, fragment = url = urlparse(url)
>>         safe_url = False
>> -        if url.netloc != '' or url.scheme in locless_schemes:
>> +        if netloc != '' or scheme in locless_schemes:
>>             safe_url = True
>>
>>         for part in url[2:]:
>> @@ -942,7 +942,7 @@ class LinkPattern (Pattern):
>>         if self.safe_mode and not safe_url:
>>             return ''
>>         else:
>> -            return url.geturl()
>> +            return urlunparse(url)
>>
>>  class ImagePattern(LinkPattern):
>>     """ Return a NanoDom img Element from the given match. """
>> --
>> 1.5.5.1
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Python-markdown-discuss mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>
>
>
>
> --
> http://sputnik.freewisdom.org/
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] GSoC plan

[Yuri T. <yu...@si...>]

> I thought about ElementTree, but there is a problem with entities
> escaping and I I didn't find any beautiful solution yet.

Sure.  I am not saying we should switch to ElementTree.  Just that
this might be worth considering at this point.  Sticking with Nanodom
and fixing the discrepancies from minidom API might be less work.

> Thanks for the suggestion. Do you mean that the first round of parsing
> should be without regexps?

No, I didn't mean to say that we should avoid regexps in parsing -
just that this wouldn't be a simple substitution.  Avoiding regular
expression would be neither necessary nor feasible.  I would actually
prefer that for now you keep the current parsing code as is for now.

Right now Markdown class has a single _transform() method which takes
markdown source and returns HTML.  _transform() calls
_processSection() on the source, which is itself recursive.  I
wouldn't mind getting rid of this recursion, but let's not worry about
this right now.  Instead, let's extricate _handleInline from all of
this.  That is, instead of the single call to _transform, I would
rather have two functions:

markdown_to_tree (markdown_source) - takes markdown, returns a Nanodom
tree, WITHOUT applying inline patterns
apply_inline_patterns (nanodom_tree) - takes a nano-dom tree and
applies inline patterns to all nodes that need it (returning either
the modified tree or a copy of it).

So, one would be able to do conversion with:

    m = Markdown()
    return m.apply_inline_patterns(m.markdown_to_tree(my_source)).to_xml()

Or maybe attach the second function as a method to NanoDom:

    return m.markdown_to_tree(my_source).apply_inline_patterns().to_xml()

What this would gain us is two things.  First we'll have better
separation of code into two areas.  I think this will make it easier
to read and maintain.  This will put us in a good position to change
how inline patterns are handled.  Second, this will give the caller
more options: they can do stuff to the tree before applying inline
patterns.  They could also come up with their own way of handling
inline patterns.

 - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] GSoC plan

[Artem <ne...@gm...>]

Yuri Takhteyev  wrote:
> It's ok to break backwards compatibility with extensions, if this buys
> us enough.  Similarly, we can think of dropping Python 2.3 support -
> again if this really buys us something big.  In particular, as far as
> the tree representation goes, we can consider a couple of things, in
> particular:
>
> 1. Stick with NanoDOM, fixing the problems you mentioned
> 2. Switch to ElementTree
>
> Let's discuss those options.  (Or other.)
>   
I thought about ElementTree, but there is a problem with entities 
escaping and I I didn't find any beautiful solution yet.
> (Artem: can you make a page for this project on the wiki, to keep
> track of the questions and the decisions, and then start separate
> threads for each question, though probably not all at the same time?)
>
>   
Sure. 
Here it is: http://www.freewisdom.org/projects/python-markdown/GSoC2008
> In other words, my suggestion is that the first round of parsing
> should turn the document into a tree of blocks, where nodes in the
> tree represent individual simple paragraphs, list items, block quotes,
> code segments, block level HTML elements, etc.  The client will then
> ideally be able to get this tree back if they want.  The second round
> of parsing would then simply go through this tree and run a different
> set of regular expressions on each node depending on the type of the
> node.
>
>   
Thanks for the suggestion. Do you mean that the first round of parsing 
should be without regexps?
> Yes, let's first do inline patterns, and see what this gives us.
> Another thing that could be done is avoiding excessive recursion in
> block parsing.  But let's do one thing at a time
Ok.
> Sounds good, but don't get too sidetracked on this.  Also, we need to
> make sure there is someone on Crunchy project who is actually
> interested in this and will make use of it.
>
>   
I think we should contact with André Roberge.
>> - Test suite extension
>>     
>
> Yes, in fact, I would consider doing this _first_.  I.e., it would be
> good to put together a unified test suite that combines all of our and
> Trent's tests and gives us a better idea of where the two
> implementations stand relative to each other.
>
> BTW, I would urge you to make sure that your modified version of
> python markdown passes all of our tests (and an increasing number of
> Trent's) at least once a week.  Let's avoid the "Version 2" problem.
>
>   
Got it.
>> - Some additional documentation, maybe adding more examples about
>> writing extension modules.
>>     
>
> Let's put this off till later.  If we'll be making serious changes, it
> makes more sense to work on the documentation after the work is done.
> But it would be good if you could at least document any changes that
> you end up making before the end of the summer.
>   
Ok.

Re: [Python-markdown-discuss] GSoC plan

[Artem <ne...@gm...>]

Waylan Limberg wrote:
> Don't worry to much about breaking old extensions. If you refactor the
> inline patterns (as noted below), extensions will need to be updated
> anyway. Just make sure your changes are actually useful, not just a
> new color for the bikeshed.
>
>   
Ok.
>> - There is something to do with Inline patterns, I didn't decide yet
>> what is the best way to fix it. That was discussed in list and there are
>> some ideas. I thought of writing syntax/lexical parser instead of
>> current Inline Patterns mechanism, but I think it'll be a bit  slower.
>>
>> - I'll try to boost performance, I think choosing the right way of
>> inline patterns modification is the best way to boost Markdown.
>>     
>
> I agree. I'd really encourage you to spend the most time on the inline
> pattern issue. That's where markdown.py needs the most help IMO.
>   
Yep, I already understood it.
> Just be sure to update any extensions in the repo that you break. Feel
> free to check with me on any of them as I wrote most of them.
>   
Ok, thanks.

Re: [Python-markdown-discuss] GSoC plan

[Yuri T. <qar...@gm...>]

> Python has changed---in good ways---since 2.3  Who are we targeting by being
> compatible with 2.3?

Anyone using python markdown on a shared hosting system that does not
support the more recent versionsn of Python.  For example, Dreamhost
defaults to python 2.3 at this point and python 2.4 is the highest
version that is available.  If you want python2.5 on dreamhost, you'll
need to build it yourself.

So, python 2.3 support may be worth re-thinking at this point, but i
don't think it's something to be taken lightly.  Note that Django
works with Python 2.3.

We should definitely start testing the code with 2.6 once it's out.
If someone wants to look into making sure that the code is compatible
with Python 3000 (in the sense that it can be converted successfully
via 2to3), then I am all for it and I will check in any patches that
are needed to make it compatible.  (Assuming it still works with the
earlier versions too.)   But I do not myself plan to look into Py3K at
the moment, since I don't yet see much demand for that.

  - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] GSoC plan

[Sam's L. <sam...@gm...>]

Hi...

I hope no one minds me asking....

But I think perhaps you should at least consider freezing the old code---and
having these changes target the upcoming Python 3.  Perhaps also being
compatible with the upcoming 2.6.

Python has changed---in good ways---since 2.3  Who are we targeting by being
compatible with 2.3?

Thanks

On Wed, Jun 4, 2008 at 10:55 AM, Yuri Takhteyev <qar...@gm...> wrote:

> > As I understand code must be compatible with python 2.3, 2.4, 2.5, isn't
> > it?
>
> We can reconsider this decision, if there is a good reason.  We just
> shouldn't take it lightly.
>
>
> <http://sputnik.freewisdom.org/>
>
>

Re: [Python-markdown-discuss] GSoC plan

[Yuri T. <qar...@gm...>]

> - Some code refactoring, there are a lot of ideas about it in Greg
> Wilson's review, and I as well have some ideas.
> For instance, in current Markdown DOM implementation there are some
> differences from standard DOM libraries, for example usually in
> Element.replaceChild the first argument is newNode and the second is
> oldNode, in Markdown implementation first is oldNode and the second is
> newNode. Usually Element's parent property name is "parentNode", but
> here it's just "parent" etc.
> I realize that it'll break some extensions, but I think it'll help
> people in future to avoid reading code if they already know some DOM
> library.

It's ok to break backwards compatibility with extensions, if this buys
us enough.  Similarly, we can think of dropping Python 2.3 support -
again if this really buys us something big.  In particular, as far as
the tree representation goes, we can consider a couple of things, in
particular:

1. Stick with NanoDOM, fixing the problems you mentioned
2. Switch to ElementTree

Let's discuss those options.  (Or other.)

(Artem: can you make a page for this project on the wiki, to keep
track of the questions and the decisions, and then start separate
threads for each question, though probably not all at the same time?)

> - There is something to do with Inline patterns, I didn't decide yet
> what is the best way to fix it. That was discussed in list and there are
> some ideas. I thought of writing syntax/lexical parser instead of
> current Inline Patterns mechanism, but I think it'll be a bit  slower.

I agree with Waylan, this should be the focus.  I would avoid trying
to do serious parsing.  I think this part might be best done using
straight regular expressions, and we might even be able to "steal" a
lot of codes from Trent's markdown2.

In other words, my suggestion is that the first round of parsing
should turn the document into a tree of blocks, where nodes in the
tree represent individual simple paragraphs, list items, block quotes,
code segments, block level HTML elements, etc.  The client will then
ideally be able to get this tree back if they want.  The second round
of parsing would then simply go through this tree and run a different
set of regular expressions on each node depending on the type of the
node.

If Python had a good PEG implementation, it perhaps would make sense
to consider rewriting markdown using PEG.  But at this point I think
it's premature.

> - I'll try to boost performance, I think choosing the right way of
> inline patterns modification is the best way to boost Markdown.

Yes, let's first do inline patterns, and see what this gives us.
Another thing that could be done is avoiding excessive recursion in
block parsing.  But let's do one thing at a time.

> - An extension for Crunchy to load files using the Markdown syntax.

Sounds good, but don't get too sidetracked on this.  Also, we need to
make sure there is someone on Crunchy project who is actually
interested in this and will make use of it.

> - Test suite extension

Yes, in fact, I would consider doing this _first_.  I.e., it would be
good to put together a unified test suite that combines all of our and
Trent's tests and gives us a better idea of where the two
implementations stand relative to each other.

BTW, I would urge you to make sure that your modified version of
python markdown passes all of our tests (and an increasing number of
Trent's) at least once a week.  Let's avoid the "Version 2" problem.

> - Some additional documentation, maybe adding more examples about
> writing extension modules.

Let's put this off till later.  If we'll be making serious changes, it
makes more sense to work on the documentation after the work is done.
But it would be good if you could at least document any changes that
you end up making before the end of the summer.

> Also I wrote to Django community asking if they need something special,
> but they said that nothing Django-specific, but API stability.  Someone
> suggested adding markdown extras, but Waylan said that it was already
> almost done.

Yes, let's assume that the basic API will stay the same.

> As I understand code must be compatible with python 2.3, 2.4, 2.5, isn't
> it?

We can reconsider this decision, if there is a good reason.  We just
shouldn't take it lightly.

 - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] GSoC plan

[Waylan L. <wa...@gm...>]

Hi Artem,

On Wed, Jun 4, 2008 at 5:34 AM, Artem <ne...@gm...> wrote:
> Hi all,
>
> So, I plan to do the following:
>
> - Some code refactoring, there are a lot of ideas about it in Greg
> Wilson's review, and I as well have some ideas.
> For instance, in current Markdown DOM implementation there are some
> differences from standard DOM libraries, for example usually in
> Element.replaceChild the first argument is newNode and the second is
> oldNode, in Markdown implementation first is oldNode and the second is
> newNode. Usually Element's parent property name is "parentNode", but
> here it's just "parent" etc.
> I realize that it'll break some extensions, but I think it'll help
> people in future to avoid reading code if they already know some DOM
> library.

Don't worry to much about breaking old extensions. If you refactor the
inline patterns (as noted below), extensions will need to be updated
anyway. Just make sure your changes are actually useful, not just a
new color for the bikeshed.

>
> - There is something to do with Inline patterns, I didn't decide yet
> what is the best way to fix it. That was discussed in list and there are
> some ideas. I thought of writing syntax/lexical parser instead of
> current Inline Patterns mechanism, but I think it'll be a bit  slower.
>
> - I'll try to boost performance, I think choosing the right way of
> inline patterns modification is the best way to boost Markdown.

I agree. I'd really encourage you to spend the most time on the inline
pattern issue. That's where markdown.py needs the most help IMO.
>
> - An extension for Crunchy to load files using the Markdown syntax.
>
> - Test suite extension
>
> - Some additional documentation, maybe adding more examples about
> writing extension modules.

I've been meaning to do this myself, but your more than welcome to.
>
> Also I wrote to Django community asking if they need something special,
> but they said that nothing Django-specific, but API stability.  Someone
> suggested adding markdown extras, but Waylan said that it was already
> almost done.

Just be sure to update any extensions in the repo that you break. Feel
free to check with me on any of them as I wrote most of them.
>
> Maybe while working I'll find some other ideas, of what else I can do.
>
> As I understand code must be compatible with python 2.3, 2.4, 2.5, isn't
> it?

Yes, that is correct.




-- 
----
Waylan Limberg
wa...@gm...

[Python-markdown-discuss] GSoC plan

[Artem <ne...@gm...>]

Hi all,
 
So, I plan to do the following:
 
- Some code refactoring, there are a lot of ideas about it in Greg 
Wilson's review, and I as well have some ideas.
For instance, in current Markdown DOM implementation there are some 
differences from standard DOM libraries, for example usually in 
Element.replaceChild the first argument is newNode and the second is 
oldNode, in Markdown implementation first is oldNode and the second is 
newNode. Usually Element's parent property name is "parentNode", but 
here it's just "parent" etc.
I realize that it'll break some extensions, but I think it'll help 
people in future to avoid reading code if they already know some DOM 
library.
 
- There is something to do with Inline patterns, I didn't decide yet 
what is the best way to fix it. That was discussed in list and there are 
some ideas. I thought of writing syntax/lexical parser instead of 
current Inline Patterns mechanism, but I think it'll be a bit  slower.
 
- I'll try to boost performance, I think choosing the right way of 
inline patterns modification is the best way to boost Markdown.
 
- An extension for Crunchy to load files using the Markdown syntax.

- Test suite extension
 
- Some additional documentation, maybe adding more examples about 
writing extension modules.
 
Also I wrote to Django community asking if they need something special, 
but they said that nothing Django-specific, but API stability.  Someone 
suggested adding markdown extras, but Waylan said that it was already 
almost done.

Maybe while working I'll find some other ideas, of what else I can do.
 
As I understand code must be compatible with python 2.3, 2.4, 2.5, isn't 
it?

Re: [Python-markdown-discuss] [PATCH] unwrapping urlparse result tuple manually for python 2.4 compatibility.

[Waylan L. <wa...@gm...>]

On Fri, May 30, 2008 at 6:18 PM, Yuri Takhteyev <qar...@gm...> wrote:
> Thanks.  I pushed this commit.  Sorry about this, not sure when this
> has slipped in.  My goal has been to keep compatibility not only with
> 2.4 but also with 2.3.  I also noticed that while markdown.py works
> with 2.3 (well used to, and now again with your patch), the
> test-markdown.py script did not.  So, I changed test-markdown.py to
> work with 2.3.
>
> So, let's all agree that test-markdown.py should be ran with
> python2.3, python2.4 and python2.5 after each change.  If you cannot
> check it with all three, then make a commit in a branch and email it
> to me so that I could test before merging.

Yeah, I forget to test in multiple versions sometimes. Thanks for the reminder.
>
> Also, Waylan and David, can you check in test cases for the fixes you
> made in the recent months?

I've been adding tests for most of the changes I make. There's just
been a few things that aren't directly related to the syntax. And,
IIRC, David has made very few syntax changes - all with tests.
>
> BTW, I also noticed that we have a regression on one of the tests.  I
> suspect this might be the price we are paying for some of the other
> issues that Waylan fixed.

Yes, but I forget which one now. Anyway, I wasn't worried about it
because the test shouldn't have been passing anyway. If you look at
the html file for that test it is obvious that that is not the
intended result. I'd rather have it fail than create some false sense
that it's passing.

> Hopefully we'll resolve this with Artem's
> help over the next few months.

Yeah, if he's working on how InlinePatterns nest, that should fix the issue.



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] [PATCH] create the test output directory if it doesn't already exist.

[Yuri T. <qar...@gm...>]

Thanks.  I merged and pushed it.

BTW, it looks like something went very wrong with the "To:" field in
your message.

 - yuri

On Fri, May 30, 2008 at 4:52 PM, Jeff Balogh <its...@gm...> wrote:
> codecs.open was failing without it.
> ---
>  test-markdown.py |    3 +++
>  1 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/test-markdown.py b/test-markdown.py
> index e3997a4..bc2b891 100644
> --- a/test-markdown.py
> +++ b/test-markdown.py
> @@ -214,6 +214,9 @@ def testDirectory(dir, measure_time=False, safe_mode=False) :
>     except:
>         hd = None
>
> +    if not os.path.exists(TMP_DIR):
> +        os.mkdir(TMP_DIR)
> +
>     htmlDiffFilePath = os.path.join(TMP_DIR, os.path.split(dir)[-1]) + ".html"
>     htmlDiffFile = codecs.open(htmlDiffFilePath, "w", encoding=encoding)
>     htmlDiffFile.write(DIFF_FILE_TEMPLATE)
> --
> 1.5.5.1
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
http://sputnik.freewisdom.org/