python-markdown-discuss — A general purpose discussion list

Re: [Python-markdown-discuss] New TOC extension.

[Waylan L. <wa...@gm...>]

Oh, one other thing, wrap the toctitle in an if statement and only add
it if a title is set in the config settings. Right now your getting a
blank title ``<span class="toctitle" />``.

On Fri, Oct 17, 2008 at 9:25 AM, Waylan Limberg <wa...@gm...> wrote:
> Hi Jack,
>
> Interesting. I was just thinking about how to do that the other day.
> It looks like your approach is very similar to how mine would have
> been.
>
> There is a few things though:
>
> * Why are you creating anchors in each header? Why not just assign
> each header an "id" attribute? That is generally considered the
> correct way to do things and works just the same.
>
> * It would be nice if it worked with the HeaderID extension. That is,
> if a header already has an id attribute, that is used rather than
> recreating one which may be different.
>
> On Fri, Oct 17, 2008 at 2:18 AM, Jack Miller <ja...@co...> wrote:
>> Hi all.
>>
>> I just wanted to point out that I wrote a TOC extension that's nicer than the
>> one linked to the wiki and (more importantly) is 2.0+ compatible. Some
>> differences from the aforementioned extension:
>>
>> 1. The anchor names aren't mangled... they are anchors generated by
>> slugifying the header contents (ala Django), which leads to much more
>> readable/memorizable links.
>>
>> 2. Header text is replaced by a self link of type toclink.
>>
>> 3. All instances of the marked (by default [TOC]) are replaced by the TOC
>> div, not just the first one. It also replaces the entire containing tag too.
>> It's designed for having
>>
>>    Textextextextextextextextextext
>>
>>    [TOC]
>>
>>    Moretextextextextextextex
>>
>> So, to avoid messing up validation, it replaces the <p> tags created by
>> having the breaks.
>>
>> Anyway, the code is available: http://codezen.org/static/toc.py
>> And it's in use on my project pages, if you want to see an example:
>> http://codezen.org/canto/config/
>>
>> I'd like to get this included with the main source, if possible. Otherwise,
>> I'll just add a link to the wiki when 2.0 is officially released.
>>
>> --Jack
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Python-markdown-discuss mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>
>
>
>
> --
> ----
> Waylan Limberg
> wa...@gm...
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] New TOC extension.

[Waylan L. <wa...@gm...>]

Hi Jack,

Interesting. I was just thinking about how to do that the other day.
It looks like your approach is very similar to how mine would have
been.

There is a few things though:

* Why are you creating anchors in each header? Why not just assign
each header an "id" attribute? That is generally considered the
correct way to do things and works just the same.

* It would be nice if it worked with the HeaderID extension. That is,
if a header already has an id attribute, that is used rather than
recreating one which may be different.

On Fri, Oct 17, 2008 at 2:18 AM, Jack Miller <ja...@co...> wrote:
> Hi all.
>
> I just wanted to point out that I wrote a TOC extension that's nicer than the
> one linked to the wiki and (more importantly) is 2.0+ compatible. Some
> differences from the aforementioned extension:
>
> 1. The anchor names aren't mangled... they are anchors generated by
> slugifying the header contents (ala Django), which leads to much more
> readable/memorizable links.
>
> 2. Header text is replaced by a self link of type toclink.
>
> 3. All instances of the marked (by default [TOC]) are replaced by the TOC
> div, not just the first one. It also replaces the entire containing tag too.
> It's designed for having
>
>    Textextextextextextextextextext
>
>    [TOC]
>
>    Moretextextextextextextex
>
> So, to avoid messing up validation, it replaces the <p> tags created by
> having the breaks.
>
> Anyway, the code is available: http://codezen.org/static/toc.py
> And it's in use on my project pages, if you want to see an example:
> http://codezen.org/canto/config/
>
> I'd like to get this included with the main source, if possible. Otherwise,
> I'll just add a link to the wiki when 2.0 is officially released.
>
> --Jack
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

[Python-markdown-discuss] New TOC extension.

[Jack M. <ja...@co...>]

Hi all.

I just wanted to point out that I wrote a TOC extension that's nicer than the
one linked to the wiki and (more importantly) is 2.0+ compatible. Some
differences from the aforementioned extension:

1. The anchor names aren't mangled... they are anchors generated by
slugifying the header contents (ala Django), which leads to much more
readable/memorizable links.

2. Header text is replaced by a self link of type toclink.

3. All instances of the marked (by default [TOC]) are replaced by the TOC
div, not just the first one. It also replaces the entire containing tag too.
It's designed for having

    Textextextextextextextextextext

    [TOC]

    Moretextextextextextextex

So, to avoid messing up validation, it replaces the <p> tags created by
having the breaks.

Anyway, the code is available: http://codezen.org/static/toc.py
And it's in use on my project pages, if you want to see an example:
http://codezen.org/canto/config/

I'd like to get this included with the main source, if possible. Otherwise,
I'll just add a link to the wiki when 2.0 is officially released.

--Jack

Re: [Python-markdown-discuss] refactoring and treap integration

[Waylan L. <wa...@gm...>]

On Mon, Oct 13, 2008 at 7:35 PM, Yuri Takhteyev <qar...@gm...> wrote:
>> Why not just make the InlineProcessor be one of the 'postprocessors'
>> and then extensions can add additional postprocessors either before or
>> after it as needed?
>
> Good point.  If we then also get rid of the
> preprocessor/textpreprocessor distinction, we can just reduce it all
> to three:
>
> Preprocessor treap: HtmlBlock, Header, Line, Reference
> Treeprocessors treap: Inline
> Postprocessors treap: Prettify, RawHtml, AndSubstitute

Except that Prettify is a Treeprocessor. In any event, I like this
naming much better (pre, tree, post). It's much clearer whats going
on.

>
> Extensions can then insert processors into one of those three treaps
> and also insert patterns into the inline processor.  (Or they can
> replace InlineProcessor with their own.)

...or they can replace/subclass the MarkdownParser.

With this api, someone could use the Markdown engine and rewrite a
completely different markup language. Not that one should, but the
fact that one can is a testament to the api IMO.

--
---
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] refactoring and treap integration

[Yuri T. <qar...@gm...>]

> Why not just make the InlineProcessor be one of the 'postprocessors'
> and then extensions can add additional postprocessors either before or
> after it as needed?

Good point.  If we then also get rid of the
preprocessor/textpreprocessor distinction, we can just reduce it all
to three:

Preprocessor treap: HtmlBlock, Header, Line, Reference
Treeprocessors treap: Inline
Postprocessors treap: Prettify, RawHtml, AndSubstitute

Extensions can then insert processors into one of those three treaps
and also insert patterns into the inline processor.  (Or they can
replace InlineProcessor with their own.)

  - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] refactoring and treap integration

[Waylan L. <wa...@gm...>]

On Mon, Oct 13, 2008 at 1:40 PM, Yuri Takhteyev <qar...@gm...> wrote:
[snip]
>
>>    md.parser._MarkdownParser__processCodeBlock = __hiliteCodeBlock
>
> I think there is an entirely different (and better) way to do this
> now.  Use the standard MarkdownParser, then write a postprocessor to
> modify the eTree.

Don't know why I didn't think of this before. eTree makes is easy. I
just pushed a refactored CodeHilite extension. Much cleaner.

> At the moment, it appears that we don't offer an
> option of modifying the tree before the patterns are run, but we
> should.  I.e., our pipeline should be:
>
> 1. text pre-processors (text-in, text-out) - tempated to drop this
> 2. line pre-processors (line list in, line list out)
> 3. MarkdownParser.parseDocument() - substitute your own if you want
> 4. pre-pattern post-processors (modify the tree before any patterns are run)
> 5. InlineProcessor.applyInlinePatterns()
> 6. etree postprocessors (modify eTree)
> 7. serialization of the etree into a string
> 8. text postprocessors (text-in, text-out)

Why not just make the InlineProcessor be one of the 'postprocessors'
and then extensions can add additional postprocessors either before or
after it as needed?




-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] refactoring and treap integration

[Yuri T. <qar...@gm...>]

> Thanks. Hey, I noticed the commits. Can y'all share more on the Wiki

Let me finish the code first.  There are still a few issues.  First,
there are a few tests that break.  Second, the Treap implementation
now requires Python 2.4.  I think we can get it to work with 2.3
without too much hassle, though.  Finally, I want to extend Treap a
little to make the initial construction of the treap a little easier.
I think the third argument to add() should be optional, defaulting to
"after the whatever is currently in the end").  I.e., I want to be
able to just write:

        self.inlinePatterns.add("escape", SimpleTextPattern(ESCAPE_RE))
        self.inlinePatterns.add("link", LinkPattern(LINK_RE))
        self.inlinePatterns.add("image_link", ImagePattern(IMAGE_LINK_RE))

> I'm surprised I've not been getting these messages in a while. I'm in
>
> the process of moving domain names, and suddenly I'm picking them up.

Well, glad you are still with us.  To make a long story shot, Artem
Yunusov did a lot of work this summer posting the code to use
ElementTree and starting the separation that I finished yesterday.
(Artem really did most of the hard work, I just took the methods,
sorted them into two classes, and then gave them simpler names.)  This
was my original goal for "2.0", which also created the opportunity to
also move to treap - only after 15 months of delay!

If you want to suggest any modifications to your treap implementation
(or other things), you can send me patches, create a "clone" on
gitorious, or email me your user name and will add you as a committer
so that you could create branches in our current repository.

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] refactoring and treap integration

[Yuri T. <qar...@gm...>]

> Well, maybe not so cool. By making all (most) of the methods truly
> private it makes moneypatching more difficult. I realize it's not that
> hard to use a subclass rather than monkeypatch, but what happens when
> two extensions each create their own subclass changing a different
> method? With monkeypatches, we just used the same instance and all was
> good.

They don't have to stay private.  I decided to start by making them
private in order to expose any questionable dependencies that we may
have.  We can then think of whether there are a few more methods that
may be worth exposing.

>    md.parser._MarkdownParser__processCodeBlock = __hiliteCodeBlock

I think there is an entirely different (and better) way to do this
now.  Use the standard MarkdownParser, then write a postprocessor to
modify the eTree.  At the moment, it appears that we don't offer an
option of modifying the tree before the patterns are run, but we
should.  I.e., our pipeline should be:

1. text pre-processors (text-in, text-out) - tempated to drop this
2. line pre-processors (line list in, line list out)
3. MarkdownParser.parseDocument() - substitute your own if you want
4. pre-pattern post-processors (modify the tree before any patterns are run)
5. InlineProcessor.applyInlinePatterns()
6. etree postprocessors (modify eTree)
7. serialization of the etree into a string
8. text postprocessors (text-in, text-out)

My generic recommendation now would be that extension writers first
look into whether they can do what they want to do by adding
post-processors at steps 4, 6 or 8, or by adding patterns.

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] refactoring and treap integration

[Yuri T. <qar...@gm...>]

> As a sidenote, I'm intrigued by the MarkdownParser class. One could
> conceivably replace that class with their own which works differently
> internally - as long as it has the same public methods and returns an
> etree instance. This really opens up the possibility of
> overriding/changing the core stuff. Cool! -- And if we want to change
> the internal stuff, it should have little to no effect on the external
> api.

This wasn't the intension, but yes this is true.  Note also that you
use MarkdownParser's parseChunk() method in your custom parser.  That
is, you can parse certain things yourself, then delegate the rest to
the original parser with parseChunk(parent, lines).

Again, the main motivation for splitting was to make it easy for
people (even myself!) to understand what does what.  Now if you want
to understand how high-level parsing works, you only need to review
362 lines, not 2000+.  It also creates good granularity for adding
unit testing.  E.g., we can now write tests around MarkdownParser to
keep track of both correctness and performance.

--
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] refactoring and treap integration

[Waylan L. <wa...@gm...>]

On Mon, Oct 13, 2008 at 9:33 AM, Waylan Limberg <wa...@gm...> wrote:
> As a sidenote, I'm intrigued by the MarkdownParser class. One could
> conceivably replace that class with their own which works differently
> internally - as long as it has the same public methods and returns an
> etree instance. This really opens up the possibility of
> overriding/changing the core stuff. Cool!

Well, maybe not so cool. By making all (most) of the methods truly
private it makes moneypatching more difficult. I realize it's not that
hard to use a subclass rather than monkeypatch, but what happens when
two extensions each create their own subclass changing a different
method? With monkeypatches, we just used the same instance and all was
good.

Now, thats not so easy. Sure, it's possible, but definitely feels more
hacky. For example, the CodeHilite extension has to do this:

    md.parser._MarkdownParser__processCodeBlock = __hiliteCodeBlock

instead of this:

    md._processCodeBlock = _hiliteCodeBlock

Or does someone have any suggestions of how to resolve different
extensions all using different subclasses of MarkdownParser without
each extension being specifically aware of the others? I don't see how
mixins would work here either. Or am I missing something obvious?

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] refactoring and treap integration

[Ben W. <bw...@da...>]

Thanks. Hey, I noticed the commits. Can y'all share more on the Wiki

Links?



I'm surprised I've not been getting these messages in a while. I'm in

the process of moving domain names, and suddenly I'm picking them up.

So, I think I'm a bit behind.



On 10/13/2008, "Waylan Limberg" <wa...@gm...> wrote:



>Well, we haven't release yet...

>

>but the code is all in our git repo on gitorious.org [1]

>

>[1]: http://gitorious.org/projects/python-markdown

>

>On Mon, Oct 13, 2008 at 9:40 AM, Ben Wilson <bw...@da...> wrote:

>>

>> Anyway to get an advanced look at the new 2.0? I went looking on the site

>>

>> and only found 1.7...

>>

>>

>>

>> On 10/13/2008, "Yuri Takhteyev" <qar...@gm...> wrote:

>>

>>

>>

>>>I made a whole bunch of changes to the code, most of them just a

>>

>>>matter of refactoring, but some affect the functionality too.

>>

>>>

>>

>>>In terms of refactoring, the biggest change is splitting Markdown

>>

>>>class into three and making most of their methods private.  I think

>>

>>>this will make it easier to understand the code: you can now study one

>>

>>>class at a time.  Two of those three classes are still a little messy,

>>

>>>but at least the messiness is contained.

>>

>>>

>>

>>>So, we now have:

>>

>>>

>>

>>>1. MarkdownParser - parses pre-processed Markdown source into an ElementTr
ee

>> .

>>

>>>

>>

>>>Usage:

>>

>>>

>>

>>>    tree = MarkdownParser.parseDocument(markdown_string)

>>

>>>

>>

>>>The only other exposed methods are parseChunk() and detectTabbed().  I

>>

>>>am tempted to hide them as well, but at the moment they are needed by

>>

>>>some extensions.

>>

>>>

>>

>>>2. InlineProcessor - runs inline patterns on an ElementTree

>>

>>>

>>

>>>Usage:

>>

>>>

>>

>>>    InlineProcessor(patterns).applyInlinePatterns(tree)

>>

>>>

>>

>>>This is the only exposed method.  I also folded into this the InlineStash 
cl

>> ass.

>>

>>>

>>

>>>3. Markdown - puts it all together.

>>

>>>

>>

>>>Usage:

>>

>>>

>>

>>>    Markdown(extensions).convert(markdown_string)

>>

>>>    Markdown(extensions).convertFile(input_file_path,

>>

>>>output_path_or_stream, encoding)

>>

>>>

>>

>>>markdownFromFile() function still exists, but only has two lines now.

>>

>>>

>>

>>>Another change, which does affects functionality, is that I

>>

>>>incorporated Ben's treap implementation as a way of organizing

>>

>>>pre-processors, patterns, etc.  This kills two birds: we now have a

>>

>>>better way of organizing those things, and this should also fix the

>>

>>>problem reported by Eric Abrahamsen last week, which required a major

>>

>>>change anyway.  This breaks many extensions, and also breaks one

>>

>>>non-extension test.  But 2.0 is about as good of a chance as we will

>>

>>>get for breaking backwards compatibility.

>>

>>>

>>

>>>I updated the footnotes extension as an example of how to use the new syst
em

>> .

>>

>>>

>>

>>>- yuri

>>

>>>

>>

>>>--

>>

>>>http://sputnik.freewisdom.org/

>>

>>>

>>

>>>-------------------------------------------------------------------------

>>

>>>This SF.Net email is sponsored by the Moblin Your Move Developer's challen
ge

>>

>>>Build the coolest Linux based applications with Moblin SDK & win great pri
ze

>> s

>>

>>>Grand prize is a trip for two to an Open Source event anywhere in the worl
d

>>

>>>http://moblin-contest.org/redirect.php?banner_id=100&url=/

>>

>>>_______________________________________________

>>

>>>Python-markdown-discuss mailing list

>>

>>>Pyt...@li...

>>

>>>https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss

>>

>> -------------------------------------------------------------------------

>> This SF.Net email is sponsored by the Moblin Your Move Developer's challen
ge

>> Build the coolest Linux based applications with Moblin SDK & win great pri
zes

>> Grand prize is a trip for two to an Open Source event anywhere in the worl
d

>> http://moblin-contest.org/redirect.php?banner_id=100&url=/

>> _______________________________________________

>> Python-markdown-discuss mailing list

>> Pyt...@li...

>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss

>>

>

>

>

>--

>----

>Waylan Limberg

>wa...@gm...

>

>-------------------------------------------------------------------------

>This SF.Net email is sponsored by the Moblin Your Move Developer's challenge

>Build the coolest Linux based applications with Moblin SDK & win great prize
s

>Grand prize is a trip for two to an Open Source event anywhere in the world

>http://moblin-contest.org/redirect.php?banner_id=100&url=/

>_______________________________________________

>Python-markdown-discuss mailing list

>Pyt...@li...

>https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss

Re: [Python-markdown-discuss] refactoring and treap integration

[Waylan L. <wa...@gm...>]

Well, we haven't release yet...

but the code is all in our git repo on gitorious.org [1]

[1]: http://gitorious.org/projects/python-markdown

On Mon, Oct 13, 2008 at 9:40 AM, Ben Wilson <bw...@da...> wrote:
>
> Anyway to get an advanced look at the new 2.0? I went looking on the site
>
> and only found 1.7...
>
>
>
> On 10/13/2008, "Yuri Takhteyev" <qar...@gm...> wrote:
>
>
>
>>I made a whole bunch of changes to the code, most of them just a
>
>>matter of refactoring, but some affect the functionality too.
>
>>
>
>>In terms of refactoring, the biggest change is splitting Markdown
>
>>class into three and making most of their methods private.  I think
>
>>this will make it easier to understand the code: you can now study one
>
>>class at a time.  Two of those three classes are still a little messy,
>
>>but at least the messiness is contained.
>
>>
>
>>So, we now have:
>
>>
>
>>1. MarkdownParser - parses pre-processed Markdown source into an ElementTree
> .
>
>>
>
>>Usage:
>
>>
>
>>    tree = MarkdownParser.parseDocument(markdown_string)
>
>>
>
>>The only other exposed methods are parseChunk() and detectTabbed().  I
>
>>am tempted to hide them as well, but at the moment they are needed by
>
>>some extensions.
>
>>
>
>>2. InlineProcessor - runs inline patterns on an ElementTree
>
>>
>
>>Usage:
>
>>
>
>>    InlineProcessor(patterns).applyInlinePatterns(tree)
>
>>
>
>>This is the only exposed method.  I also folded into this the InlineStash cl
> ass.
>
>>
>
>>3. Markdown - puts it all together.
>
>>
>
>>Usage:
>
>>
>
>>    Markdown(extensions).convert(markdown_string)
>
>>    Markdown(extensions).convertFile(input_file_path,
>
>>output_path_or_stream, encoding)
>
>>
>
>>markdownFromFile() function still exists, but only has two lines now.
>
>>
>
>>Another change, which does affects functionality, is that I
>
>>incorporated Ben's treap implementation as a way of organizing
>
>>pre-processors, patterns, etc.  This kills two birds: we now have a
>
>>better way of organizing those things, and this should also fix the
>
>>problem reported by Eric Abrahamsen last week, which required a major
>
>>change anyway.  This breaks many extensions, and also breaks one
>
>>non-extension test.  But 2.0 is about as good of a chance as we will
>
>>get for breaking backwards compatibility.
>
>>
>
>>I updated the footnotes extension as an example of how to use the new system
> .
>
>>
>
>>- yuri
>
>>
>
>>--
>
>>http://sputnik.freewisdom.org/
>
>>
>
>>-------------------------------------------------------------------------
>
>>This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>
>>Build the coolest Linux based applications with Moblin SDK & win great prize
> s
>
>>Grand prize is a trip for two to an Open Source event anywhere in the world
>
>>http://moblin-contest.org/redirect.php?banner_id=100&url=/
>
>>_______________________________________________
>
>>Python-markdown-discuss mailing list
>
>>Pyt...@li...
>
>>https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] refactoring and treap integration

[Ben W. <bw...@da...>]

Anyway to get an advanced look at the new 2.0? I went looking on the site

and only found 1.7...



On 10/13/2008, "Yuri Takhteyev" <qar...@gm...> wrote:



>I made a whole bunch of changes to the code, most of them just a

>matter of refactoring, but some affect the functionality too.

>

>In terms of refactoring, the biggest change is splitting Markdown

>class into three and making most of their methods private.  I think

>this will make it easier to understand the code: you can now study one

>class at a time.  Two of those three classes are still a little messy,

>but at least the messiness is contained.

>

>So, we now have:

>

>1. MarkdownParser - parses pre-processed Markdown source into an ElementTree
.

>

>Usage:

>

>    tree = MarkdownParser.parseDocument(markdown_string)

>

>The only other exposed methods are parseChunk() and detectTabbed().  I

>am tempted to hide them as well, but at the moment they are needed by

>some extensions.

>

>2. InlineProcessor - runs inline patterns on an ElementTree

>

>Usage:

>

>    InlineProcessor(patterns).applyInlinePatterns(tree)

>

>This is the only exposed method.  I also folded into this the InlineStash cl
ass.

>

>3. Markdown - puts it all together.

>

>Usage:

>

>    Markdown(extensions).convert(markdown_string)

>    Markdown(extensions).convertFile(input_file_path,

>output_path_or_stream, encoding)

>

>markdownFromFile() function still exists, but only has two lines now.

>

>Another change, which does affects functionality, is that I

>incorporated Ben's treap implementation as a way of organizing

>pre-processors, patterns, etc.  This kills two birds: we now have a

>better way of organizing those things, and this should also fix the

>problem reported by Eric Abrahamsen last week, which required a major

>change anyway.  This breaks many extensions, and also breaks one

>non-extension test.  But 2.0 is about as good of a chance as we will

>get for breaking backwards compatibility.

>

>I updated the footnotes extension as an example of how to use the new system
.

>

>- yuri

>

>--

>http://sputnik.freewisdom.org/

>

>-------------------------------------------------------------------------

>This SF.Net email is sponsored by the Moblin Your Move Developer's challenge

>Build the coolest Linux based applications with Moblin SDK & win great prize
s

>Grand prize is a trip for two to an Open Source event anywhere in the world

>http://moblin-contest.org/redirect.php?banner_id=100&url=/

>_______________________________________________

>Python-markdown-discuss mailing list

>Pyt...@li...

>https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss

Re: [Python-markdown-discuss] refactoring and treap integration

[Waylan L. <wa...@gm...>]

On Mon, Oct 13, 2008 at 4:40 AM, Yuri Takhteyev <qar...@gm...> wrote:
> I made a whole bunch of changes to the code, most of them just a
> matter of refactoring, but some affect the functionality too.
>
[snip]
> non-extension test.  But 2.0 is about as good of a chance as we will
> get for breaking backwards compatibility.
>

Wow! You were busy last night. And I agree, now is definitely the time
to make those changes.

Unless you beat me to it, I'll start working on the extensions as soon
as I can. And I'll update the writing_extensions.txt docs in the repo
once I'm confident in how everything works.

As a sidenote, I'm intrigued by the MarkdownParser class. One could
conceivably replace that class with their own which works differently
internally - as long as it has the same public methods and returns an
etree instance. This really opens up the possibility of
overriding/changing the core stuff. Cool! -- And if we want to change
the internal stuff, it should have little to no effect on the external
api.

-- 
----
Waylan Limberg
wa...@gm...

[Python-markdown-discuss] refactoring and treap integration

[Yuri T. <qar...@gm...>]

I made a whole bunch of changes to the code, most of them just a
matter of refactoring, but some affect the functionality too.

In terms of refactoring, the biggest change is splitting Markdown
class into three and making most of their methods private.  I think
this will make it easier to understand the code: you can now study one
class at a time.  Two of those three classes are still a little messy,
but at least the messiness is contained.

So, we now have:

1. MarkdownParser - parses pre-processed Markdown source into an ElementTree.

Usage:

    tree = MarkdownParser.parseDocument(markdown_string)

The only other exposed methods are parseChunk() and detectTabbed().  I
am tempted to hide them as well, but at the moment they are needed by
some extensions.

2. InlineProcessor - runs inline patterns on an ElementTree

Usage:

    InlineProcessor(patterns).applyInlinePatterns(tree)

This is the only exposed method.  I also folded into this the InlineStash class.

3. Markdown - puts it all together.

Usage:

    Markdown(extensions).convert(markdown_string)
    Markdown(extensions).convertFile(input_file_path,
output_path_or_stream, encoding)

markdownFromFile() function still exists, but only has two lines now.

Another change, which does affects functionality, is that I
incorporated Ben's treap implementation as a way of organizing
pre-processors, patterns, etc.  This kills two birds: we now have a
better way of organizing those things, and this should also fix the
problem reported by Eric Abrahamsen last week, which required a major
change anyway.  This breaks many extensions, and also breaks one
non-extension test.  But 2.0 is about as good of a chance as we will
get for breaking backwards compatibility.

I updated the footnotes extension as an example of how to use the new system.

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] wrapping character ranges in tags

[Eric A. <gi...@gm...>]

On Oct 7, 2008, at 1:39 PM, Yuri Takhteyev wrote:

> Which version are you using?  When I run this with the last released
> version (1.7), I actually get an error.
>
> If you want to try this with the latest code from git, then the
> following code does work:

Brilliant! That does work. I was using 1.7.0 R 66 before, it was  
definitely working (I copied and pasted the code exactly), just not  
producing the result I was after. Thanks very much.

Yours,
Eric


>
>
> class SpanPattern(markdown.Pattern):
>    def handleMatch(self, m):
>        el = etree.Element('span')
>        el.text=m.group(2)
>        el.set('class','char')
>        return el
>
> md.inlinePatterns.insert(-1,SpanPattern(ur'([\u4e00-\u9fff]+)'))
> print md.convert(u"including 沙發 shāfā - sofa")
>
> produces:
>
> <p>including <span class="char">沙發</span> shāfā - sofa</p>
>
> - yuri
>
> On Wed, Oct 1, 2008 at 2:04 AM, Eric Abrahamsen <gi...@gm...>  
> wrote:
>> Hi there,
>>
>> I've made a custom inline pattern to wrap Chinese characters in <span
>> class="char"></span> tags, using a regex (Chinese characters  
>> generally
>> fall between \u4e00 and \u9fff).
>>
>> The below works, but wraps each individual character in its own span
>> tags, rather than wrapping consecutive runs of characters in a single
>> set of tags:
>>
>> class SpanPattern(markdown.Pattern):
>>    def handleMatch(self, m, doc):
>>        el = doc.createElement('span')
>>        el.appendChild(doc.createTextNode(m.group(2)))
>>        el.setAttribute('class','char')
>>        return el
>>
>> md.inlinePatterns.insert(-1,SpanPattern(ur'([\u4e00-\u9fff]+)'))
>>
>> The result is the same whether the + is included in the regex or not;
>> is there some other trick I can use to make sure that five characters
>> in row, for instance, will get wrapped together in one pair of spans?
>>
>> TIA,
>>
>> Eric
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's  
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win  
>> great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in  
>> the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Python-markdown-discuss mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>>
>
>
>
> -- 
> http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Yuri T. <qar...@gm...>]

Fixed with 6d719bd
(http://gitorious.org/projects/python-markdown/repos/mainline/commits/6d719bd).

  - yuri

> sounds right !  thanks
>

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] wrapping character ranges in tags

[Yuri T. <qar...@gm...>]

Which version are you using?  When I run this with the last released
version (1.7), I actually get an error.

If you want to try this with the latest code from git, then the
following code does work:

class SpanPattern(markdown.Pattern):
    def handleMatch(self, m):
        el = etree.Element('span')
        el.text=m.group(2)
        el.set('class','char')
        return el

md.inlinePatterns.insert(-1,SpanPattern(ur'([\u4e00-\u9fff]+)'))
print md.convert(u"including 沙 shāfā - sofa")

produces:

<p>including <span class="char">沙</span> shāfā - sofa</p>

 - yuri

On Wed, Oct 1, 2008 at 2:04 AM, Eric Abrahamsen <gi...@gm...> wrote:
> Hi there,
>
> I've made a custom inline pattern to wrap Chinese characters in <span
> class="char"></span> tags, using a regex (Chinese characters generally
> fall between \u4e00 and \u9fff).
>
> The below works, but wraps each individual character in its own span
> tags, rather than wrapping consecutive runs of characters in a single
> set of tags:
>
> class SpanPattern(markdown.Pattern):
>     def handleMatch(self, m, doc):
>         el = doc.createElement('span')
>         el.appendChild(doc.createTextNode(m.group(2)))
>         el.setAttribute('class','char')
>         return el
>
> md.inlinePatterns.insert(-1,SpanPattern(ur'([\u4e00-\u9fff]+)'))
>
> The result is the same whether the + is included in the regex or not;
> is there some other trick I can use to make sure that five characters
> in row, for instance, will get wrapped together in one pair of spans?
>
> TIA,
>
> Eric
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Michael B. <mi...@zz...>]

On Oct 6, 2008, at 7:41 PM, Yuri Takhteyev wrote:

>> the "library" point of view would leave logging levels and handlers
>> undefined.  libraries just call  
>> logging.getLogger('MARKDOWN').debug("my
>> message"), and possibly isEnabledFor() (although that call is known  
>> to be
>> slow), and the configuration of the "MARKDOWN" logger is left to  
>> external
>> actors.
>
> Ok. And thanks for your clarification in your previous message, which
> for some reason landed in my inbox only after I responded to Waylan's
> response to your response.
>
> So: I'll move the logging configuration into main(), define a function
> getLogger() as a shorthand for logging.getLogger('MARKDOWN'), and do
> logging through it.  We won't configure logging except when getting
> called from the command line.  Does this sound like the right plan?


sounds right !  thanks

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Yuri T. <qar...@gm...>]

> the "library" point of view would leave logging levels and handlers
> undefined.  libraries just call logging.getLogger('MARKDOWN').debug("my
> message"), and possibly isEnabledFor() (although that call is known to be
> slow), and the configuration of the "MARKDOWN" logger is left to external
> actors.

Ok. And thanks for your clarification in your previous message, which
for some reason landed in my inbox only after I responded to Waylan's
response to your response.

So: I'll move the logging configuration into main(), define a function
getLogger() as a shorthand for logging.getLogger('MARKDOWN'), and do
logging through it.  We won't configure logging except when getting
called from the command line.  Does this sound like the right plan?

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Michael B. <mi...@zz...>]

On Oct 6, 2008, at 2:31 PM, Waylan Limberg wrote:

> Mike thanks for the bug report on Ticket 18 [1]. I'm copying the list
> with my response.
>
>> markdown.py has a hardcoded logging setting of DEBUG,
>> which allows all DEBUG messages to go out to all
>> configured handlers. There's then a comment which says
>> something to the effect of "this is restricted by handlers
>> later". I'd like to propose that this is the incorrect usage
>> of logging
>
> Actually, based on my testing the current behavior is correct. Suppose
> we set the logger to CRITICAL. Then, someone adds a handler (perhaps
> that writes to a log file) set to DEBUG. Because the logger only
> allows CRITICAL messages through, the handler would never receive
> anything less than CRITICAL even though its set to DEBUG. That's
> unacceptable - and why the logger is wide open and the handlers later
> restrict the level.
>
> The intention is that users would set (and/or remove) handlers and set
> the level for each handler as they need. In fact, we realize that the
> StreamHandler will break in some web server environments (it's
> indented for command line use) and may need to be removed/replaced
> with a handler appropriate for your system. That's why we recently
> adopted use of  Python's logger rather than our own old inflexible
> home-grown solution. It's much easier to override handlers than
> monkeypatch the old system.

The source comments indicate that this is the intention, but using  
handlers to filter log messages has the effect that *all* loggers, of  
all names, are filtered equally.  I dont see how this could ever be  
useful in an environment that has other loggers present besides  
"MARKDOWN".    Using it this way, there is simply no way for other  
libraries to have their logging output altered independently - you are  
controlling the water in the bathtub by turning the spigot on the  
central water main supplying the whole building, instead of the  
individual faucet.   You're not really getting any benefit from using  
logging in this way.   Theres a diagram at http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/overview.html#1.1 
  which might make this more visible (although both pep-282 and the  
java spec don't enunciate the distinction between "library" and  
"application environment").

> Actually I envisioned something very similar, except altering the
> *handler* attached to the logger, not the logger itself. Perhaps it's
> not the most convenient (in that it doesn't integrate with your
> environment's settings file), but a solution that will work in a
> variety of different situations is certainly welcome. I'm not
> personally familiar with Pylons and/or Paste, but would certainly like
> to see easy use of markdown in such a context. At the same time,
> markdown needs to continue to work as it does now from the command
> line as well as in other environments.

its simple.  markdown uses the "MARKDOWN" logger, and sends messages  
out using whichever levels it likes; i.e. logger.info(),  
logger.debug(), etc.  But it does *not* within markdown.py's module  
initialization set the actual log level or configure any handlers.   
This is because markdown.py in total is not a commandline application,  
its first and foremost a library - it should not assume awareness  
about the environment in which it runs, and especially not attempt to  
alter its runtime environment (which setting log levels and  
configuring handlers is).   logging was meant to be used in such a way  
that you don't actually access elements of the module itself - the  
getLogger() call and resulting object is the only interaction.

If you have a specific function that assumes a certain application  
environment and requires logging, that's fine - configure the handlers  
and logging levels there.  The specific settings of DEBUG and  
streamhandler are appropriate in this case to when one uses the  
markdownFromFile function.  *That* is where the environment-specific  
logger should be configured, since markdownFromFile assumes a  
particular environment; markdown.markdown() should not.  A global flag  
"loggingConfigured" may be used to detect that markdownFromFile has  
already been called to prevent the same configuration from being  
called twice.

In SQLAlchemy, we have a flag called "echo=True" which also sets up an  
environment-specific logging configuration.  But the point is, if the  
user does not set echo=True, SQLA doesn't set up any log configuration  
so that it's compatible with the rest of the world.  We also use a  
module-level flag to determine, when an echo=True is detected, if  
we've already set our "hardwired" logging environment.   We don't  
configure any kind of environmental log settings by default.

The Python logging module is modeled extremely closely to the Java  
log4j library which is the standard logger used by virtually all Java  
libraries.  The practices I'm advising here are the  same used within  
the Java community, which is how "logging" was meant to be used (PEP  
282 references the java spec and is largely copied verbatim).    
Otherwise, every Java library I imported into my application would be  
setting all kinds of log levels without my intervention, ignoring  
existing settings.

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Yuri T. <yu...@cs...>]

>> its simple.  markdown uses the "MARKDOWN" logger, and sends messages out
>> using whichever levels it likes; i.e. logger.info(), logger.debug(), etc.
>>  But it does *not* within markdown.py's module initialization set the actual
>> log level or configure any handlers.  This is because markdown.py in total
>> is not a commandline application, its first and foremost a library
>
> This is were things seem to fall apart. I agree here, however,
> whenever I've suggested changes of this type, I always get a response
> from Yuri that indicates he sees markdown first and foremost as a
> commandline script and second as a library. Perhaps we've gotten our
> wires crossed on this and that's not how he sees it - but I'm pretty
> sure he at least leans more in that direction that I do. Correct me if
> I'm wrong.

Actually, if I personally had to choose between markdown.py as a
library and as a command line application, I would probably choose the
library.  (If you just want a command line tool, you have more
alternatives, including Markdown.pl.)  However, I've always meant it
to work as both, and in this case I don't think we have a conflict
between them.  As far as command line use goes, we can just configure
the logger inside main().

So, the question is: from the point of view of library users, does it
make more sense to setup logging levels?  Or should we just leave them
as undefined?  One potential compromise would be to move all logging
setup into a function, leaving the library caller an option of calling
it:

    import markdown
    markdown.setup_logging()

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Michael B. <mi...@zz...>]

On Oct 6, 2008, at 4:27 PM, Yuri Takhteyev wrote:

>>>
> So, the question is: from the point of view of library users, does it
> make more sense to setup logging levels?  Or should we just leave them
> as undefined?  One potential compromise would be to move all logging
> setup into a function, leaving the library caller an option of calling
> it:
>
>    import markdown
>    markdown.setup_logging()

the "library" point of view would leave logging levels and handlers  
undefined.  libraries just call  
logging.getLogger('MARKDOWN').debug("my message"), and possibly  
isEnabledFor() (although that call is known to be slow), and the  
configuration of the "MARKDOWN" logger is left to external actors.

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Waylan L. <wa...@gm...>]

On Mon, Oct 6, 2008 at 3:02 PM, Michael Bayer <mi...@zz...> wrote:
[snip]
> its simple.  markdown uses the "MARKDOWN" logger, and sends messages out
> using whichever levels it likes; i.e. logger.info(), logger.debug(), etc.
>  But it does *not* within markdown.py's module initialization set the actual
> log level or configure any handlers.  This is because markdown.py in total
> is not a commandline application, its first and foremost a library

This is were things seem to fall apart. I agree here, however,
whenever I've suggested changes of this type, I always get a response
from Yuri that indicates he sees markdown first and foremost as a
commandline script and second as a library. Perhaps we've gotten our
wires crossed on this and that's not how he sees it - but I'm pretty
sure he at least leans more in that direction that I do. Correct me if
I'm wrong.

> - it should not assume awareness about the environment in which it runs, and
> especially not attempt to alter its runtime environment (which setting log
> levels and configuring handlers is).   logging was meant to be used in such
> a way that you don't actually access elements of the module itself - the
> getLogger() call and resulting object is the only interaction.
>
> If you have a specific function that assumes a certain application
> environment and requires logging, that's fine - configure the handlers and
> logging levels there.  The specific settings of DEBUG and streamhandler are
> appropriate in this case to when one uses the markdownFromFile function.
>  *That* is where the environment-specific logger should be configured, since
> markdownFromFile assumes a particular environment; markdown.markdown()
> should not.  A global flag "loggingConfigured" may be used to detect that
> markdownFromFile has already been called to prevent the same configuration
> from being called twice.

I like this suggestion. Although I'm not so sure the
"loggingConfigured" flag is necessary. Thanks for your input. I'll
play around with this and see what I can come up with. Unfortunately,
I'm a little busy right now, so it may be a few weeks.



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Ticket 18: DEBUG log setting

[Yuri T. <qar...@gm...>]

What about something like this:

if logger.level = logging.NOTSET :
    logger.setLevel(DEBUG)

This way the caller can set the logging level before importing markdown:

>>> import logging
>>> logging.getLogger("MARKDOWN").setLevel(logging.critical)
>>> import markdown
>>> markdown.logger.level
<function critical at 0xb7c19a3c>

(I committed this change as 2c6a74f, in case you want to look at it.
But feel free to revert if we come up with a better solution.)

  - yuri

On Mon, Oct 6, 2008 at 11:31 AM, Waylan Limberg <wa...@gm...> wrote:
> Mike thanks for the bug report on Ticket 18 [1]. I'm copying the list
> with my response.
>
>> markdown.py has a hardcoded logging setting of DEBUG,
>> which allows all DEBUG messages to go out to all
>> configured handlers. There's then a comment which says
>> something to the effect of "this is restricted by handlers
>> later". I'd like to propose that this is the incorrect usage
>> of logging
>
> Actually, based on my testing the current behavior is correct. Suppose
> we set the logger to CRITICAL. Then, someone adds a handler (perhaps
> that writes to a log file) set to DEBUG. Because the logger only
> allows CRITICAL messages through, the handler would never receive
> anything less than CRITICAL even though its set to DEBUG. That's
> unacceptable - and why the logger is wide open and the handlers later
> restrict the level.
>
> The intention is that users would set (and/or remove) handlers and set
> the level for each handler as they need. In fact, we realize that the
> StreamHandler will break in some web server environments (it's
> indented for command line use) and may need to be removed/replaced
> with a handler appropriate for your system. That's why we recently
> adopted use of  Python's logger rather than our own old inflexible
> home-grown solution. It's much easier to override handlers than
> monkeypatch the old system.
>
>> The workaround for now is that you have to hardcode a call
>> to "logging.getLogger("MARKDOWN").setLevel(logging.WARN)"
>> to silence the MARKDOWN logger in your application code
>
> Actually I envisioned something very similar, except altering the
> *handler* attached to the logger, not the logger itself. Perhaps it's
> not the most convenient (in that it doesn't integrate with your
> environment's settings file), but a solution that will work in a
> variety of different situations is certainly welcome. I'm not
> personally familiar with Pylons and/or Paste, but would certainly like
> to see easy use of markdown in such a context. At the same time,
> markdown needs to continue to work as it does now from the command
> line as well as in other environments.
>
> If anyone has any suggestions and/or solutions I'm all ears.
>
> [1]: http://www.freewisdom.org/projects/python-markdown/Tickets/000018
>
> --
> ----
> Waylan Limberg
> wa...@gm...
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
http://sputnik.freewisdom.org/