python-markdown-discuss — A general purpose discussion list

Re: [Python-markdown-discuss] Extra Whitespace in Output

[Artem Y. <ne...@gm...>]

Waylan Limberg wrote:
> On Wed, Aug 27, 2008 at 6:16 PM, Waylan Limberg <wa...@gm...> wrote:
>   
>>    >>> markdown.markdown('foo *bar __blah__* baz')
>>    u'<p>foo <em>bar <strong>blah</strong>\n  </em> baz</p>'
>>
>> What's up with the newline and space between the closing tags
>> ``</strong>`` & ``</em>``? Is that from the (IMO unnecessary)
>> ``IndentTree`` function or something in
>> ``Markdown._processPlaceholders``? I'm not sure.
>>
>>     
>
> Ok, I just realized that that is the output we were getting before my
> change to the inline placeholder. Weird! IMO that is not correct. We
> need to loose that extra whitespace. I realize it's irrelevant in the
> browser, but no other markdown implementation does that. Either the
> indentTree function needs to only work on block level elements or it
> needs to go away.
>
>   

Yes, it's because of indentETree function. We don't have now any other 
way of managing output, maybe in future releases ElementTree will get 
some pretty print function, but there is no such for now. So we should 
modify indentETree, but we shouldn't throw it away :) Looked at your 
placeholder branch, now extractId looks much prettier and cleaner :)

[Python-markdown-discuss] Extra Whitespace in Output

[Waylan L. <wa...@gm...>]

On Wed, Aug 27, 2008 at 6:16 PM, Waylan Limberg <wa...@gm...> wrote:
>
>    >>> markdown.markdown('foo *bar __blah__* baz')
>    u'<p>foo <em>bar <strong>blah</strong>\n  </em> baz</p>'
>
> What's up with the newline and space between the closing tags
> ``</strong>`` & ``</em>``? Is that from the (IMO unnecessary)
> ``IndentTree`` function or something in
> ``Markdown._processPlaceholders``? I'm not sure.
>

Ok, I just realized that that is the output we were getting before my
change to the inline placeholder. Weird! IMO that is not correct. We
need to loose that extra whitespace. I realize it's irrelevant in the
browser, but no other markdown implementation does that. Either the
indentTree function needs to only work on block level elements or it
needs to go away.


-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] AtomicString

[Waylan L. <wa...@gm...>]

I haven't added to this discussion yet as I wasn't sure what position
to take. Here's my thoughts, observations and almost working solution:

Everyone seems to be going back and forth on which random string
generator is better. Personally I'm wondering what all the fuss is
about. What we want is a unique string that identifies said string as
a placeholder for a specific item in a stash. We have 2 stashes
(rawHtml and inline) so we also need to identify which stash. The
thing is, the "start" and "end" chars give us the uniqueness that
identifies the string as a placeholder. If we only had one stash, all
we would need is the id number.

So the question then, is how do we identify which stash this
placeholder is for? Currently, each stash's placeholder either
contains the string "inline" or "html" (there are currently a couple
other subtle differences but there easily removable). Now, as the
current wikilink bug demonstrates, using actual real words that could
legitimately appear in the document and perhaps even have patterns
matching against it causes problems. So, we need 2 strings that will
never (or at least very unlikely) be matched by any other pattern.

The popular solutions in this dicusion thus far seem to have a string
of random chars generated at import time. Depending on the generation
method used, there will be x chances of a collision with a real, valid
string. Obviously, the higher x is, the better - or so it seems.
Suppose I am serving a document via a cgi script which will cause an
import and a new, different random string on each page view. I only
have x page views before there will be a collision. Whoops! Now try
debugging that!

Therefore, I propose that we select 2 strings of random chars (using
whatever method you desire) and **hardcode** those 2 strings into
markdown.py. That way, on each import (each page view in the above
scenario) the placeholder strings will be the same and debugging will
be consistent.  What we really want is a string that will never be
matched by another inline pattern's regex. We just need a string of
all same-case chars between a-z of length n. As long as it does not
contain any known words or abbreviations it works for me.
Additionally, if the string is consistent, that makes it easier for an
extension author to write  the regex for inline patterns that will not
match the string in the placeholder.

I have commited an *almost* working branch [1] that has everything
except the random strings (it still uses "inline" & "html"). I say
"almost working" because the output includes a lot of extra,
unnecessary whitespace. The problem is not creating the placeholder,
but replacing the placeholder with the real content later - at least
the way Artem's code works. Based upon docstrings, I determined that I
needed to refactor ``InlineStash.extractId``, which I did. However, it
seems that Artem's code was jumping through an awful lot of hoops and
I haven't fully groked what ``Markdown._processPlaceholders`` is doing
when it calls ``InlineStash.extractId``. Wouldn't it be better if we
simply used the indexes ``m.start`` and ``m.end`` from a regex match
rather than the string manipulation hoops it's doing now?

Once we get that worked out, I'll replace the strings "inline" &
"html" with something more random. Here's the output of a few simple
tests:

    >>> markdown.markdown('foo *bar* baz')
    u'<p>foo <em>bar</em> baz</p>'

    >>> markdown.markdown('foo *bar __blah__* baz')
    u'<p>foo <em>bar <strong>blah</strong>\n  </em> baz</p>'

What's up with the newline and space between the closing tags
``</strong>`` & ``</em>``? Is that from the (IMO unnecessary)
``IndentTree`` function or something in
``Markdown._processPlaceholders``? I'm not sure.

Any thoughts?

[1]: http://gitorious.org/projects/python-markdown/repos/mainline/commits/ab57ff93b5b2750c082c87072ced774881190744

On Tue, Aug 26, 2008 at 9:22 AM, David Wolever <wo...@cs...> wrote:
> On 25-Aug-08, at 7:55 PM, Artem Yunusov wrote:
>> Yes, I agree, it's not necessary here. But thanks David, I didn't know
>> about it before, and used to use md5 for such things.
> Ah, well, I'm glad I could be helpful anyway :)
>
> And, re: more readable:
>> "abcdefghijklmnopqrstuvwxyz"[random.randint(1,26)]
> I agree, that's pretty nice :)
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Codehilite extension bug

[Waylan L. <wa...@gm...>]

On Tue, Aug 26, 2008 at 1:23 PM, Kjell Magne Fauske <kj...@gm...> wrote:
>
> Regarding breaking css, I have filed a new ticket #16 and asked for
> the addition of an option for setting the name of the wrapper css
> class:
>
> http://www.freewisdom.org/projects/python-markdown/Tickets/000016
>
> Patch is attached, but I have probably overlooked something.
>

Yeah I saw that. It looks good from a quick glance. I'll add it to my
list. Not sure why I didn't do that to begin with.


-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Codehilite extension bug

[Kjell M. F. <kj...@gm...>]

On Tue, Aug 26, 2008 at 1:05 PM, Waylan Limberg <wa...@gm...> wrote:
> On Tue, Aug 26, 2008 at 11:40 AM, Kjell Magne Fauske <kj...@gm...> wrote:
>> On Tue, Aug 26, 2008 at 9:58 AM, Waylan Limberg <wa...@gm...> wrote:
>>> Thanks Kjell. I suspected it was something like that. I just haven't
>>> had the time to work on it. I was thinking perhaps I would have the
>>> highlighter return the highlighted source without a div wrapping it,
>>> then we could place that within a ET generated div. IIRC, there is a
>>> way to get pygments to do that, but I couldn't tell you off the top of
>>> my head what it is. If you want to give that a shot - feel free.
>>> Otherwise I'll get to it when I have the time.
>>>
>>
>> I now think I have found a fix. You can find the code attached.  I am
>> not familiar with Git, so I'm not sure if the patch is correctly
>> formatted.
>>
>
> Thanks Kjell, your patch was fine - and in the simple case it works.
> However, when pygments adds line numbers to the source, the source
> (and containing div) are wrapped in a table. However, with your patch
> the table is wrapped in the div instead. I realize the div is only
> there for styling hooks, but if css authors are expecting the div to
> be a child of the table, that might break their css - so we need to be
> consistent. After all, the idea is that people can use any of the
> publicly available styles for pygments without any modification to
> work with markdown.
>
> So I came up with another solution over lunch - which occurred to me
> after I sent you my last message. See the comment and diff here:
> http://gitorious.org/projects/python-markdown/repos/mainline/commits/a4c80246770b87ca3d2cce5b5184901fd2495a7d
>
> --
> ----
> Waylan Limberg
> wa...@gm...
>

Great! Your solution was much cleaner than mine. No hard feelings :)

Regarding breaking css, I have filed a new ticket #16 and asked for
the addition of an option for setting the name of the wrapper css
class:

http://www.freewisdom.org/projects/python-markdown/Tickets/000016

Patch is attached, but I have probably overlooked something.

Regards,
Kjell Magne Fauske

Re: [Python-markdown-discuss] Codehilite extension bug

[Waylan L. <wa...@gm...>]

On Tue, Aug 26, 2008 at 1:05 PM, Waylan Limberg <wa...@gm...> wrote:
>
> So I came up with another solution over lunch - which occurred to me
> after I sent you my last message. See the comment and diff here:
> http://gitorious.org/projects/python-markdown/repos/mainline/commits/a4c80246770b87ca3d2cce5b5184901fd2495a7d
>

Btw Kjell, don't feel bad that I didn't use your patch. I would have
tried the same thing first myself before realizing the problem with it
- which means we wouldn't have a fix yet. So thanks for saving me the
time. You were a big help.



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Codehilite extension bug

[Waylan L. <wa...@gm...>]

On Tue, Aug 26, 2008 at 11:40 AM, Kjell Magne Fauske <kj...@gm...> wrote:
> On Tue, Aug 26, 2008 at 9:58 AM, Waylan Limberg <wa...@gm...> wrote:
>> Thanks Kjell. I suspected it was something like that. I just haven't
>> had the time to work on it. I was thinking perhaps I would have the
>> highlighter return the highlighted source without a div wrapping it,
>> then we could place that within a ET generated div. IIRC, there is a
>> way to get pygments to do that, but I couldn't tell you off the top of
>> my head what it is. If you want to give that a shot - feel free.
>> Otherwise I'll get to it when I have the time.
>>
>
> I now think I have found a fix. You can find the code attached.  I am
> not familiar with Git, so I'm not sure if the patch is correctly
> formatted.
>

Thanks Kjell, your patch was fine - and in the simple case it works.
However, when pygments adds line numbers to the source, the source
(and containing div) are wrapped in a table. However, with your patch
the table is wrapped in the div instead. I realize the div is only
there for styling hooks, but if css authors are expecting the div to
be a child of the table, that might break their css - so we need to be
consistent. After all, the idea is that people can use any of the
publicly available styles for pygments without any modification to
work with markdown.

So I came up with another solution over lunch - which occurred to me
after I sent you my last message. See the comment and diff here:
http://gitorious.org/projects/python-markdown/repos/mainline/commits/a4c80246770b87ca3d2cce5b5184901fd2495a7d

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Codehilite extension bug

[Kjell M. F. <kj...@gm...>]

On Tue, Aug 26, 2008 at 9:58 AM, Waylan Limberg <wa...@gm...> wrote:
> Thanks Kjell. I suspected it was something like that. I just haven't
> had the time to work on it. I was thinking perhaps I would have the
> highlighter return the highlighted source without a div wrapping it,
> then we could place that within a ET generated div. IIRC, there is a
> way to get pygments to do that, but I couldn't tell you off the top of
> my head what it is. If you want to give that a shot - feel free.
> Otherwise I'll get to it when I have the time.
>

I now think I have found a fix. You can find the code attached.  I am
not familiar with Git, so I'm not sure if the patch is correctly
formatted.

Regards,
Kjell Magne Fauske

Re: [Python-markdown-discuss] Codehilite extension bug

[Waylan L. <wa...@gm...>]

Thanks Kjell. I suspected it was something like that. I just haven't
had the time to work on it. I was thinking perhaps I would have the
highlighter return the highlighted source without a div wrapping it,
then we could place that within a ET generated div. IIRC, there is a
way to get pygments to do that, but I couldn't tell you off the top of
my head what it is. If you want to give that a shot - feel free.
Otherwise I'll get to it when I have the time.

On Tue, Aug 26, 2008 at 4:20 AM, Kjell Magne Fauske <kj...@gm...> wrote:
> Dear Python-markdown developers,
>
> I have taken a closer look at ticket #12 regarding the bug in
> codehilite that inserts the formatted code at the start of the
> document. The problem seems to be that the code is inserted in the
> parent node's text attribute. According to the ElementTree
> documentation [1] the text attribute is text *before* the first
> subelement. That is why the code is inserted at the start of the
> document.
>
> I have posted a partial solution to the problem as a comment to the ticket:
> http://www.freewisdom.org/projects/python-markdown/Tickets/000012
>
> Regards,
> Kjell Magne Fauske
>
> [1] http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.ElementTree._ElementInterface.text-attribute
>
> ( Why can't I find this information in Python's documentation of ElementTree? )
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Python-markdown-discuss mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/python-markdown-discuss
>



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] AtomicString

[David W. <wo...@cs...>]

On 25-Aug-08, at 7:55 PM, Artem Yunusov wrote:
> Yes, I agree, it's not necessary here. But thanks David, I didn't know
> about it before, and used to use md5 for such things.
Ah, well, I'm glad I could be helpful anyway :)

And, re: more readable:
> "abcdefghijklmnopqrstuvwxyz"[random.randint(1,26)]
I agree, that's pretty nice :)

[Python-markdown-discuss] Codehilite extension bug

[Kjell M. F. <kj...@gm...>]

Dear Python-markdown developers,

I have taken a closer look at ticket #12 regarding the bug in
codehilite that inserts the formatted code at the start of the
document. The problem seems to be that the code is inserted in the
parent node's text attribute. According to the ElementTree
documentation [1] the text attribute is text *before* the first
subelement. That is why the code is inserted at the start of the
document.

I have posted a partial solution to the problem as a comment to the ticket:
http://www.freewisdom.org/projects/python-markdown/Tickets/000012

Regards,
Kjell Magne Fauske

[1] http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.ElementTree._ElementInterface.text-attribute

( Why can't I find this information in Python's documentation of ElementTree? )

Re: [Python-markdown-discuss] Lying tests

[Yuri T. <qar...@gm...>]

Waylan, thanks for noticing this!  How confident do you feel about the
tests at this point?  Perhaps we should do more test testing.

I just tried running John Gruber's test suite (MarkdownTest.pl) with
--tidy option and it appears that the current version does pass the
old version of the test (from 2004), which is what I used before
writing test_markdown.py.  However, it fails some of the new ones.
This is not a regression, since those represent the more recent
changes to Markdown.pl and we most probably never passed them.

Just so that we are all on the same page, I committed those tests into
our tree.  I also modified MarkdownTest.pl to work with the version of
tidy that came with Ubuntu 7.10.

To run older tests:

perl MarkdownTest/MarkdownTest.pl --tidy --script ./markdown.py
--testdir MarkdownTest/Tests_2004

(this will pass)

perl MarkdownTest/MarkdownTest.pl --tidy --script ./markdown.py
--testdir MarkdownTest/Tests_2007

(some of those will fail)

 - yuri

>> I fixed bugs, all test, except codehilite and wikilink passes now, by
>> the way the same problems exists in Markdown 1.7, it's legacy problems.
>>
> Thanks Artem. Well, yes, the bug you fixed was legacy, but the ones I
> fixed yesterday were not. See commit fa014ac. I only changed one line
> in markdown.py, yet the patch included 956 lines changed in 18 files.
> The rest were all changes to the tests - tests that used to be right
> back in 1.7.

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] AtomicString

[Artem Y. <ne...@gm...>]

Yuri Takhteyev wrote:
>> Now, if we're going to be arguing about generating universally unique
>> identifiers... I feel like I should chime in with one module I've been
>> making heavy use of recently: uuid!
>>     
>
> This is only included with python2.5.  We _could_ do something like
> what we do now with other things, i.e., wrap it and fall back on
> something else if uuid module cannot be imported.  Or we can tell
> people to install an extra uuid module if they are using older
> versions of python.  The question is: do we get any benefits from real
> uniqueness?
>
> (Before we get back into the question of whether anyone still uses old
> versions of Python: Django can run on 2.3 and as long as they can, I
> think so should we.)
>
> -  yuri
>
>   

Yes, I agree, it's not necessary here. But thanks David, I didn't know 
about it before, and used to use md5 for such things.

Re: [Python-markdown-discuss] AtomicString

[Artem Y. <ne...@gm...>]

Yuri Takhteyev wrote:
>> def getRandomToken(n=10, seed=RND_SEED):
>>    random.seed(seed)
>>    return "".join([chr(random.randint(97, 122)) for i in range(n)])
>>     
>
> Personally, I tend to use a static string with all the letters in
> cases like this just to save the reader from having to remember or
> guess what characters are between chr(97) to chr(122).  Sometimes a
> "dumber" solution is just easier to understand.  Eitherway, there
> should be no magic numbers in the code, so if we want to use chr()
> then the proper thing to do would be to define constants
>
> LOWERCASE_ASCII_A = 97
> LOWERCASE_ASCII_Z =122
>
> and then later:
>
> return "".join([chr(random.randint(LOWERCASE_ASCII_A,
> LOWERCASE_ASCII_Z)) for i in range(n)])
>
> And then suddenly "abcdefghijklmnopqrstuvwxyz"[random.randint(1,26)]
> doesn't look so verbose. :)
>
> - yuri
>
>   

Yes, in terms of readability, your variant looks better, let's decide in 
favour of it.

Re: [Python-markdown-discuss] AtomicString

[Yuri T. <qar...@gm...>]

> Now, if we're going to be arguing about generating universally unique
> identifiers... I feel like I should chime in with one module I've been
> making heavy use of recently: uuid!

This is only included with python2.5.  We _could_ do something like
what we do now with other things, i.e., wrap it and fall back on
something else if uuid module cannot be imported.  Or we can tell
people to install an extra uuid module if they are using older
versions of python.  The question is: do we get any benefits from real
uniqueness?

(Before we get back into the question of whether anyone still uses old
versions of Python: Django can run on 2.3 and as long as they can, I
think so should we.)

-  yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] AtomicString

[David W. <wo...@cs...>]

On 25-Aug-08, at 3:19 PM, Yuri Takhteyev wrote:

>> def getRandomToken(n=10, seed=RND_SEED):
>>    random.seed(seed)
>>    return "".join([chr(random.randint(97, 122)) for i in range(n)])
> ....
> return "".join([chr(random.randint(LOWERCASE_ASCII_A,
> LOWERCASE_ASCII_Z)) for i in range(n)])
> And then suddenly "abcdefghijklmnopqrstuvwxyz"[random.randint(1,26)]
> doesn't look so verbose. :)

Now, if we're going to be arguing about generating universally unique  
identifiers... I feel like I should chime in with one module I've  
been making heavy use of recently: uuid!

 >>> from uuid import uuid4
 >>> uuid4().hex
'6908a7924f4d46bf915da19876c93368'
 >>>

Re: [Python-markdown-discuss] AtomicString

[Yuri T. <qar...@gm...>]

> def getRandomToken(n=10, seed=RND_SEED):
>    random.seed(seed)
>    return "".join([chr(random.randint(97, 122)) for i in range(n)])

Personally, I tend to use a static string with all the letters in
cases like this just to save the reader from having to remember or
guess what characters are between chr(97) to chr(122).  Sometimes a
"dumber" solution is just easier to understand.  Eitherway, there
should be no magic numbers in the code, so if we want to use chr()
then the proper thing to do would be to define constants

LOWERCASE_ASCII_A = 97
LOWERCASE_ASCII_Z =122

and then later:

return "".join([chr(random.randint(LOWERCASE_ASCII_A,
LOWERCASE_ASCII_Z)) for i in range(n)])

And then suddenly "abcdefghijklmnopqrstuvwxyz"[random.randint(1,26)]
doesn't look so verbose. :)

- yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] AtomicString

[Artem Y. <ne...@gm...>]

Yuri Takhteyev wrote:
>> Maybe we should use some random hashes, like
>> `md5.new(str(random.random())).hexdigest()` ?
>> I don't think that users will be handle with placeholders, in case if
>> everything works fine.
>>     
>
> We could do that, though it might be better to avoid any actual
> randomness by using a fixed seed:
>
> def reset_placeholders(seed = DEFAULT_SEED) :
>     self.seed = seed
>     random.seed(seed)
>     self.html_placeholder_prefix  = STX+"%x" % (random.random()*1000000000)
>     self.html_placeholder  = self.html_placeholder.prefix  + "%x" + ETX
>     ...
>
> and later generate HTML placeholders with
>
>      random.seed(self.seed + i)
>      self.html_placeholder % (random.random()*1000000000)
>
> Or something more like:
>
> def get_random_token(n = 10)
>     return "".join(["abcdefghijklmnopqrstuvwxyx"[int(random.random()*26)]
> for i in range(n)])
>
> def reset_placeholders(seed = DEFAULT_SEED) :
>     random.seed(seed)
>     self.html_placeholder_prefix  = STX + get_random_token()
>     self.html_placeholder  = self.html_placeholder_prefix  + "%s" + ETX
>
>  - yuri
>
>   

I agree, using seed is better idea.

What about something like this:

RND_SEED = 789654
 
def getRandomToken(n=10, seed=RND_SEED):
    random.seed(seed)
    return "".join([chr(random.randint(97, 122)) for i in range(n)])

STX = u'\u0002'  # Use STX ("Start of text") for start-of-placeholder
ETX = u'\u0003'  # Use ETX ("End of text") for end-of-placeholder
HTML_PLACEHOLDER_PREFIX = STX + getRandomToken()
HTML_PLACEHOLDER = HTML_PLACEHOLDER_PREFIX + "%s" + ETX
INLINE_PLACEHOLDER_PREFIX = STX + getRandomToken()
INLINE_PLACEHOLDER = INLINE_PLACEHOLDER_PREFIX + "%s" + ETX

or maybe wrap it in class, as you did.

Re: [Python-markdown-discuss] Lying tests

[Waylan L. <wa...@gm...>]

On Sat, Aug 23, 2008 at 4:30 PM, Artem Yunusov <ne...@gm...> wrote:
> Waylan Limberg wrote:
>> I've been doing the former, and in the process I've looked at a few
>> tests and discovered additional problems. When I have, I've been
>> updating the tests to what they should be *even if I haven't fixed the
>> bug*. That means those tests are still failing. Please don't change
>> the test - fix the bug so the test passes. Or, if you disagree with
>> what the test should expect, bring it up here for discussion.
>>
>>
>
> I fixed bugs, all test, except codehilite and wikilink passes now, by
> the way the same problems exists in Markdown 1.7, it's legacy problems.
>
Thanks Artem. Well, yes, the bug you fixed was legacy, but the ones I
fixed yesterday were not. See commit fa014ac. I only changed one line
in markdown.py, yet the patch included 956 lines changed in 18 files.
The rest were all changes to the tests - tests that used to be right
back in 1.7.



-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] Lying tests

[Artem Y. <ne...@gm...>]

Waylan Limberg wrote:
> I've been doing the former, and in the process I've looked at a few
> tests and discovered additional problems. When I have, I've been
> updating the tests to what they should be *even if I haven't fixed the
> bug*. That means those tests are still failing. Please don't change
> the test - fix the bug so the test passes. Or, if you disagree with
> what the test should expect, bring it up here for discussion.
>
>   

I fixed bugs, all test, except codehilite and wikilink passes now, by 
the way the same problems exists in Markdown 1.7, it's legacy problems.

Re: [Python-markdown-discuss] Lying tests

[Artem Y. <ne...@gm...>]

Waylan Limberg wrote:
> Just an FYI to everyone. I've come to realize more than ever today
> that just because the tests included in the repo are passing does not
> mean *anything*. If you look at the numerous commits I've made today,
> most are changes to the tests themselves. I've spent hours fixing bad
> tests and only minutes on actual bugs in the code.
>
> It seems that when the tests were altered to fit the output of
> ElemenTree, a large number of new bugs have been introduced into the
> tests. So while it looks like everything is ok - I've found a few
> glaring blunders. The only way to catch them is to either do your own
> testing outside the provided tests and fix the bugs  - then run the
> tests and fix all the tests that fail, or to review each and every
> test by hand and try to spot problems.
>   

Hmm, tests modification was done in that way:
I changed output of NanoDOM, without any changes in core, then I ran 
through all tests and wrote results to separate files.
So, I think problems concerning switching to ElementTree  could only 
exist in TextPostprocessors. We have only 2  TextPostprocessors, I don't 
think that there are a lot of problems concerning this issue.
Also, before porting to ElementTree and modify tests, I found some 
incorrect tests too, most of them concerned inline patterns.

And now I realize, why changing of NanoDOM output changed this output: 
<hr />, to <p><hr /></p> :) I even created branch called stripped-dom, 
to find out what the problem is, but later I decided to I add 
Markdown._processHR function.

[Python-markdown-discuss] Lying tests

[Waylan L. <wa...@gm...>]

Just an FYI to everyone. I've come to realize more than ever today
that just because the tests included in the repo are passing does not
mean *anything*. If you look at the numerous commits I've made today,
most are changes to the tests themselves. I've spent hours fixing bad
tests and only minutes on actual bugs in the code.

It seems that when the tests were altered to fit the output of
ElemenTree, a large number of new bugs have been introduced into the
tests. So while it looks like everything is ok - I've found a few
glaring blunders. The only way to catch them is to either do your own
testing outside the provided tests and fix the bugs  - then run the
tests and fix all the tests that fail, or to review each and every
test by hand and try to spot problems.

I've been doing the former, and in the process I've looked at a few
tests and discovered additional problems. When I have, I've been
updating the tests to what they should be *even if I haven't fixed the
bug*. That means those tests are still failing. Please don't change
the test - fix the bug so the test passes. Or, if you disagree with
what the test should expect, bring it up here for discussion.

Even better, spend some time reviewing the tests for more errors and
submit patches. :-)

-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] AtomicString

[Yuri T. <qar...@gm...>]

> Maybe we should use some random hashes, like
> `md5.new(str(random.random())).hexdigest()` ?
> I don't think that users will be handle with placeholders, in case if
> everything works fine.

We could do that, though it might be better to avoid any actual
randomness by using a fixed seed:

def reset_placeholders(seed = DEFAULT_SEED) :
    self.seed = seed
    random.seed(seed)
    self.html_placeholder_prefix  = STX+"%x" % (random.random()*1000000000)
    self.html_placeholder  = self.html_placeholder.prefix  + "%x" + ETX
    ...

and later generate HTML placeholders with

     random.seed(self.seed + i)
     self.html_placeholder % (random.random()*1000000000)

Or something more like:

def get_random_token(n = 10)
    return "".join(["abcdefghijklmnopqrstuvwxyx"[int(random.random()*26)]
for i in range(n)])

def reset_placeholders(seed = DEFAULT_SEED) :
    random.seed(seed)
    self.html_placeholder_prefix  = STX + get_random_token()
    self.html_placeholder  = self.html_placeholder_prefix  + "%s" + ETX

 - yuri

-- 
http://sputnik.freewisdom.org/

Re: [Python-markdown-discuss] AtomicString

[Waylan L. <wa...@gm...>]

On Fri, Aug 22, 2008 at 3:49 PM, Yuri Takhteyev <qar...@gm...> wrote:
[snip]
> with our placeholders via extensions.  So, I we should do both: use a
> meaningless combination of letters (without any punctuation), and then
> wrap it with characters that users aren't allowed to put in the input
> (STX and ETX).  E.g.:
>
> STX = u'\u0002'  # Use STX ("Start of text") for start-of-placeholder
> ETX = u'\u0003'  # Use ETX ("End of text") for end-of-placeholder
> HTML_PLACEHOLDER_PREFIX = STX+"wyxhzde38k"
> HTML_PLACEHOLDER = HTML_PLACEHOLDER_PREFIX + "%d"+ETX
> INLINE_PLACEHOLDER_PREFIX = STX+"0ix2bavflj"
> INLINE_PLACEHOLDER_SUFFIX = ETX
> AMP_SUBSTITUTE = STX+"k75lziz62a"+ETX
>
> Actually, come to think of it, perhaps even that %d is not a good idea.
>
> (I am not checking this in, since Waylan seems to be actively working
> on the file.)
>

Go ahead Yuri. I'm working on something else right now.


-- 
----
Waylan Limberg
wa...@gm...

Re: [Python-markdown-discuss] AtomicString

[Artem Y. <ne...@gm...>]

Yuri Takhteyev wrote:
>> There is absolutely no camelcase words in the text we pass in, so no
>> wikilinks should be generated. However, it appears that the
>> placeholder for the "markdownlink" is inserted into the text as
>> ``\x02LinkPattern:000\x03``. Normally, this would later be replaced by
>>     
>
> Yeah, that's an whoops.  I think rather than just avoiding camelcase
> in placeholders, we should avoid anything meangingfull in them at all,
> apart from STX and ETX codes.  We used to have some random combination
> of characters.  Adding STX and ETX around it made it safer against us
> trying to replace the occurrence of the placeholder in the original
> text.  However, switching from a random combination to meaningful
> things like "LinkPattern" creates the possibility of users messing
> with our placeholders via extensions.  So, I we should do both: use a
> meaningless combination of letters (without any punctuation), and then
> wrap it with characters that users aren't allowed to put in the input
> (STX and ETX).  E.g.:
>
> STX = u'\u0002'  # Use STX ("Start of text") for start-of-placeholder
> ETX = u'\u0003'  # Use ETX ("End of text") for end-of-placeholder
> HTML_PLACEHOLDER_PREFIX = STX+"wyxhzde38k"
> HTML_PLACEHOLDER = HTML_PLACEHOLDER_PREFIX + "%d"+ETX
> INLINE_PLACEHOLDER_PREFIX = STX+"0ix2bavflj"
> INLINE_PLACEHOLDER_SUFFIX = ETX
> AMP_SUBSTITUTE = STX+"k75lziz62a"+ETX
>
> Actually, come to think of it, perhaps even that %d is not a good idea.
>   

Maybe we should use some random hashes, like 
`md5.new(str(random.random())).hexdigest()` ?
I don't think that users will be handle with placeholders, in case if 
everything works fine. In preporcessors they'll be given just plain 
input, in postprocessors they'll receive already processed with inline 
patterns ElementTree.