Python Markdown

I didn't know where else to send this, but I'm having trouble with the
wiki at http://www.freewisdom.org/projects/python-markdown/.

When I try to create an account, I get:

...-2007-08-15-install-markdown/share/lua/5.1/cosmo.lua:103: Cosmo: the iterator
for dofields failed: ...08-15-install-markdown/share/lua/5.1/wikistorage.lua:110:
bad argument #1 to 'pairs' (table expected, got nil)
stack traceback:
  [C]: in function 'error'
  ...-2007-08-15-install-markdown/share/lua/5.1/cosmo.lua:103: in function
<...-2007-08-15-install-markdown/share/lua/5.1/cosmo.lua:91>
  (tail call): ?
  [C]: in function 'gsub'
  ...-2007-08-15-install-markdown/share/lua/5.1/cosmo.lua:51: in function
<...-2007-08-15-install-markdown/share/lua/5.1/cosmo.lua:48>
  (tail call): ?
  (tail call): ?
  ...08-15-install-markdown/share/lua/5.1/wikistorage.lua:138: in function
<...08-15-install-markdown/share/lua/5.1/wikistorage.lua:131>
  (tail call): ?
  sputnik.lua:51: in function 'save'
  sputnik.lua:61: in function 'check_password'
  sputnik.lua:618: in function 'get_params'
  sputnik.lua:643: in function 'main'
  sputnik.lua:663: in main chunk
  (tail call): ?
  (tail call): ?
  (tail call): ?
  [C]: in function 'xpcall'
  ...2007-08-15-install-markdown/share/lua/5.1/cgilua.lua:426: in function
'_xpcall'
  ...2007-08-15-install-markdown/share/lua/5.1/cgilua.lua:533: in function
<...2007-08-15-install-markdown/share/lua/5.1/cgilua.lua:508>
  (tail call): ?



-- 
Brian Jaress
brian_jaress@...
http://brian-jaress.livejournal.com

For as long as I can remember the following comment has been in the
CorePatterns class:

    """This class is scheduled for removal as part of a refactoring effort."""

My question is; what would that refactor entail? Is there any help I
can provide?

The reason I ask is highlighted in ticket #1793419 [1]. I'm sure the
quick fix I provide there isn't anywhere near the goal of the
refactor. The way I see it. the problem is that while one could
override the patterns within CorePatterns, is is currently very
difficult to override the code that uses those patterns. It is even
more difficult to add your own patterns.

As the comments suggest just above the CorePatterns class, we should
try to use preprocessors, inlinepatterns or postprocessors if
possible. While understandable, when working with block level markup,
preprocessors don't output to the dom, inlinepatterns will get wrapped
in blocklevel elements defined in CorePatterns (block level elements
would get inappropriately wrapped in <p> tags, for example) and, by
the time we get to postprocessors, it's a little late.

The example I give in the ticket is adding header ids as php markdown
extra does [2]. I could use a preproccesor to extract the id, but now
I need to store it in some way that I know which header it goes with.
Then, how would I add that id to the header? A postprocessor would be
ugly here. The best would be to override the code that puts the header
into the dom. But that is buried in the core. If I'm doing that, I
might as well override the header corepattern.

I'm not posting this to complain. I'd just like to see that refactor
move forward if that's still the plan. If I can help, even better. The
problem is, I don't know what the plan is, so its a little difficult
to help. At the same time I realize this is complex stuff.

PS: As a side note, python-markdown eats php markdown extra's header
id syntax. The header is completely stripped from the output. At the
very least, I would have expected it to be wrapped in a <p> tag. At
best I would have expected the header without the id. Initially, I was
surprised no one implemented this extension yet. Now I know why. It's
currently too hard.

[1]: https://sourceforge.net/tracker/?func=detail&atid=790198&aid=1793419&group_id=153041
[2]: http://michelf.com/projects/php-markdown/extra/#header-id
-- 
----
Waylan Limberg
waylan@...

Hi,

Markdown 1.6b doesn't work with UTF-8-encoded text. It fails with a 
UnicodeDecodeError in removeBOM():

In [3]: import markdown
In [4]: text = u'\xe2'.encode('utf-8')
In [6]: print text
â
In [7]: print markdown.markdown(text)
------------------------------------------------------------
Traceback (most recent call last):
   File "<ipython console>", line 1, in <module>
   File 
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/markdown.py", 
line 1722, in markdown
     return md.convert(text)
   File 
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/markdown.py", 
line 1614, in convert
     self.source = removeBOM(self.source, self.encoding)
   File 
"/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/markdown.py", 
line 74, in removeBOM
     if text.startswith(bom):
<type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 
0xc3 in position 0: ordinal not in range(128)

The problem is that the BOM being tested is unicode so to execute
   text.startswith(bom)
Python tries to convert text to Unicode using the default encoding 
(ascii). This fails because the text is not ascii.

I'm trying to understand what the encoding parameter is for; it doesn't 
seem to do much. There also seems to be some confusion with the use of 
encoding in markdownFromFile() vs markdown(); the file is converted to 
Unicode on input so I don't understand why the same encoding parameter 
is passed to markdown()?

ISTM the encoding passed to markdown should match the encoding of the 
text passed to markdown, and the values in the BOMS table should be in 
the encoding of the key, not in unicode. Then the __unicode__() method 
should actually decode. Or is the intent that the text passed to 
markdown() should always be ascii or unicode?

I can put together a patch if you like but I wanted to make sure that I 
am not missing some grand plan...

Kent