Re: [Python-markdown-discuss] Markdown encoding

[Yuri T. <qar...@gm...>]

> Hmm...one problem with this (and Waylan's suggestion of making the
> encoding parameter to markdown() do something useful) is that until
> 1.6b, markdown() did in fact work perfectly well with encoded text and
> it was not at all clear that this was not the intended usage. When 1.6b
> came out I just commented out the call to removeBOM(), complained to the
> list, and continued on my way.

Good point...  But I think that was a mistake, which needs to be
corrected.  In the very least, I don't want any new users to use it
that way.

So, the question is: what would be a good balance between fixing this
problem and not screwing existing users?

I suggest releasing 1.7 with all of Waylan's recent fixes and this
change and putting a clear message in release notes that in 1.7
markdown.markdown() expects unicode and that if you've got
utf8-encoded strings, then you should call it with
markdown.markdown(input.encode("utf8"))

> I use markdown from Django with the markdown support included with
> Django; presumably many other people are also. For example:
> http://www.freewisdom.org/projects/python-markdown/Django

I haven't touched Django for some time, so I am not sure what it does
with unicode today.  I remember that in December 2006 it was a mess.
At that time they did pass encoded bytestrings around.  At this point
they seem to give you an option of either using bytestrings or unicode
(http://www.djangoproject.com/documentation/unicode/)

I think the thing to do here is to write a new version of the plugin,
which would check if the input is unicode, and if not would decode it
from utf8 before sending it to markdown.  People who update the 1.7
will also need to update to the new plugin, which doesn't seem to be
so bad.  We should probably also send the new plugin to the Django
team and ask them to include it instead of the old one.  (BTW, does
Django actually include markdown or just the plugin?)

Perhaps the django plugin should be included with markdown release?
We should probably also put it in SVN.

- yuri

-- 
Yuri Takhteyev
http://www.freewisdom.org/