python-markdown-discuss

[Python-markdown-discuss] HTML Encoding Issues within Output of Custom Extension

[andrew t. <ath...@gm...>]

Hello,

I am kind of new to python, but I needed the ability to insert a
gallery into my django project. I have been working on creating a
python markdown extension that will insert an image gallery within my
django project when a custom tag is used. The actual extension is
working, but the HTML that the extension returns is all encoded. Here
is the extension that I am using:

    #!/usr/bin/env python
    from django.template.loader import render_to_string
    from main.models import *

    import markdown

    version = "0.1.0"

    class GalleriaExtension(markdown.Extension):
        def __init__(self, configs):
            self.config = {
            }

            # Override defaults with user settings
            for key, value in configs:
                self.setConfig(key, value)

        def add_inline(self, md, name, klass, re):
            pattern = klass(re)
            pattern.md = md
            pattern.ext = self
            md.inlinePatterns.add(name, pattern, "<reference")

        def extendMarkdown(self, md, md_globals):
            self.add_inline(md, 'galleria', Galleria,
                r'\[\[(G|g)allery (?P<superpage_id>\w+)\]\]')

    class Galleria(markdown.inlinepatterns.Pattern):
        def handleMatch(self, m):
            try:
                images =
SuperPage.objects.get(id=m.group('superpage_id')).superpageimage_set.all()
            except:
                images = None
            if images:
                rendered = render_to_string('galleria.html', {
'images': images })
            else:
                rendered = '<b>There are no images for the given ID</b>'
            return rendered

    def makeExtension(configs=None) :
        return GalleriaExtension(configs=configs)

I ensured that django's render_to_string was actually returning html
that was not encoded. From the shell here is an example of the output:

Output from render_to_string:

    >>> from django.template.loader import render_to_string
    >>> images = SuperPage.objects.get(id=8).superpageimage_set.all()
    >>> render_to_string('galleria.html', { 'images': images })
    u'<div class=\'galleria_std\'>\n    <div class=\'gallery\' >\n
 <div id=\'stage_gallery\' >\n'

Here is output from markdown extension that is encoded:

    >>>markdown.markdown('test [[gallery 8]] test', ['galleria'])
    u'<p>test &lt;div class=\'galleria_std\'&gt;\n    &lt;div
class=\'gallery\' &gt;\n'


How can I make rendered return HTML mark up instead of encoded markup
when using the markdown extension? Also, I would appreciate any
pointers on coding this differently (syntax, layout, etc). I
appreciate it.

Thanks

Re: [Python-markdown-discuss] HTML Encoding Issues within Output of Custom Extension

[Waylan L. <wa...@gm...>]

On Wed, Jan 18, 2012 at 11:43 AM, andrew thornton
<ath...@gm...> wrote:
> Hello,
>
> I am kind of new to python, but I needed the ability to insert a
> gallery into my django project. I have been working on creating a
> python markdown extension that will insert an image gallery within my
> django project when a custom tag is used. The actual extension is
> working, but the HTML that the extension returns is all encoded. Here
> is the extension that I am using:
>
[snip]
>    class Galleria(markdown.inlinepatterns.Pattern):
>        def handleMatch(self, m):
>            try:
>                images =
> SuperPage.objects.get(id=m.group('superpage_id')).superpageimage_set.all()
>            except:
>                images = None
>            if images:
>                rendered = render_to_string('galleria.html', {
> 'images': images })
>            else:
>                rendered = '<b>There are no images for the given ID</b>'
>            return rendered

Your problem is in the last few lines above. Inlinepatterns must
return ElementTree instances, not strings. Therefore, you need to
build a ET Element (or tree of elements) and return that. For example
consider this abbreviated rewrite of your code:

    if images:
        # build and return gallery here
    else:
        b = markdown.util.etree.Element('b')
        b.text = 'There are no images for the given ID'
        return b

Of course, this means no Django templates for your gallery without a
few workarounds.

A few approaches you might want to look at:

1) Follow the method used by the CodeHilite Extension and save the
html output in the htmlStash (marked as 'safe' so is doesn't get eat
by safe_mode) and return the placeholder. Then the placeholder will be
automatically replaced with the html after the Tree is searialized
into a string. See the code here [1].

2) Parse the html into a ET Tree and return it. But be careful here,
most parsers (including ElementTree's) fall flat on there face with
the simplest of html syntax errors. I would suggest lxml or html5lib,
both of which adds an extra dependency (lxml is the best IMO, but
requires a C lib).  See this excellent comparison [2] of the options.
Just make sure the one you use will build an ET Tree.

[1]: https://github.com/waylan/Python-Markdown/blob/master/markdown/extensions/codehilite.py#L184
[2]: http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/

-- 
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg

Re: [Python-markdown-discuss] HTML Encoding Issues within Output of Custom Extension

[Waylan L. <wa...@gm...>]

On Wed, Jan 18, 2012 at 2:28 PM, andrew thornton
<ath...@gm...> wrote:
> On Wed, Jan 18, 2012 at 2:03 PM, Waylan Limberg <wa...@gm...> wrote:
>> On Wed, Jan 18, 2012 at 1:12 PM, andrew thornton
>> <ath...@gm...> wrote:
>>>
>>> Thank you for your reply,
>>>
>>> I have been trying to use htmlStash within the Galleria class, but
>>> this isn't working. I have however been able to use ET to return the
>>> errors. Here is what I was trying:
>>>
>>> class Galleria(markdown.inlinepatterns.Pattern):
>>>    def handleMatch(self, m):
>>>        try:
>>>            images =
>>> SuperPage.objects.get(id=m.group('superpage_id')).superpageimage_set.all()
>>>        except:
>>>            images = None
>>>        if images:
>>>            placeholder =
>>> self.markdown.htmlStash.store(render_to_string('galleria.html', {
>>> 'images': images }))
>>>            block.clear()
>>>            block.tag = 'p'
>>>            block.text = placeholder
>>>        else:
>>>            b = markdown.util.etree.Element('b')
>>>            b.text = 'There are no images for the given ID'
>>>            return b
>>>
>>>
>>> Should I be using markdown.treeprocessors.Treeprocessor instead of the
>>> inlinepatterns? Can I use it in this way?
>>>
>>
>> You're not returning the placeholder. Try removing the 3 lines with
>> "block" in them and add `return placeholder` in their place. I'd also
>> suggest adding `safe=True` to the call to `htmlStash.store`. Like
>> this:
>>
>>        if images:
>>            placeholder = self.markdown.htmlStash.store(
>>                   render_to_string('galleria.html', {'images': images }),
>>                   safe=True
>>           )
>>            return placeholder
>>        else:
>>            b = markdown.util.etree.Element('b')
>>            b.text = 'There are no images for the given ID'
>>            return b
>>
>> Perhaps I should have pointed out that the CodeHilite extension uses a
>> Blockprocessor, thus the access to the parent "block". That won't work
>> in an inlinepattern (which has no access to the parent). However, if
>> your custom markup is the only thing in a paragraph (on a line by
>> itself), just return the placeholder and you should be done (see below
>> for why). If that doesn't work for you then either a Blockprocessor or
>> Treeprocessor would be the way to go - although it will take more work
>> (code) on your part to detect your custom markup.
>>
>> As a side note, when markdown swaps the htmlStash back in, if a
>> paragraph contains only a placeholder (with nothing else, not even
>> white space, i.e., `<p>some_placeholder</p>`) _and_ the stashed html
>> is block level, then the entire paragraph (`<p>` tag and all) gets
>> replaced with the stashed html. However, in all other situations, only
>> the placeholder itself gets replaced with the stashed html. If your
>> gallery is wrapped in div tags, you probably want the first behavior.
>> Perhaps the relevant code [1] will help you understand what's gong on.
>>
>> [1]: https://github.com/waylan/Python-Markdown/blob/master/markdown/postprocessors.py#L60
>>
>> --
>> ----
>> \X/ /-\ `/ |_ /-\ |\|
>> Waylan Limberg
>
> This worked great. Thank you for your help.

Your welcome.

-- 
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg