Author: milde Date: 2011-08-27 00:23:29 +0200 (Sat, 27 Aug 2011) New Revision: 7105 Added: trunk/sandbox/code-block-directive/rst2html-highlight.py trunk/sandbox/code-block-directive/rst2latex-highlight.py Removed: trunk/sandbox/code-block-directive/docs/pygments_with_docutils-latex-problems.txt trunk/sandbox/code-block-directive/rst2html-highlight trunk/sandbox/code-block-directive/rst2html-highlight.txt trunk/sandbox/code-block-directive/rst2latex-highlight Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.htm trunk/sandbox/code-block-directive/docs/myfunction.py.pdf trunk/sandbox/code-block-directive/docs/myfunction.py.pseudoxml trunk/sandbox/code-block-directive/docs/myfunction.py.tex trunk/sandbox/code-block-directive/docs/myfunction.py.txt trunk/sandbox/code-block-directive/docs/myfunction.py.xml trunk/sandbox/code-block-directive/docs/syntax-highlight.txt trunk/sandbox/code-block-directive/pygments_code_block_directive.py Log: Prepare code directive for inclusion in the core. Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.htm =================================================================== --- trunk/sandbox/code-block-directive/docs/myfunction.py.htm 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/myfunction.py.htm 2011-08-26 22:23:29 UTC (rev 7105) @@ -3,7 +3,7 @@ <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> -<meta name="generator" content="Docutils 0.8: http://docutils.sourceforge.net/" /> +<meta name="generator" content="Docutils 0.9: http://docutils.sourceforge.net/" /> <title></title> <link rel="stylesheet" href="../data/pygments-default.css" type="text/css" /> </head> @@ -12,20 +12,36 @@ <p>This is a test of the new "code" directive:</p> -<!-- Translate this document to HTML with a pygments enhanced frontend, e.g. +<!-- Translate this document with a pygments enhanced frontend, e.g. -../rst2html-highlight - -stylesheet=../data/pygments-default.css + ../rst2html-highlight.py - -stylesheet=../data/pygments-default.css + ../rst2latex-highlight.py - -stylesheet=../data/pygments-docutilsroles.sty -../rst2latex-highlight - -stylesheet=../data/pygments-docutileroles.sty --> -<pre class="code python silly literal-block"> -<span class="ln">1 </span><span class="k">def</span> <span class="nf">my_function</span><span class="p">():</span> -<span class="ln">2 </span> <span class="sd">"""Test the lexer. -</span><span class="ln">3 </span><span class="sd"> -</span><span class="ln">4 </span><span class="sd"> just a test"""</span> -<span class="ln">5 </span> -<span class="ln">6 </span> <span class="c"># and now for something completely different</span> -<span class="ln">7 </span> <span class="k">print</span> <span class="mi">8</span><span class="o">/</span><span class="mi">2</span> +or via the test case in + + ../pygments_code_block_directive.py - -traceback --> +<p>The example from Docutils TODO list:</p> +<pre class="code python literal-block"> +<span class="k">print</span> <span class="s">'This is Python code.'</span> +<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> + <span class="k">print</span> <span class="n">i</span> </pre> +<p>Numbered lines:</p> +<pre class="code python literal-block"> +<span class="ln">1 </span><span class="c"># This is Python code,</span> +<span class="ln">2 </span><span class="c"># that prints the integers from 0 to 9</span> +<span class="ln">3 </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> +<span class="ln">4 </span> <span class="k">print</span> <span class="n">i</span> +</pre> +<p>Another example:</p> +<pre class="code python silly literal-block" id="my-function"> +<span class="ln"> 7 </span><span class="k">def</span> <span class="nf">my_function</span><span class="p">():</span> +<span class="ln"> 8 </span> <span class="sd">"""Test the lexer. +</span><span class="ln"> 9 </span><span class="sd"> """</span> +<span class="ln">10 </span> +<span class="ln">11 </span> <span class="c"># and now for something completely different</span> +<span class="ln">12 </span> <span class="k">print</span> <span class="mi">8</span><span class="o">/</span><span class="mi">2</span> +</pre> <p>The end.</p> </div> </body> Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.pdf =================================================================== (Binary files differ) Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.pseudoxml =================================================================== --- trunk/sandbox/code-block-directive/docs/myfunction.py.pseudoxml 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/myfunction.py.pseudoxml 2011-08-26 22:23:29 UTC (rev 7105) @@ -1,15 +1,25 @@ -<document source="/home/milde/.python/PyLit/trunk/rstdocs/features/myfunction.py.txt"> - <decoration> - <footer> - <paragraph> - Generated on: 2007-06-05. +<document source="/home/milde/Code/Python/docutils-svn/sandbox/code-block-directive/docs/myfunction.py.txt"> <paragraph> - This is a test of the new code-block directive: + This is a test of the new "code" directive: <comment xml:space="preserve"> Translate this document to HTML with a pygments enhanced frontend, e.g. - rst2html-highlight --stylesheet=pygments-default.css --link-stylesheet - <literal_block classes="code-block python" raw_content="[u'def my_function():', u' "just a test"', u' print 8/2']" xml:space="preserve"> + ../rst2html-highlight.py --stylesheet=../data/pygments-default.css + + ../rst2latex-highlight.py --stylesheet=../data/pygments-docutilsroles.sty + + ../rst2pseudoxml-highlight.py + <paragraph> + The example from Docutils TODO list: + <literal_block classes="code pythonsi" xml:space="preserve"> + print 'This is Python code.' + for i in range(10): + print i + <paragraph> + Another example: + <literal_block classes="code python silly" ids="my-function" names="my-function" xml:space="preserve"> + <inline classes="ln"> + 7 <inline classes="k"> def @@ -18,11 +28,32 @@ <inline classes="p"> (): + <inline classes="ln"> + 8 - <inline classes="s"> - "just a test" + <inline classes="sd"> + """Test the lexer. + <inline classes="ln"> + 9 + <inline classes="sd"> + + <inline classes="ln"> + 10 + <inline classes="sd"> + just a test""" + <inline classes="ln"> + 11 + + <inline classes="ln"> + 12 + <inline classes="c"> + # and now for something completely different + + <inline classes="ln"> + 13 + <inline classes="k"> print @@ -32,4 +63,5 @@ / <inline classes="mi"> 2 - + <paragraph> + The end. Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.tex =================================================================== --- trunk/sandbox/code-block-directive/docs/myfunction.py.tex 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/myfunction.py.tex 2011-08-26 22:23:29 UTC (rev 7105) @@ -44,20 +44,43 @@ This is a test of the new ``code'' directive: -% Translate this document to HTML with a pygments enhanced frontend, e.g. +% Translate this document with a pygments enhanced frontend, e.g. % -% ../rst2html-highlight --stylesheet=../data/pygments-default.css +% ../rst2html-highlight.py --stylesheet=../data/pygments-default.css +% ../rst2latex-highlight.py --stylesheet=../data/pygments-docutilsroles.sty % -% ../rst2latex-highlight --stylesheet=../data/pygments-docutilsroles.sty +% or via the test case in +% +% ../pygments_code_block_directive.py --traceback + +The example from Docutils TODO list: % \begin{quote}{\ttfamily \raggedright \noindent +\DUrole{k}{print}~\DUrole{s}{'This~is~Python~code.'}~\\ +\DUrole{k}{for}~\DUrole{n}{i}~\DUrole{ow}{in}~\DUrole{nb}{range}\DUrole{p}{(}\DUrole{mi}{10}\DUrole{p}{):}~\\ +~~~~\DUrole{k}{print}~\DUrole{n}{i} +} +\end{quote} + +Numbered lines: +% +\begin{quote}{\ttfamily \raggedright \noindent +\DUrole{ln}{1~}\DUrole{c}{\#~This~is~Python~code,}~\\ +\DUrole{ln}{2~}\DUrole{c}{\#~that~prints~the~integers~from~0~to~9}~\\ +\DUrole{ln}{3~}\DUrole{k}{for}~\DUrole{n}{i}~\DUrole{ow}{in}~\DUrole{nb}{range}\DUrole{p}{(}\DUrole{mi}{10}\DUrole{p}{):}~\\ +\DUrole{ln}{4~}~~~~\DUrole{k}{print}~\DUrole{n}{i} +} +\end{quote} + +Another example: +% +\begin{quote}{\ttfamily \raggedright \noindent \DUrole{ln}{~7~}\DUrole{k}{def}~\DUrole{nf}{my\_function}\DUrole{p}{():}~\\ \DUrole{ln}{~8~}~~~~\DUrole{sd}{"{}"{}"Test~the~lexer.\\ -}\DUrole{ln}{~9~}\DUrole{sd}{~\\ -}\DUrole{ln}{10~}\DUrole{sd}{~~~~just~a~test"{}"{}"}~\\ -\DUrole{ln}{11~}~\\ -\DUrole{ln}{12~}~~~~\DUrole{c}{\#~and~now~for~something~completely~different}~\\ -\DUrole{ln}{13~}~~~~\DUrole{k}{print}~\DUrole{mi}{8}\DUrole{o}{/}\DUrole{mi}{2} +}\DUrole{ln}{~9~}\DUrole{sd}{~~~~"{}"{}"}~\\ +\DUrole{ln}{10~}~\\ +\DUrole{ln}{11~}~~~~\DUrole{c}{\#~and~now~for~something~completely~different}~\\ +\DUrole{ln}{12~}~~~~\DUrole{k}{print}~\DUrole{mi}{8}\DUrole{o}{/}\DUrole{mi}{2} } \end{quote} Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.txt =================================================================== --- trunk/sandbox/code-block-directive/docs/myfunction.py.txt 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/myfunction.py.txt 2011-08-26 22:23:29 UTC (rev 7105) @@ -1,20 +1,44 @@ This is a test of the new "code" directive: -.. Translate this document to HTML with a pygments enhanced frontend, e.g. +.. Translate this document with a pygments enhanced frontend, e.g. - ../rst2html-highlight --stylesheet=../data/pygments-default.css + ../rst2html-highlight.py --stylesheet=../data/pygments-default.css + ../rst2latex-highlight.py --stylesheet=../data/pygments-docutilsroles.sty + + or via the test case in + + ../pygments_code_block_directive.py --traceback + - ../rst2latex-highlight --stylesheet=../data/pygments-docutilsroles.sty +The example from Docutils TODO list: .. code:: python + + print 'This is Python code.' + for i in range(10): + print i + +Numbered lines: + +.. code:: python + :number-lines: + + # This is Python code, + # that prints the integers from 0 to 9 + for i in range(10): + print i + +Another example: + +.. code:: python :class: silly + :name: my_function :number-lines: 7 def my_function(): """Test the lexer. + """ - just a test""" - # and now for something completely different print 8/2 Modified: trunk/sandbox/code-block-directive/docs/myfunction.py.xml =================================================================== --- trunk/sandbox/code-block-directive/docs/myfunction.py.xml 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/myfunction.py.xml 2011-08-26 22:23:29 UTC (rev 7105) @@ -1,10 +1,18 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="utf-8"?> <!DOCTYPE document PUBLIC "+//IDN docutils.sourceforge.net//DTD Docutils Generic//EN//XML" "http://docutils.sourceforge.net/docs/ref/docutils.dtd"> -<!-- Generated by Docutils 0.4.1 --> -<document source="/home/milde/.python/PyLit/trunk/rstdocs/features/myfunction.py.txt"><decoration><footer><paragraph>Generated on: 2007-06-05. -</paragraph></footer></decoration><paragraph>This is a test of the new code-block directive:</paragraph><comment xml:space="preserve">Translate this document to HTML with a pygments enhanced frontend, e.g. +<!-- Generated by Docutils 0.9 --> +<document source="/home/milde/Code/Python/docutils-svn/sandbox/code-block-directive/docs/myfunction.py.txt"><paragraph>This is a test of the new "code" directive:</paragraph><comment xml:space="preserve">Translate this document to HTML with a pygments enhanced frontend, e.g. -rst2html-highlight --stylesheet=pygments-default.css --link-stylesheet</comment><literal_block classes="code-block python" raw_content="[u'def my_function():', u' "just a test"', u' print 8/2']" xml:space="preserve"><inline classes="k">def</inline> <inline classes="nf">my_function</inline><inline classes="p">():</inline> - <inline classes="s">"just a test"</inline> - <inline classes="k">print</inline> <inline classes="mi">8</inline><inline classes="o">/</inline><inline classes="mi">2</inline> -</literal_block></document> +../rst2html-highlight.py --stylesheet=../data/pygments-default.css + +../rst2latex-highlight.py --stylesheet=../data/pygments-docutilsroles.sty + +../rst2pseudoxml-highlight.py</comment><paragraph>The example from Docutils TODO list:</paragraph><literal_block classes="code python" xml:space="preserve"><inline classes="k">print</inline> <inline classes="s">'This is Python code.'</inline> + <inline classes="k">for</inline> <inline classes="n">i</inline> <inline classes="ow">in</inline> <inline classes="nb">range</inline><inline classes="p">(</inline><inline classes="mi">10</inline><inline classes="p">):</inline> + <inline classes="k">print</inline> <inline classes="n">i</inline></literal_block><paragraph>Another example:</paragraph><literal_block classes="code python ['silly']" ids="my-function" names="my-function" xml:space="preserve"><inline classes="ln"> 7 </inline><inline classes="k">def</inline> <inline classes="nf">my_function</inline><inline classes="p">():</inline> +<inline classes="ln"> 8 </inline> <inline classes="sd">"""Test the lexer. +</inline><inline classes="ln"> 9 </inline><inline classes="sd"> +</inline><inline classes="ln">10 </inline><inline classes="sd"> just a test"""</inline> +<inline classes="ln">11 </inline> +<inline classes="ln">12 </inline> <inline classes="c"># and now for something completely different</inline> +<inline classes="ln">13 </inline> <inline classes="k">print</inline> <inline classes="mi">8</inline><inline classes="o">/</inline><inline classes="mi">2</inline></literal_block><paragraph>The end.</paragraph></document> Modified: trunk/sandbox/code-block-directive/docs/syntax-highlight.txt =================================================================== --- trunk/sandbox/code-block-directive/docs/syntax-highlight.txt 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/docs/syntax-highlight.txt 2011-08-26 22:23:29 UTC (rev 7105) @@ -43,13 +43,6 @@ There are already docutils extensions providing syntax colouring, e.g: -SilverCity_, - a C++ library and Python extension that can provide lexical - analysis for over 20 different programming languages. A recipe__ for a - "code-block" directive provides syntax highlight by SilverCity. - -__ http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252170 - `listings`_, Since Docutils 0.5, the "latex2e" writer supports syntax highlight of literal blocks via the `listings` package with the @@ -57,19 +50,34 @@ style sheet. The stylesheets_ repository provides two LaTeX style sheets for highlighting literal-blocks with "listings". -Trac_ - has `reStructuredText support`__ and offers syntax highlighting with - a "code-block" directive using GNU Enscript_, SilverCity_, or Pygments_. +Odtwriter_, experimental writer for Docutils OpenOffice export supports syntax + colours using Pygments_. See also the (outdated) section `Odtwriter syntax`_. -__ http://trac.edgewall.org/wiki/WikiRestructuredText +Pygments_ + is a generic syntax highlighter written completely in Python. + * Usable as a command-line tool and as a Python package. + * Supports about 200 `languages and markup formats`_ (version 1.4). + * Already used by the odtwriter_ and Sphinx. + * Support for new languages, formats, and styles is added easily (modular + structure, Python code, existing documentation). + * Well documented and actively maintained. + * The web site provides a recipe for `using Pygments in ReST documents`_ + (used in the legacy `Pygments enhanced docutils front-ends`_). + rest2web_, the "site builder" provides the `colorize`__ macro (using the `Moin-Moin Python colorizer`_) __ http://www.voidspace.org.uk/python/rest2web/macros.html#colorize +SilverCity_, + a C++ library and Python extension that can provide lexical + analysis for over 20 different programming languages. A recipe__ for a + "code-block" directive provides syntax highlight by SilverCity. +__ http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252170 + Sphinx_ features automatic highlighting using the Pygments_ highlighter. It introduces the custom directives @@ -80,23 +88,12 @@ (see http://sphinx.pocoo.org/markup/code.html). -Pygments_ - is a generic syntax highlighter written completely in Python. +Trac_ + has `reStructuredText support`__ and offers syntax highlighting with + a "code-block" directive using GNU Enscript_, SilverCity_, or Pygments_. - * Usable as a command-line tool and as a Python package. - * A wide range of common `languages and markup formats`_ is supported. - * Additionally, OpenOffice's ``*.odt`` is supported by the odtwriter_. - * The layout is configurable by style sheets. - * Several built-in styles and an option for line-numbering. - * Built-in output formats include HTML, LaTeX, rtf - * Support for new languages, formats, and styles is added easily (modular - structure, Python code, existing documentation). - * Well documented and actively maintained. - * The web site provides a recipe for `using Pygments in ReST documents`_. - It is used in the `Pygments enhanced docutils front-ends`_ below. +__ http://trac.edgewall.org/wiki/WikiRestructuredText -Odtwriter_, experimental writer for Docutils OpenOffice export supports syntax - colours using Pygments_. (See the (outdated) section `Odtwriter syntax`_.) Summary """"""" @@ -153,7 +150,7 @@ :Directive Type: "code" :Doctree Element: literal_block :Directive Arguments: One (`language`), optional. -:Directive Options: name, class. +:Directive Options: name, class, number-lines. :Directive Content: Becomes the body of the literal block. The "code-block" directive constructs a literal block where the content is @@ -162,8 +159,19 @@ issued and the content is rendered as ordinary literal block with additional class arguments: "code" and the value of `language`. + :number-lines: let pygments include line-numbers + + +The following options are recognized: + +``number-lines`` : [start line number] + Precede every code line with a line number. + The optional argument is the number of the first line (defaut 1). + +and the common options `:class:`_ and `:name:`_. + Example:: - The Python code snippet :: + The content of the following directive :: .. code:: python @@ -171,11 +179,13 @@ "just a test" print 8/2 - will be parsed and marked up as Python source code. The actual rendering + is parsed and marked up as Python source code. The actual rendering depends on the style-sheet. -Remarks: +Remarks +""""""" + * Without language argument, the parsing step is skipped. Use cases: * Mark a literal block as pseudo-code. @@ -186,16 +196,15 @@ the listings_ package). The language's name can be given as `class` option. - - Alternative: - make the `language` argument compulsory and add a "no-parsing" option. -* Additional arguments might be defined and passed to the pygments_ - parser. + Alternative: + make the `language` argument compulsory and add a "no-highlight" option. - :number-lines: let pygments include line-numbers +* TODO: Pygments_ provides filters like VisibleWhitespaceFilter + add options to use them? + Include directive option """""""""""""""""""""""" Modified: trunk/sandbox/code-block-directive/pygments_code_block_directive.py =================================================================== --- trunk/sandbox/code-block-directive/pygments_code_block_directive.py 2011-08-25 13:10:04 UTC (rev 7104) +++ trunk/sandbox/code-block-directive/pygments_code_block_directive.py 2011-08-26 22:23:29 UTC (rev 7105) @@ -37,6 +37,7 @@ from docutils import nodes from docutils.parsers.rst import directives, Directive +from docutils.parsers.rst.roles import set_classes try: import pygments from pygments.lexers import get_lexer_by_name @@ -45,54 +46,62 @@ except ImportError: with_pygments = False - # Customisation # ------------- # # Do not insert inline nodes for the following tokens. # (You could add e.g. Token.Punctuation like ``['', 'p']``.) :: -unstyled_tokens = [''] +unstyled_tokens = [''] # Token.Text -# Tokenizer -# ----------------- +# Lexer +# --------- # # This interface class combines code from # pygments.formatters.html and pygments.formatters.others. -class Tokenizer(object): - """Parse `code` string and yield "classified" tokens. +class Lexer(object): + """Parse `code` lines and yield "classified" tokens. Arguments - code -- string of source code to parse + code -- list of source code lines to parse language -- formal language the code is written in. Merge subsequent tokens of the same token-type. - Yields the tokens as ``(ttype_class, value)`` tuples, - where ttype_class is taken from pygments.token.STANDARD_TYPES and - corresponds to the class argument used in pygments html output. + Iterating over an instance yields the tokens as ``(ttype_class, value)`` + tuples, where `ttype_class` is taken from pygments.token.STANDARD_TYPES + and corresponds to the class argument used in pygments html output. """ def __init__(self, code, language): + """ + Set up a lexical analyzer for `code` in `language`. + """ self.code = code self.language = language - - def lex(self): - # Get lexer for language (use text as fallback) + self.lexer = None + # get lexical analyzer for `language`: + if language in ('', 'text'): + return + if not with_pygments: + raise ApplicationError('Cannot highlight code. ' + 'Pygments package not found.') try: - lexer = get_lexer_by_name(self.language) - except ValueError: - # info: 'no pygments lexer for %s, using "text"' % self.language - lexer = get_lexer_by_name('text') - return pygments.lex(self.code, lexer) + self.lexer = get_lexer_by_name(self.language) + except pygments.util.ClassNotFound: + raise ApplicationError('Cannot highlight code. ' + 'No Pygments lexer found for "%s".' % language) + # Since version 1.2. (released Jan 01, 2010) Pygments has a + # TokenMergeFilter. ``self.merge(tokens)`` in __iter__ can be + # replaced by ``self.lexer.add_filter('tokenmerge')`` in __init__. - def join(self, tokens): - """Join subsequent tokens of same token-type. + def merge(self, tokens): + """Merge subsequent tokens of same token-type. - Also, leave out the final '\n' (added by pygments). + Also strip the final '\n' (added by pygments). """ tokens = iter(tokens) (lasttype, lastval) = tokens.next() @@ -106,34 +115,46 @@ yield(lasttype, lastval) def __iter__(self): - """parse code string and yield "classified" tokens + """Parse self.code and yield "classified" tokens """ - tokens = self.lex() - for ttype, value in self.join(tokens): - # yield (ttype, value) - yield (_get_ttype_class(ttype), value) + codestring = u'\n'.join(self.code) + if self.lexer is None: + yield [('', codestring)] + return + tokens = pygments.lex(codestring, self.lexer) + for ttype, value in self.merge(tokens): + # yield (ttype, value) # token type objects + yield (_get_ttype_class(ttype), value) # short name strings class NumberLines(object): - """Insert linenumber-tokens in front of every newline + """Insert linenumber-tokens in front of every newline. + + Arguments + + tokens -- iterable of ``(ttype_class, value)`` tuples + startline -- first line number + endline -- last line number - Nontrivial, as we need to weave these into the possibly - multi-line tokens from pygments. - """ + Iterating over an instance yields the tokens preceded by + a ``('ln', '<line number>')`` token for every line. + Multi-line tokens from pygments are splitted. """ - def __init__(self, tokens, startline, fmt_str): + def __init__(self, tokens, startline, endline): self.tokens = tokens - self.lineno = startline - self.fmt_str = fmt_str + self.startline = startline + # pad linenumbers, e.g. endline == 100 -> fmt_str = '%3d ' + self.fmt_str = '%%%dd ' % len(str(endline)) def __iter__(self): - yield ('ln', self.fmt_str % self.lineno) + lineno = self.startline + yield ('ln', self.fmt_str % lineno) for ttype, value in self.tokens: lines = value.split('\n') for line in lines[:-1]: yield (ttype, line + '\n') - self.lineno += 1 - yield ('ln', self.fmt_str % self.lineno) + lineno += 1 + yield ('ln', self.fmt_str % lineno) yield (ttype, lines[-1]) @@ -144,51 +165,54 @@ class CodeBlock(Directive): """Parse and mark up content of a code block. """ - required_arguments = 1 + optional_arguments = 1 option_spec = {'class': directives.class_option, - 'number-lines': directives.unchanged + 'name': directives.unchanged, + 'number-lines': directives.unchanged # integer or None } has_content = True def run(self): - language = self.arguments[0] - # Process number-lines with optional argument `startline` - startline = self.options.get('number-lines', '1') - try: - startline = int(startline or 1) # default to 1 for empty str - except ValueError: - raise self.error( - ':number-lines: option with non-integer start value') self.assert_has_content() + if self.arguments: + language = self.arguments[0] + else: + language = '' + set_classes(self.options) + classes = ['code', language] + if 'classes' in self.options: + classes.extend(self.options['classes']) + + # TODO: config setting to skip lexical analysis: + ## if document.settings.no_highlight: + ## language = '' - # create a literal block element and set class argument - code_block = nodes.literal_block(classes=['code', language] - + self.options['class']) + # set up lexical analyzer + tokens = Lexer(self.content, language) - # iterator returning code tokens - if with_pygments: - tokens = Tokenizer(u'\n'.join(self.content), language) - else: - # TODO: warning or info? - self.warning('Cannot highlight code, Pygments lexer not found.') - tokens = [('', u'\n'.join(self.content))] - if 'number-lines' in self.options: - # pad linenumbers, e.g. endline == 100 -> fmt_str = '%3d ' + # optional argument `startline`, defaults to 1 + try: + startline = int(self.options['number-lines'] or 1) + except ValueError: + raise self.error(':number-lines: with non-integer start value') endline = startline + len(self.content) - fmt_str = "%%%dd " % len(str(endline)) - # print startline, '...', endline, repr(fmt_str) - tokens = NumberLines(tokens, startline, fmt_str) + # add linenumber filter: + tokens = NumberLines(tokens, startline, endline) - # parse content with pygments and add to code_block element + node = nodes.literal_block('\n'.join(self.content), classes=classes) + self.add_name(node) + + # analyze content and add nodes for every token for cls, value in tokens: + # print (cls, value) if cls in unstyled_tokens: # insert as Text to decrease the verbosity of the output. - code_block += nodes.Text(value, value) + node += nodes.Text(value, value) else: - code_block += nodes.inline(value, value, classes=[cls]) + node += nodes.inline(value, value, classes=[cls]) - return [code_block] + return [node] # Register Directive @@ -220,5 +244,5 @@ # Uncomment the desired output format: # publish_cmdline(writer_name='pseudoxml', description=description) # publish_cmdline(writer_name='xml', description=description) - publish_cmdline(writer_name='html', description=description) - # publish_cmdline(writer_name='latex', description=description) + # publish_cmdline(writer_name='html', description=description) + publish_cmdline(writer_name='latex', description=description) Copied: trunk/sandbox/code-block-directive/rst2html-highlight.py (from rev 6488, trunk/sandbox/code-block-directive/rst2html-highlight) =================================================================== --- trunk/sandbox/code-block-directive/rst2html-highlight.py (rev 0) +++ trunk/sandbox/code-block-directive/rst2html-highlight.py 2011-08-26 22:23:29 UTC (rev 7105) @@ -0,0 +1,53 @@ +#!/usr/bin/python +# coding: utf-8 + +# rst2html-highlight +# ================== +# +# Docutils front-end with syntax highlight. +# +# :Author: David Goodger, Georg Brandl, Günter Milde +# :Date: $Date: 2008-05-22 08:42:52 +0200 (Do, 22. Mai 2008) $ +# :Copyright: This module has been placed in the public domain. +# +# This is a merge of the docutils_ `rst2html` front end with an extension +# suggestion by Felix Wiemann. +# +# :: + +""" +A front end to docutils, producing HTML with syntax colouring using pygments + +Generates (X)HTML documents from standalone reStructuredText sources. Uses +`pygments` to parse and mark up the content of ``.. code::` directives. +Needs an adapted stylesheet +""" + +# Requirements +# ------------ +# +# :: + +try: + import locale + locale.setlocale(locale.LC_ALL, '') +except: + pass + +from docutils.core import publish_cmdline, default_description + +# The `pygments_code_block_directive`_ module defines and registers a new +# directive `code` that uses the `pygments`_ source highlighter to +# render code in color:: + +import pygments_code_block_directive + +# Call the docutils publisher to render the input as html:: + +description = __doc__ + default_description +publish_cmdline(writer_name='html', description=description) + +# .. _docutils: http://docutils.sf.net/ +# .. _pygments_code_block_directive: pygments_code_block_directive.py +# .. _pygments: http://pygments.org/ +# .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/ Copied: trunk/sandbox/code-block-directive/rst2latex-highlight.py (from rev 6424, trunk/sandbox/code-block-directive/rst2latex-highlight) =================================================================== --- trunk/sandbox/code-block-directive/rst2latex-highlight.py (rev 0) +++ trunk/sandbox/code-block-directive/rst2latex-highlight.py 2011-08-26 22:23:29 UTC (rev 7105) @@ -0,0 +1,53 @@ +#!/usr/bin/python + +# rst2latex-highlight +# =================== +# +# Docutils front-end with syntax highlight. +# +# :Author: David Goodger, a Pygments author|contributor, Guenter Milde +# :Date: $Date: 2008-05-22 08:42:52 +0200 (Do, 22. Mai 2008) $ +# :Copyright: This module has been placed in the public domain. +# +# This is a merge of the docutils_ `rst2latex` front end with an extension +# suggestion taken from the Pygments_ documentation. +# +# :: + +""" +A front end to docutils, producing LaTeX with syntax colouring using pygments + +Generates LaTeX documents from standalone reStructuredText sources. Uses the +`Pygments` syntax highlighter to parse and mark up the content of ``.. +code::` directives. Needs an adapted stylesheet. +""" + +# Requirements +# ------------ +# +# :: + +try: + import locale + locale.setlocale(locale.LC_ALL, '') +except: + pass + +from docutils.core import publish_cmdline, default_description + +# `<pygments_code_block_directive.py>`_ defines and registers a new +# directive `code` that uses the `Pygments`_ syntax highlighter to +# render code in color:: + +import pygments_code_block_directive + +# Call the docutils publisher to render the input as latex:: + +description = __doc__ + default_description +publish_cmdline(writer_name='latex2e', description=description) + + +# .. References: +# .. _docutils: http://docutils.sf.net/ +# .. _pygments: http://pygments.org/ +# .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/ |