[Htmltmpl] Performance on long loops

[Daniel L. <dan...@gm...>]

Hi,

I'm using htmltmpl in a medium-sized web site and I'm delighted with its ease 
of use and flexibility. This is a trivial patch against htmltmpl-1.22 that 
greatly helps htmltmpl's performance on pages with very long and/or nested 
loops (e.g. a table containing hundreds of rows). Those pages tend to spend a 
lot of time in htmltmpl's function "process" where the individual tokens are 
analyzed and concatenated. It's the string concatenation that detracts from 
htmltmpl's performance: it uses a single output string (out) and simply 
appends to it every time a new token is processed. This is fine for smaller 
pages, but doing string concatenation 1000s of times is inefficient because 
Python strings are immutable, so each "out += something" actually creates a 
new string, thus killing performance. A more efficient possibility is to 
store all tokens in a list, then join the list using string.join. This way 
the string is created only once. I saw performance go up by 50-70% on pages 
spending most time in TemplateProcessor.process with no problems until now 
(the test suite is passed, too).

Here's the diff:

diff -u htmltmpl-1.22/htmltmpl.py htmltmpl/htmltmpl.py
--- htmltmpl-1.22/htmltmpl.py   2001-12-15 23:11:28.000000000 +0100
+++ htmltmpl/htmltmpl.py        2004-05-05 12:57:25.000000000 +0200
@@ -574,7 +574,7 @@

         tokens = template.tokens()
         len_tokens = len(tokens)
-        out = ""              # buffer for processed output
+        out = []              # buffer for processed output

         # Recover position at which we ended after processing of last part.
         i = self._current_pos
@@ -605,7 +605,7 @@
                     if DISABLE_OUTPUT not in output_control:
                         value = str(self.find_value(var, loop_name, 
loop_pass,
                                                     loop_total, globalp))
-                        out += self.escape(value, escape)
+                        out += [self.escape(value, escape)]
                         self.DEB("VAR: " + str(var))

                 elif token == "<TMPL_LOOP":
@@ -731,21 +731,21 @@
                     # when it was not replaced by the parser.
                     skip_params = 1
                     filename = tokens[i + PARAM_NAME]
-                    out += """
+                    out += ["""
                         <br />
                         <p>
                         <strong>HTMLTMPL WARNING:</strong><br />
                         Cannot include template: <strong>%s</strong>
                         </p>
                         <br />
-                    """ % filename
+                    """ % filename]
                     self.DEB("CANNOT INCLUDE WARNING")

                 elif token == "<TMPL_GETTEXT":
                     skip_params = 1
                     if DISABLE_OUTPUT not in output_control:
                         text = tokens[i + PARAM_GETTEXT_STRING]
-                        out += gettext.gettext(text)
+                        out += [gettext.gettext(text)]
                         self.DEB("GETTEXT: " + text)

                 else:
@@ -756,7 +756,7 @@
                 # Raw textual template data.
                 # If output of current block is not disabled, then
                 # append template data to the output buffer.
-                out += token
+                out += [token]

             i += 1
             # end of the big while loop
@@ -764,7 +764,7 @@
         # Check whether all opening statements were closed.
         if loop_name: raise TemplateError, "Missing </TMPL_LOOP>."
         if output_control: raise TemplateError, "Missing </TMPL_IF> or 
</TMPL_UNLESS>"
-        return out
+        return "".join(out)


Take care,
Daniel