[Plone-developers] LinguaPlone perfomance patch for large sites

[Hans-Peter L. <han...@in...>]

Hi,

We (ramonski and myself) have created a patch for LinguaPlone, using instance.memoize.
It should enhance performance on large sites.



--------------------------------------------patch----------------------------------------------------

--- Products/LinguaPlone/LanguageIndex.py	old
+++ Products/LinguaPlone/LanguageIndex.py	new
@@ -1,5 +1,6 @@
 from logging import getLogger
 
+from plone.memoize.instance import memoize
 from plone.indexer.interfaces import IIndexableObjectWrapper
 from zope.interface import implements
 
@@ -305,8 +306,15 @@
         for sublanguage in subs:
             result = oo_union(result, self._index[main][sublanguage])
 
-        return IISet(entry.docid for entry in result)
+        docids = tuple(entry.docid for entry in result)
+        return self.get_result_set(docids)
 
+    @memoize
+    def get_result_set(self, docids=()):
+        """ building an IISet is expensive for many docids
+        """
+        return IISet(docids)
+
 InitializeClass(LanguageIndex)
 
--------------------------------------------end patch---------------------------------------------------- 


The test with current LinguaPlone trunk do pass, with patch applied (-r35856)


Background
----------------

We've migrated a large Plone site ( 75000 Archetypes objects ) from Plone 2 to Plone 3.
The site runs on a Windows server using Enfold Server / Enfold Proxy
We've experienced a very bad logged in performance after the migration.

We've recorded a selenium test which logs in to the site,
clicks through varius navigation items
and logs out.

The first test run took 470 seconds...
The avarage of 10 testruns was 160 seconds.

Using ZopeProfiler we found out that LinguaPlone was the bottleneck.

The method _apply_index 

was called 28x for each click, and took 0,5 seconds each call.

We could not reduce the number of calls, as this would have meant to reduce the features of the site
(like disallowing the reviewlist portlet).

Finally we found a solution by refactoring the code and using instance.memoize.

The created patch lead to following performance (with the drawback of increased memory consumption) enhancement:

The first run took 94 seconds...
The average of 10 testruns was 28 seconds.

Plone: 3.3
LinguaPlone: 3.0.1

Cheerio! 
mr_savage