From: Ng, V. <Victor.Ng@CIBC.ca> - 2008-12-08 16:55:33
|
Hi all, I'm trying to run the Django testsuite against my new SQL Server backend, but I'm getting OOM errors which I'm pretty sure are unrelated to my database code. I'm running tests like this: jython --verify c:\dev\django-trunk\tests\runtests.py --settings=settings -v 2 and my stacktrace is below. Qualitatively - the django testsuite runs *really* slow. Has anyone seen anything like this? It looks like some kind of difference in the GC behaviour between Java and CPython. vic Running post-sync handlers for application admin_views Adding content type 'admin_views | section' Traceback (most recent call last): File "c:\dev\django-trunk\tests\runtests.py", line 183, in <module> django_tests(int(options.verbosity), options.interactive, args) File "c:\dev\django-trunk\tests\runtests.py", line 153, in django_tests failures = run_tests(test_labels, verbosity=verbosity, interactive=interactive, extra_tests=extra_tests) File "C:\jython2.5b0\Lib\site-packages\django\test\simple.py", line 142, in run_tests connection.creation.create_test_db(verbosity, autoclobber=not interactive) File "C:\jython2.5b0\Lib\site-packages\django\db\backends\creation.py", line315, in create_test_db call_command('syncdb', verbosity=verbosity, interactive=False) File "C:\jython2.5b0\Lib\site-packages\django\core\management\__init__.py", line 158, in call_command return klass.execute(*args, **options) File "C:\jython2.5b0\Lib\site-packages\django\core\management\base.py", line 222, in execute output = self.handle(*args, **options) File "C:\jython2.5b0\Lib\site-packages\django\core\management\base.py", line 351, in handle return self.handle_noargs(**options) File "C:\jython2.5b0\Lib\site-packages\django\core\management\commands\syncdb .py", line 98, in handle_noargs emit_post_sync_signal(created_models, verbosity, interactive) File "C:\jython2.5b0\Lib\site-packages\django\core\management\sql.py", line 203, in emit_post_sync_signal models.signals.post_syncdb.send(sender=app, app=app, File "C:\jython2.5b0\Lib\site-packages\django\dispatch\dispatcher.py", line 148, in send response = receiver(signal=self, sender=sender, **named) File "C:\jython2.5b0\Lib\site-packages\django\contrib\contenttypes\management .py", line 18, in update_contenttypes ct = ContentType.objects.get(app_label=opts.app_label, File "C:\jython2.5b0\Lib\site-packages\django\db\models\manager.py", line 93, in get return self.get_query_set().get(*args, **kwargs) File "C:\jython2.5b0\Lib\site-packages\django\db\models\query.py", line 303, in get clone = self.filter(*args, **kwargs) File "C:\jython2.5b0\Lib\site-packages\django\db\models\query.py", line 489, in filter return self._filter_or_exclude(False, *args, **kwargs) File "C:\jython2.5b0\Lib\site-packages\django\db\models\query.py", line 503, in _filter_or_exclude clone = self._clone() File "C:\jython2.5b0\Lib\site-packages\django\db\models\query.py", line 601, in _clone query = self.query.clone() File "C:\jython2.5b0\Lib\site-packages\django\db\models\sql\query.py", line 172, in clone obj.dupe_avoidance = self.dupe_avoidance.copy() java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.locks.ReentrantLock.<init>(ReentrantLock.java:234) at java.util.concurrent.ConcurrentHashMap$Segment.<init>(ConcurrentHashMap. java:289) at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:613 ) at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:664 ) at org.python.core.PyDictionary.<init>(PyDictionary.java:55) at org.python.core.PyDictionary.dict_copy(PyDictionary.java:394) at org.python.core.PyDictionary$dict_copy_exposer.__call__(Unknown Source) at django.db.models.sql.query$py.clone$9(C:\jython2.5b0\Lib\site-packages\d jango\db\models\sql\query.py) at django.db.models.sql.query$py.call_function(C:\jython2.5b0\Lib\site-pack ages\django\db\models\sql\query.py) at org.python.core.PyTableCode.call(PyTableCode.java:199) at org.python.core.PyTableCode.call(PyTableCode.java:427) at org.python.core.PyTableCode.call(PyTableCode.java:321) at org.python.core.PyFunction.__call__(PyFunction.java:333) at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:130) at org.python.core.PyMethod.__call__(PyMethod.java:99) at org.python.core.PyObject.__call__(PyObject.java:266) at django.db.models.query$py._clone$49(C:\jython2.5b0\Lib\site-packages\dja ngo\db\models\query.py) at django.db.models.query$py.call_function(C:\jython2.5b0\Lib\site-packages \django\db\models\query.py) at org.python.core.PyTableCode.call(PyTableCode.java:199) at org.python.core.PyTableCode.call(PyTableCode.java:427) at org.python.core.PyTableCode.call(PyTableCode.java:321) at org.python.core.PyFunction.__call__(PyFunction.java:333) at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:130) at org.python.core.PyMethod.__call__(PyMethod.java:99) at org.python.core.PyObject.__call__(PyObject.java:266) at django.db.models.query$py._filter_or_exclude$41(C:\jython2.5b0\Lib\site- packages\django\db\models\query.py) at django.db.models.query$py.call_function(C:\jython2.5b0\Lib\site-packages \django\db\models\query.py) at org.python.core.PyTableCode.call(PyTableCode.java:199) at org.python.core.PyTableCode.call(PyTableCode.java:427) at org.python.core.PyTableCode.call(PyTableCode.java:321) at org.python.core.PyFunction.__call__(PyFunction.java:333) at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:130) java.lang.OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space |
From: Leo S. M. <leo...@gm...> - 2008-12-08 17:17:51
|
On Mon, Dec 8, 2008 at 1:55 PM, Ng, Victor <Vic...@ci...> wrote: > Hi all, > > I'm trying to run the Django testsuite against my new SQL Server backend, > but > I'm getting OOM errors which I'm pretty sure are unrelated to my database > code. > > I'm running tests like this: > > jython --verify c:\dev\django-trunk\tests\runtests.py --settings=settings > -v 2 Yeah, I've experienced this. By now, I'm running jython with a bigger heap space, using the "-J-Xmx1024m" flag. This is not ideal, of course. We need to do some memory profiling to determine if jython is doing something wrong, but nobody have had the time yet, AFAIK :( -- Leo Soto M. http://blog.leosoto.com |
From: Victor Ng <vic...@ci...> - 2008-12-09 20:16:38
|
Leo Soto M. <leo.soto <at> gmail.com> writes: > Yeah, I've experienced this. By now, I'm running jython with a bigger > heap space, using the > "-J-Xmx1024m" flag. > > This is not ideal, of course. We need to do some memory profiling to > determine if jython is doing something wrong, but nobody have had the > time yet, AFAIK :( Have you experienced this while running in production mode or just when running your tests? I can reproduce the memory growth pretty consistently - it happens on the simplest things. Simply doing: from java.lang import System from app.foo.models import MyModel import datetime def test_load(): while True: objects = [o for o in MyModel.objects.all()] del objects System.gc() print datetime.datetime.now() print "Loaded one set of data..." Given a large list of instances of MyModel persisted into the database, I can see the number of strings grow consistently through each iteration of the loop. I can see a pool of strings that grows despite the garbage collection. On top of that, model instances take up *huge* amounts of space. An instance of: class Simple(models.Model): auto = models.AutoField(primary_key=True) first_name = models.CharField(max_length=40) last_name = models.CharField(max_length=40) takes up 1.8MB of memory *per object instance* according to my profiler. Just wanted to share what I've got so far - if there's anyone else looking at memory leak problems - I'd be more than happy to share results to track down what's going on. vic |