since about a day I have recurrent 'Out of memory errors' when using PyDev. All is fine when I start, but I can see the CPU-activity of the java process gradually going up until Eclipse slowly freezes and error alerts appear.
I'm not aware I have changed anything, there might have been a java-update I missed, but I've tried it with two different vm's and they both have the same problem (the Sun jvm is faster, so it takes less time for eclipse to freeze).
I tried installing Indigo with the latest PyDev downloaded 10 minutes ago, but the problem persists.
I tried commenting out large imports, gave more memory, removing and re-adding the interpreter, but none of these have made a difference.
help ?
any pointers welcome,
Joost.
Am I the only one experiencing this at the moment ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
which should be enough, no ?
And what mystifies me the most is that these problems started completely out-of-the-blue; i've been using PyDev/Eclipse for years and it's great !
I am going to look into this MemoryAnalyzer thing now; at first sight the documentation of it seems a bit lacking for my kind of user..
J.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
finally managed to do a 554MB heapdump and uploaded it to http://rekveld.home.xs4all.nl/stuff
(i have to run now and it still has 30mins of uploading to go)
I modified the settings a bit in order to avoid having to upload a 1.2GB dump, but I don't think that makes a difference for tracking down the problem.
And for those who ever need to do this, what I figured out is to add these two lines to your eclipse.ini file. The last one specifies the directory and it is wise to make a directory especially for it, since the analysis produces lots of little files you don't want to pollute your working directory or home folder.
It took me a while to find this out, since all the documentation is mostly explaining you how to find memoryleaks in your own code, not in the IDE you're using.
I had a look and it seems to have to do with code analysis. A couple of times Eclipse froze on completing an Open_GL attribute, and in the system monitor I could see it filling up 1.2G of memory in 2 or 3 minutes.
From taking a look at the heap it really looks that PyDev got lost when searching things in pyopengl (it recreates many times a module searching for it until you run out of memory… if you had even more memory allocated, it would probably end up raising a stack overflow error), so, it seems that some specific construct triggered that (will have to analyze it better to know what could trigger that).
Which pyopengl version are you using? I'm not sure if your project is closed or open source, but if it's open source, it'd help if you could upload the project that has this problem + the python interpreter with all the libs you're using (if not it's open source, having only the interpreter+libs could be helpful too - if you can't do that either, can you specify the python version, pyopengl version you're using)?
Cheers,
Fabio
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
as usual, thanks for the speedy and clarifying response. For now, the project is closed source. I am an artist and I am mostly coding things for my own use, but there are large parts of this project of which I hope they can be of general use and which I waould like to share. But deadlines first :)
From your pointer I think I will be able to piece together a minimum case that would trigger this too. I am away from my box now, but I will try what I have to do to trigger it starting from a new, empty workspace.
I recently started using pygtkglext, from what you write I get the feeling the problem might be related to that. More detailed info soon !
many thanks,
Joost.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've been playing around a bit in a new workspace and I found my error is relatively simple to reproduce after your previous message:
this file: http://python-gtkglext1.sourcearchive.com/documentation/1.1.0-4/coolwave_8py-source.html
is one of the examples that comes with PyGTKGLExt.
If I start adding lines like glEnable(GL_WORLDPEACE) I can see in the systemmonitor that the java thread starts going up to 100% and memory usage starts increasing rapidly.
(Strange is that just now I tried again after giving Eclipse a lot less memory (in order to give you a smaller heapdump) and then it also happens, but it takes muuuuch longer to reach the Out of Memory point, and sometimes I don't have the patience to get there. That is not what I expected, but not a solution I guess, it does slow everything down.)
I am on Python 2.6.6
using pygtkglext-1.1.0, pyOpenGL 3.0.1, pyGTK 2.0,
what else would you need to know ?
What I just discovered is that in the import section it says
fromgtk.gtkgl.apputilsimport*fromOpenGL.GLimport*
and gtk.gtkgl.apputils imports/redefines (I don't know which, I guess the latter) A LOT of the definitions in OpenGL.GL
I suspect that might have to do with the problem ?
It is confusing to use wildcard imports like in this example; in my own project I import all the OpenGL stuff I need from gtk.gtkgl.apputils, and only two functions from OpenGL.GLU, nothing at all from OpenGL.GL. (I even tried importing all the OpenGL definitions one by one, without a wildcard, but that did not change anything).
I hope this gives you enough info to reproduce ?
Or should I try excluding libs from my interpreter ?
many thanks,
Joost.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just to note: I tried reproducing in windows and wasn't able to get to the OutOfMemory condition… I may have to setup a linux install (maybe something is different there).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'll try to think of something else to try to reproduce it… meanwhile, can you pass me some more info?
go to window > preferences > pydev > logging and mark 'enable logging for code-completion', then, try to do a code-completion in the case that'd give you an out of memory error and pass me the output you get in the console in that case.
Cheers,
Fabio
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I got the exact same problem with pyOpenGL and PyDev in Eclipse.
Ubuntu 10.04 64bit
Eclipse Indigo + PyDev
When I try to open "pyOpenGL" example py file @ http://pypi.python.org/pypi/PyOpenGL-Demo. Eclipse IDE hangs, and java is using 1.2GB of my memory, 100% CPU in 1~2 minutes. Then I got "gc overhead limit exceeded" and "out of memory" error.
Launch Eclipse to a new workspace
Create a new PyDev project (add /usr/bin/python as python interpreter)
download pygtkglext-1.1.0.tar.gz @ https://sourceforge.net/projects/gtkglext/files/pygtkglext/1.1.0/pygtkglext-1.1.0.tar.gz/download
unzip the file, and Copy and paste the example folder into the new PyDev project folder.
Refresh PyDev project, and open one .py file (e.g., "NeHe6Multi.py") in the Eclipse Editor.
wait… check the memory usage by java… growing …growing… very quickly… When memory by java reaches 1.2GB, eclipse hangs…
Even you can quit eclipse, this java process will stay as zombie process.
Q
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If no .py file is displayed in the Eclipse Editor, then everything is OK.
I can right click on the .py file, and choose "run as" "python run", program will run.
As long as you open .py file in the Eclipse editor, Eclipse will hang.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here is the log file for Eclipse (found in the project .metadata folder):
!ENTRY org.python.pydev.core 4 4 2011-12-09 13:51:27.066
!MESSAGE Java heap space
!STACK 0
java.lang.OutOfMemoryError: Java heap space
at org.python.pydev.parser.grammarcommon.AbstractTreeBuilder.openNode(AbstractTreeBuilder.java:187)
at org.python.pydev.parser.grammar27.PythonGrammar27.comparison(PythonGrammar27.java:4089)
at org.python.pydev.parser.grammar27.PythonGrammar27.not_test(PythonGrammar27.java:4078)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_test(PythonGrammar27.java:3984)
at org.python.pydev.parser.grammar27.PythonGrammar27.or_test(PythonGrammar27.java:3940)
at org.python.pydev.parser.grammar27.PythonGrammar27.test(PythonGrammar27.java:3859)
at org.python.pydev.parser.grammar27.PythonGrammar27.listmaker(PythonGrammar27.java:5969)
at org.python.pydev.parser.grammar27.PythonGrammar27.atom(PythonGrammar27.java:5173)
at org.python.pydev.parser.grammar27.PythonGrammar27.atomtrailer(PythonGrammar27.java:4909)
at org.python.pydev.parser.grammar27.PythonGrammar27.power(PythonGrammar27.java:4868)
at org.python.pydev.parser.grammar27.PythonGrammar27.factor(PythonGrammar27.java:4855)
at org.python.pydev.parser.grammar27.PythonGrammar27.term(PythonGrammar27.java:4603)
at org.python.pydev.parser.grammar27.PythonGrammar27.arith_expr(PythonGrammar27.java:4522)
at org.python.pydev.parser.grammar27.PythonGrammar27.shift_expr(PythonGrammar27.java:4441)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_expr(PythonGrammar27.java:4398)
at org.python.pydev.parser.grammar27.PythonGrammar27.xor_expr(PythonGrammar27.java:4355)
at org.python.pydev.parser.grammar27.PythonGrammar27.expr(PythonGrammar27.java:4312)
at org.python.pydev.parser.grammar27.PythonGrammar27.comparison(PythonGrammar27.java:4094)
at org.python.pydev.parser.grammar27.PythonGrammar27.not_test(PythonGrammar27.java:4078)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_test(PythonGrammar27.java:3984)
at org.python.pydev.parser.grammar27.PythonGrammar27.or_test(PythonGrammar27.java:3940)
at org.python.pydev.parser.grammar27.PythonGrammar27.test(PythonGrammar27.java:3859)
at org.python.pydev.parser.grammar27.PythonGrammar27.SmartTestList(PythonGrammar27.java:5773)
at org.python.pydev.parser.grammar27.PythonGrammar27.expr_stmt(PythonGrammar27.java:1738)
at org.python.pydev.parser.grammar27.PythonGrammar27.small_stmt(PythonGrammar27.java:838)
at org.python.pydev.parser.grammar27.PythonGrammar27.simple_stmt(PythonGrammar27.java:778)
at org.python.pydev.parser.grammar27.PythonGrammar27.stmt(PythonGrammar27.java:753)
at org.python.pydev.parser.grammar27.PythonGrammar27.file_input(PythonGrammar27.java:201)
at org.python.pydev.parser.grammar27.PythonGrammar27.file_input(PythonGrammar27.java:1)
at org.python.pydev.parser.PyParser.reparseDocument(PyParser.java:647)
at org.python.pydev.editor.codecompletion.revisited.modules.AbstractModule.createModuleFromDoc(AbstractModule.java:267)
at org.python.pydev.editor.codecompletion.revisited.modules.AbstractModule.createModule(AbstractModule.java:244)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
since about a day I have recurrent 'Out of memory errors' when using PyDev. All is fine when I start, but I can see the CPU-activity of the java process gradually going up until Eclipse slowly freezes and error alerts appear.
I'm not aware I have changed anything, there might have been a java-update I missed, but I've tried it with two different vm's and they both have the same problem (the Sun jvm is faster, so it takes less time for eclipse to freeze).
I tried installing Indigo with the latest PyDev downloaded 10 minutes ago, but the problem persists.
I tried commenting out large imports, gave more memory, removing and re-adding the interpreter, but none of these have made a difference.
help ?
any pointers welcome,
Joost.
Am I the only one experiencing this at the moment ?
forgot to mention i'm on Ubuntu 11.04..
How much memory are you giving it before it runs out of memory? See: http://stackoverflow.com/questions/142357/what-are-the-best-jvm-settings-for-eclipse for a reference configuring vm settings. If the problem persists, please provide a memory dump of your use-case (see: http://wiki.eclipse.org/index.php/MemoryAnalyzer for details on getting a memory dump - zip it, upload it somewhere and pass me the link).
Cheers,
Fabio
Hi Fabio,
in the eclipse.ini I have the following parameters:
-launcher.XXMaxPermSize
512m
-launcher.defaultAction
openFile
-vmargs
-Xms128m
-Xmx1024m
which should be enough, no ?
And what mystifies me the most is that these problems started completely out-of-the-blue; i've been using PyDev/Eclipse for years and it's great !
I am going to look into this MemoryAnalyzer thing now; at first sight the documentation of it seems a bit lacking for my kind of user..
J.
Hi Fabio,
finally managed to do a 554MB heapdump and uploaded it to http://rekveld.home.xs4all.nl/stuff
(i have to run now and it still has 30mins of uploading to go)
I modified the settings a bit in order to avoid having to upload a 1.2GB dump, but I don't think that makes a difference for tracking down the problem.
And for those who ever need to do this, what I figured out is to add these two lines to your eclipse.ini file. The last one specifies the directory and it is wise to make a directory especially for it, since the analysis produces lots of little files you don't want to pollute your working directory or home folder.
It took me a while to find this out, since all the documentation is mostly explaining you how to find memoryleaks in your own code, not in the IDE you're using.
I had a look and it seems to have to do with code analysis. A couple of times Eclipse froze on completing an Open_GL attribute, and in the system monitor I could see it filling up 1.2G of memory in 2 or 3 minutes.
-launcher.XXMaxPermSize
256m
openFile
-vmargs
-Xms128m
-Xmx384m
-Xss4m
-XX:PermSize=128m
-XX:MaxPermSize=384m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/your/path/here
From taking a look at the heap it really looks that PyDev got lost when searching things in pyopengl (it recreates many times a module searching for it until you run out of memory… if you had even more memory allocated, it would probably end up raising a stack overflow error), so, it seems that some specific construct triggered that (will have to analyze it better to know what could trigger that).
Which pyopengl version are you using? I'm not sure if your project is closed or open source, but if it's open source, it'd help if you could upload the project that has this problem + the python interpreter with all the libs you're using (if not it's open source, having only the interpreter+libs could be helpful too - if you can't do that either, can you specify the python version, pyopengl version you're using)?
Cheers,
Fabio
Ha Fabio,
as usual, thanks for the speedy and clarifying response. For now, the project is closed source. I am an artist and I am mostly coding things for my own use, but there are large parts of this project of which I hope they can be of general use and which I waould like to share. But deadlines first :)
From your pointer I think I will be able to piece together a minimum case that would trigger this too. I am away from my box now, but I will try what I have to do to trigger it starting from a new, empty workspace.
I recently started using pygtkglext, from what you write I get the feeling the problem might be related to that. More detailed info soon !
many thanks,
Joost.
Hi Fabio,
I've been playing around a bit in a new workspace and I found my error is relatively simple to reproduce after your previous message:
this file:
http://python-gtkglext1.sourcearchive.com/documentation/1.1.0-4/coolwave_8py-source.html
is one of the examples that comes with PyGTKGLExt.
If I start adding lines like glEnable(GL_WORLDPEACE) I can see in the systemmonitor that the java thread starts going up to 100% and memory usage starts increasing rapidly.
(Strange is that just now I tried again after giving Eclipse a lot less memory (in order to give you a smaller heapdump) and then it also happens, but it takes muuuuch longer to reach the Out of Memory point, and sometimes I don't have the patience to get there. That is not what I expected, but not a solution I guess, it does slow everything down.)
I am on Python 2.6.6
using pygtkglext-1.1.0, pyOpenGL 3.0.1, pyGTK 2.0,
what else would you need to know ?
What I just discovered is that in the import section it says
and gtk.gtkgl.apputils imports/redefines (I don't know which, I guess the latter) A LOT of the definitions in OpenGL.GL
I suspect that might have to do with the problem ?
It is confusing to use wildcard imports like in this example; in my own project I import all the OpenGL stuff I need from gtk.gtkgl.apputils, and only two functions from OpenGL.GLU, nothing at all from OpenGL.GL. (I even tried importing all the OpenGL definitions one by one, without a wildcard, but that did not change anything).
I hope this gives you enough info to reproduce ?
Or should I try excluding libs from my interpreter ?
many thanks,
Joost.
Just to note: I tried reproducing in windows and wasn't able to get to the OutOfMemory condition… I may have to setup a linux install (maybe something is different there).
Ok, just got setup linux, gotten gtk/gtkgl/pyopengl, imported the example you passed ( http://python-gtkglext1.sourcearchive.com/documentation/1.1.0-4/coolwave_8py-source.html ) up to the point to run it and tried to work in the file and (unfortunately) all went well (no Out of memory - in fact, the heap reports as being around 90 MB).
I'll try to think of something else to try to reproduce it… meanwhile, can you pass me some more info?
go to window > preferences > pydev > logging and mark 'enable logging for code-completion', then, try to do a code-completion in the case that'd give you an out of memory error and pass me the output you get in the console in that case.
Cheers,
Fabio
I got the exact same problem with pyOpenGL and PyDev in Eclipse.
Ubuntu 10.04 64bit
Eclipse Indigo + PyDev
When I try to open "pyOpenGL" example py file @ http://pypi.python.org/pypi/PyOpenGL-Demo. Eclipse IDE hangs, and java is using 1.2GB of my memory, 100% CPU in 1~2 minutes. Then I got "gc overhead limit exceeded" and "out of memory" error.
here is my eclipse.ini file:
-startup
plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
-launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.100.v20110505
-showsplash
org.eclipse.platform
-launcher.XXMaxPermSize
512m
-launcher.defaultAction
openFile
-vmargs
-XX:MaxPermSize=512m
-Xms512m
-Xmx1024m
Q
Here are the steps to reproduce this problem
Launch Eclipse to a new workspace
Create a new PyDev project (add /usr/bin/python as python interpreter)
download pygtkglext-1.1.0.tar.gz @ https://sourceforge.net/projects/gtkglext/files/pygtkglext/1.1.0/pygtkglext-1.1.0.tar.gz/download
unzip the file, and Copy and paste the example folder into the new PyDev project folder.
Refresh PyDev project, and open one .py file (e.g., "NeHe6Multi.py") in the Eclipse Editor.
wait… check the memory usage by java… growing …growing… very quickly… When memory by java reaches 1.2GB, eclipse hangs…
Even you can quit eclipse, this java process will stay as zombie process.
Q
If no .py file is displayed in the Eclipse Editor, then everything is OK.
I can right click on the .py file, and choose "run as" "python run", program will run.
As long as you open .py file in the Eclipse editor, Eclipse will hang.
Here is the log file for Eclipse (found in the project .metadata folder):
!ENTRY org.python.pydev.core 4 4 2011-12-09 13:51:27.066
!MESSAGE Java heap space
!STACK 0
java.lang.OutOfMemoryError: Java heap space
at org.python.pydev.parser.grammarcommon.AbstractTreeBuilder.openNode(AbstractTreeBuilder.java:187)
at org.python.pydev.parser.grammar27.PythonGrammar27.comparison(PythonGrammar27.java:4089)
at org.python.pydev.parser.grammar27.PythonGrammar27.not_test(PythonGrammar27.java:4078)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_test(PythonGrammar27.java:3984)
at org.python.pydev.parser.grammar27.PythonGrammar27.or_test(PythonGrammar27.java:3940)
at org.python.pydev.parser.grammar27.PythonGrammar27.test(PythonGrammar27.java:3859)
at org.python.pydev.parser.grammar27.PythonGrammar27.listmaker(PythonGrammar27.java:5969)
at org.python.pydev.parser.grammar27.PythonGrammar27.atom(PythonGrammar27.java:5173)
at org.python.pydev.parser.grammar27.PythonGrammar27.atomtrailer(PythonGrammar27.java:4909)
at org.python.pydev.parser.grammar27.PythonGrammar27.power(PythonGrammar27.java:4868)
at org.python.pydev.parser.grammar27.PythonGrammar27.factor(PythonGrammar27.java:4855)
at org.python.pydev.parser.grammar27.PythonGrammar27.term(PythonGrammar27.java:4603)
at org.python.pydev.parser.grammar27.PythonGrammar27.arith_expr(PythonGrammar27.java:4522)
at org.python.pydev.parser.grammar27.PythonGrammar27.shift_expr(PythonGrammar27.java:4441)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_expr(PythonGrammar27.java:4398)
at org.python.pydev.parser.grammar27.PythonGrammar27.xor_expr(PythonGrammar27.java:4355)
at org.python.pydev.parser.grammar27.PythonGrammar27.expr(PythonGrammar27.java:4312)
at org.python.pydev.parser.grammar27.PythonGrammar27.comparison(PythonGrammar27.java:4094)
at org.python.pydev.parser.grammar27.PythonGrammar27.not_test(PythonGrammar27.java:4078)
at org.python.pydev.parser.grammar27.PythonGrammar27.and_test(PythonGrammar27.java:3984)
at org.python.pydev.parser.grammar27.PythonGrammar27.or_test(PythonGrammar27.java:3940)
at org.python.pydev.parser.grammar27.PythonGrammar27.test(PythonGrammar27.java:3859)
at org.python.pydev.parser.grammar27.PythonGrammar27.SmartTestList(PythonGrammar27.java:5773)
at org.python.pydev.parser.grammar27.PythonGrammar27.expr_stmt(PythonGrammar27.java:1738)
at org.python.pydev.parser.grammar27.PythonGrammar27.small_stmt(PythonGrammar27.java:838)
at org.python.pydev.parser.grammar27.PythonGrammar27.simple_stmt(PythonGrammar27.java:778)
at org.python.pydev.parser.grammar27.PythonGrammar27.stmt(PythonGrammar27.java:753)
at org.python.pydev.parser.grammar27.PythonGrammar27.file_input(PythonGrammar27.java:201)
at org.python.pydev.parser.grammar27.PythonGrammar27.file_input(PythonGrammar27.java:1)
at org.python.pydev.parser.PyParser.reparseDocument(PyParser.java:647)
at org.python.pydev.editor.codecompletion.revisited.modules.AbstractModule.createModuleFromDoc(AbstractModule.java:267)
at org.python.pydev.editor.codecompletion.revisited.modules.AbstractModule.createModule(AbstractModule.java:244)
Eclipse->Windows->Preference->PyDev->Interpreter - Python->Forced Builtins, remove "OpenGL" from the list.
Ok, I think I've been able to fix this issue. Please get the nightly build and see if it works (the current nightly should be the final build - see: http://pydev.blogspot.com/2011/12/preparing-for-pydev-230.html )
Cheers,
Fabio
Hi Fabio,
That's great. I tried the new upgrade of PyDev 2.3 for about 1 hour (add "OpenGL" back to Forced Builtins), everything is OK.
Problem is solved!
Thanks a lot for your excellent work!!
best regards & have a good weekend!
Q