#205 [python] new thisslots option. faster and less memory.
Hi folks,
We noticed that the wrapper objects which SWIG constructs in python use an instance dict to store the 'this' pointer.
So here's a small patch which allows you to pass a new option called '-thisslots' to swig which will change the swig wrapper to be:
class A(object):
__slots__ = ('this',)
...
And the swig C runtime code has a new #define SWIG_PYTHON_THIS_SLOTS.
When this is turned on, Swig_Python_GetSwigThis, Swig_Python_SetSwigThis, Swig_Python_NewShadowInstance will all first check the tp_members for a 'this' member and set that directly.
In some small tests which I did, creating 10,000 objects took 1.2Meg of memory as opposed to 3.4Meg of memory.
And object creation was ~20 times faster.
The patch is pretty small, and is also entirely conditional on the new thisslots flag.
I've tested this pretty thoroughly on windows and unix and it appears to work.
So would really appreciate it if you'd consider it for inclusion
best,
-Kirat
swigwin-1.3.34 patch for new thisslots python option.
Logged In: YES
user_id=153408
Originator: NO
Is there any reason to make this an option? Why not just replace the current code, if it is better. Or is this only for newer versions of python or something?
If we can, I suppose we should just get rid of the old code.
Logged In: YES
user_id=1695015
Originator: YES
It only works in python 2.2 > since it uses descriptors and slots so prolly will need to keep the old code around.
One caveat, if you have directors enabled, the derived classes inherit from the base and have no tp_members on the derived class, in this case it'll create a dict for the 'this' member just like before.
So I think we'll need the old code around for both previous versions of python and for the director classes.
Also if you set __slots__ on a python instance, you cannot add members by doing arbitrary setattrs on the instance. Some applications may want to add arbitrary members after the instances are created. In the case where you don't want to add more members to the wrapper classes then it makes a lot of sense to use slots.
This last reason is probably really why we don't want to turn this on by default...
best,
-Kirat
Logged In: YES
user_id=14972
Originator: NO
It would be nice to avoid adding yet more options to the python backend. I don't think we should be worrying about support for Python < 2.2 at this point, but the other reasons do sound valid, and this does sound like it might be useful.
Kirat: Have you run the SWIG testsuite with the change in place?
In a built SWIG source tree:
make check-python-test-suite
make check-python-examples
Logged In: YES
user_id=1510526
Originator: NO
There is a (very useful) %pythonnondynamic that causes the swigged python
interface to not allow the addition of new members of classes; this isn't
implemented in terms of __slots__ due to backwards-compatibility issues.
How does this interact with your new code? Could we use %pythonnondynamic
to turn on your implementation?
Logged In: YES
user_id=153408
Originator: NO
Some of this discussion on the trade offs of using the new option should be added to the python chapter of the docs in this patch. Can probably just copy and paste the comments, with a little fix up.
The upcoming -builtin option to swig makes this redundant. Wrappers generated with the -builtin option show the same 20x speedup in object construction, without any of the limitations mentioned here.
In the repository, there is a performance test specifically geared towards object construction in python:
Examples/python/performance/constructor/
To run it:
$ cd Examples/python/performance
$ make constructor
Closing as "out-of-date" since this is apparently superceded by the builtin support.