Re: [Swig-devel] python proxy class performance

[Stefan Z. <ste...@am...>]

On Mon, 25 Oct 2010, Amaury Forgeot d'Arc wrote:

> Hi,
>
> 2010/10/25 Stefan Zager <ste...@am...>:
> > Howdy,
> >
> > I'm using swig to wrap a very large API to python, perl, and ruby.  As a general
> > observation, I find that the ruby and perl wrappers are comparable in
> > performance, but the python wrappers are 5-10x slower for most test cases.
> >
> > As an experiment, I tried hand-wrapping, in optimized C code, part of the API
> > for perl.  I found that, with a few tweaks, I could get the swig wrappers to
> > within 30% of my "best effort" wrappers.
> >
> > I tried the same thing with python, and I found that the swig wrappers were more
> > than 10x slower than my "best effort" wrappers; in some pathological cases, the
> > swig wrappers were 100x slower!  A few more experiments seem to indicate that
> > the lion's share of the performance difference is attributable to the use of
> > pure-python proxy classes by swig.  That kind of performance drag is a
> > show-stopper for this project.
> >
> > I seem to remember that in bygone versions of swig, the '-noproxy' option would
> > disable the creation of proxy classes in python.  However, that no longer seems
> > to work.  Is there any way to get that option back?  Or can anyonse suggest any
> > other remediation of this issue?
>
> The '-noproxy' option just omits the proxy classes, and does not change the
> generated C code.
>
> The solution would be to directly create the classes in C code, but this is a
> considerable change.
> I did this once in a (very very) old version of Swig, and I managed to remove
> almost all code in the shadow python file.
> I did not benchmark it precisely at the time, but I remember a large library
> (wxPython) used to load 5 times faster after the changes.
>
> Since Swig changed so much since then, the complete work needs to be done
> again.  But at least you know that this is possible ;-)
> I'll try to find my old copy of the code.
>
> --
> Amaury Forgeot d'Arc


That would be much appreciated.  The API in question has a few features that
make it very amenable to wrapping with new built-in types; in particular:

- It has a large and deep class hierarchy, but only single inheritance
- Most of the interesting classes don't have ctors/dtors; their memory is
managed within the API, and they are passed as pointers.
- This API isn't suitable for extending (either in C or in scripting).

Don't know how common this case is, but it leaves a lot of performance lying on
the floor.

Thanks,

Stefan