SWIG

I think you can accomplish this by explicitly using the copy constructor for X:

def GetX():
  xvec = GetXs()
  return X(xvec.front())

I added the variable xvec to ensure that the result of GetXs() stays in scope.

Hope that helps.

Stefan

On Wed, Nov 30, 2011 at 4:14 AM, Andrew Ward
<andrew.derek.ward@...> wrote:
> Hello,
>
> Given something like:
>
> class X;
> std::vector<X> * GetXs ();
>
> %template(XVector) std::vector<X>;
> %newobject GetXs;
>
> If the following is done in Python:
>
> def GetX():
>    return GetXs().front()
>
> GetX().foo()
>
> The X returned by GetX is destroyed before foo is called causing a crash
> because it is owned by the XVector. Changing the example to use pop instead
> of front would not crash.
>
> My question is, how do I tell SWIG to return a copy of X any time an item is
> obtained from a wrapped std::vector? My objects are very lightweight, so
> copying is no problem.
>
>
> Regards,
>
>
> Andrew Ward.
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure
> contains a definitive record of customers, application performance,
> security threats, fraudulent activity, and more. Splunk takes this
> data and makes sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-novd2d
> _______________________________________________
> Swig-user mailing list
> Swig-user@...
> https://lists.sourceforge.net/lists/listinfo/swig-user

On 11/30/2011 5:14 AM, Andrew Ward wrote:
> My question is, how do I tell SWIG to return a copy of X any time an item is
> obtained from a wrapped std::vector? My objects are very lightweight, so
> copying is no problem.

I'm probably doing something wrong, but given your example, it appears to work
for me using SWIG 2.0.4 and Python 2.7.2 (debug build):

    Python 2.7.2 (default, Jun 20 2011, 09:53:03) [MSC v.1500 64 bit (AMD64)]
    on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import swigtest
    [44766 refs]
    >>> from swigtest import GetXs
    [44768 refs]
    >>> def GetX():
    ...     return GetXs().front()
    ...
    [44789 refs]
    >>> GetX().foo()
    5
    [44884 refs]
    >>>

My 'X' class and GetXs() are defined as:

    class X
    {
    public:
        X() {}
        virtual ~X() {}

        int foo()   { return 5; }
    };

    std::vector<X> * GetXs ()  { return new std::vector<X>(5); }

My interface file is:

    %module (naturalvar="1") swigtest

    %begin %{
    #include <vector>
    #include "test.h"
    %}

    %include    python.swg
    %include    <typemaps.i>
    %include    <std_vector.i>

    %include "test.h"

    %template(XVector) std::vector<X>;
    %newobject GetXs;


Render me gone,                       |||
Bob                                 ^(===)^
---------------------------------oOO--(_)--OOo---------------------------------
   "One of the penalties for refusing to participate in politics is that you
           end up being governed by your inferiors." -- /Plato/

Bob Hood <bhood2 <at> comcast.net> writes:

> 
> 
>     On 11/30/2011 5:14 AM, Andrew Ward wrote:
>     
>       My question is, how do I tell SWIG to return a copy of X any time an 
item is
> obtained from a wrapped std::vector? My objects are very lightweight, so
> copying is no problem.
> 
>     
>     
>       I'm probably doing something wrong, but given your example, it


Hello Bob,

Thanks for your response. I think you'll find if you access a member of X
from foo() that it will crash, or at least print garbage. For example:

class X
{
public:
    X() : m_i (123) {}
    virtual ~X() {}

    int foo()   { return this->m_i; }
    int m_i;
};

Regards,


Andrew

> Hello Bob,
>
> Thanks for your response. I think you'll find if you access a member of X
> from foo() that it will crash, or at least print garbage. For example:
>
> class X
> {
> public:
>     X() : m_i (123) {}
>     virtual ~X() {}
>
>     int foo()   { return this->m_i; }
>     int m_i;
> };

Again, it's probably me.  I updated my class to match yours:

    class X
    {
        int     m_i;

    public:
        X() : m_i(123) {}
        virtual ~X() {}

        int foo()   { return this->m_i; }
    };

recompiled and tried again:

    Python 2.7.2 (default, Jun 20 2011, 09:53:03) [MSC v.1500 64 bit (AMD64)]
    on win
    32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import swigtest
    [44766 refs]
    >>> def GetX():
    ...     return swigtest.GetXs().front()
    ...
    [44788 refs]
    >>> GetX().foo()
    123
    [44883 refs]
    >>>

The debug version must not be doing something that the release builds are.  At
any rate, I would have suggested the same thing Stephan did, just that you
assign the return value so its reference count is not lowered before you
access it.

Sorry I couldn't help.  :)

Render me gone,                       |||
Bob                                 ^(===)^
---------------------------------oOO--(_)--OOo---------------------------------
   "One of the penalties for refusing to participate in politics is that you
           end up being governed by your inferiors." -- /Plato/

On Wed, 30 Nov 2011, Andrew Ward wrote:

> My question is, how do I tell SWIG to return a copy of X any time an 
> item is obtained from a wrapped std::vector? My objects are very 
> lightweight, so copying is no problem.

I can see a couple of approaches.

Wrapping of vectors of built-in types, such as std::vector<int>, does 
something like this; the wrapped vector does not return references or 
pointers to its items, but copies in the form of Python integers.  You 
could looke at how SWIG does this, perhaps with the aid of the -E 
command-line option.

Alternatively, it might work to provide or %apply an "out" typemap that 
makes a copy to the relevant return types (X&, const X&, and perhaps 
pointers).  You could make this apply to all functions returning X&, or, 
with some care, just to vector methods.  However, I am not absolutely 
certain that this will work with the current implementation of std::vector 
wrapping.

Josh

On Wed, 30 Nov 2011, Bob Hood wrote:

> I'm probably doing something wrong, but given your example, it appears 
> to work for me using SWIG 2.0.4 and Python 2.7.2 (debug build):
...
> My interface file is:
> %module (naturalvar="1") swigtest
>
> %begin %{
> #include <vector>
> #include "test.h"
> %}
>
> %include    python.swg
> %include    <typemaps.i>
> %include    <std_vector.i>
>
> %include "test.h"
>
> %template(XVector) std::vector<X>;
> %newobject GetXs;

Does this %newobject come too late to have an effect, replacing the 
original problem with a memory leak?

Josh

On 11/30/2011 12:38 PM, Josh Cherry wrote:
>
> On Wed, 30 Nov 2011, Bob Hood wrote:
>
>> I'm probably doing something wrong, but given your example, it appears to
>> work for me using SWIG 2.0.4 and Python 2.7.2 (debug build):
> ...
>> My interface file is:
>> %module (naturalvar="1") swigtest
>>
>> %begin %{
>> #include <vector>
>> #include "test.h"
>> %}
>>
>> %include    python.swg
>> %include    <typemaps.i>
>> %include    <std_vector.i>
>>
>> %include "test.h"
>>
>> %template(XVector) std::vector<X>;
>> %newobject GetXs;
>
> Does this %newobject come too late to have an effect, replacing the original
> problem with a memory leak?

Yes, thanks.  That's what I was doing wrong.  :)

Render me gone,                       |||
Bob                                 ^(===)^
---------------------------------oOO--(_)--OOo---------------------------------
   "One of the penalties for refusing to participate in politics is that you
           end up being governed by your inferiors." -- /Plato/

Stefan Zager <szager <at> gmail.com> writes:

> 
> I think you can accomplish this by explicitly using the copy constructor for X:
> Stefan
> 

Hi Stefan,

Thanks for your suggestion. Unfortunately I am designing an API for others to
use, and cannot trust them to remember to do this.

Regards,

Andrew

Josh Cherry <jcherry <at> ncbi.nlm.nih.gov> writes:

> Alternatively, it might work to provide or %apply an "out" typemap that 

Hi Josh,

I have managed to get this working, here is the solution:

%define RETURN_COPY_FROM_VECTOR(TYPE)
%typemap(out) TYPE & front {
	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
        SWIG_POINTER_OWN | %newpointer_flags);
  }
%typemap(out) TYPE & back {
	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
        SWIG_POINTER_OWN | %newpointer_flags);
  }
%typemap(out) TYPE & __getitem__ {
	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
        SWIG_POINTER_OWN | %newpointer_flags);
  }
%enddef

RETURN_COPY_FROM_VECTOR(X)


In the process I have found a bug in the Python std::vector implementation.
The __getitem__ method has two versions, one takes an index, the other takes
a slice object. The version that takes a slice object returns a new std::vector
but does not use SWIG_POINTER_OWN and therefore leaks memory.
I cannot see a way to patch this up as the
typemap:
%typemap(out) TYPE & __getitem__
is only applied to the version of __getitem__ that takes an index, so there
seems no way to modify the other version through typemaps.

Also, if anyone is interested, here is the solution for C# and Java:

// For some unknown reason SWIGEXCODE causes a syntax error if used with
// RETURN_COPY_FROM_VECTOR So define our own version called SWIGEXCODE3

%define SWIGEXCODE3
 "if ($modulePINVOKE.SWIGPendingException.Pending) throw
    $modulePINVOKE.SWIGPendingException.Retrieve();"
%enddef

%define RETURN_COPY_FROM_VECTOR(TYPE)
%typemap(out) std::vector<TYPE>::const_reference getitem {
    jresult = (void *) new TYPE (*result);
  }
%typemap(csout, excode=SWIGEXCODE3) std::vector<TYPE>::const_reference getitem {
    TYPE ret = new TYPE ($imcall, true);
    $excode
    return ret;
  }
%enddef

RETURN_COPY_FROM_VECTOR(X)



%define RETURN_COPY_FROM_VECTOR(TYPE)
%typemap(out) std::vector<TYPE>::const_reference get {
    *(std::vector< TYPE >::value_type **)&jresult = new TYPE (*result);
  }
%typemap(javaout) std::vector<TYPE>::const_reference get {
    return new TYPE ($jnicall, true);
  }
%enddef

RETURN_COPY_FROM_VECTOR(X)

On Thu, Dec 1, 2011 at 3:38 AM, Andrew Ward <andrew.derek.ward@...> wrote:

> In the process I have found a bug in the Python std::vector implementation.
> The __getitem__ method has two versions, one takes an index, the other takes
> a slice object. The version that takes a slice object returns a new std::vector
> but does not use SWIG_POINTER_OWN and therefore leaks memory.

Thanks for catching this.  I have filed a bug on it, and committed a
fix which will go into the next SWIG release.

https://sourceforge.net/tracker/?func=detail&aid=3447426&group_id=1645&atid=101645

Stefan

On 01/12/11 11:38, Andrew Ward wrote:
> Josh Cherry<jcherry<at>  ncbi.nlm.nih.gov>  writes:
>
>> Alternatively, it might work to provide or %apply an "out" typemap that
>
> Hi Josh,
>
> I have managed to get this working, here is the solution:
>
> %define RETURN_COPY_FROM_VECTOR(TYPE)
> %typemap(out) TYPE&  front {
> 	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>          SWIG_POINTER_OWN | %newpointer_flags);
>    }
> %typemap(out) TYPE&  back {
> 	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>          SWIG_POINTER_OWN | %newpointer_flags);
>    }
> %typemap(out) TYPE&  __getitem__ {
> 	$result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>          SWIG_POINTER_OWN | %newpointer_flags);
>    }
> %enddef
>
> RETURN_COPY_FROM_VECTOR(X)
>
>
> In the process I have found a bug in the Python std::vector implementation.
> The __getitem__ method has two versions, one takes an index, the other takes
> a slice object. The version that takes a slice object returns a new std::vector
> but does not use SWIG_POINTER_OWN and therefore leaks memory.
Which version of SWIG are you using because I can't see this problem. 
This overload of __getitem__:

   std::vector<_Tp,_Alloc >* __getitem__(PySliceObject *slice);

has a %feature("new") for ensuring SWIG_POINTER_OWN gets generated to 
take ownership of the memory. And indeed the generated code does have 
SWIG_POINTER_OWN in the wrappers, so there is no memory leak.

William

On 03/02/12 06:54, William S Fulton wrote:
> On 01/12/11 11:38, Andrew Ward wrote:
>> Josh Cherry<jcherry<at> ncbi.nlm.nih.gov> writes:
>>
>>> Alternatively, it might work to provide or %apply an "out" typemap that
>>
>> Hi Josh,
>>
>> I have managed to get this working, here is the solution:
>>
>> %define RETURN_COPY_FROM_VECTOR(TYPE)
>> %typemap(out) TYPE& front {
>> $result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>> SWIG_POINTER_OWN | %newpointer_flags);
>> }
>> %typemap(out) TYPE& back {
>> $result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>> SWIG_POINTER_OWN | %newpointer_flags);
>> }
>> %typemap(out) TYPE& __getitem__ {
>> $result = SWIG_NewPointerObj(%new_copy(*$1, $*ltype), $descriptor,
>> SWIG_POINTER_OWN | %newpointer_flags);
>> }
>> %enddef
>>
>> RETURN_COPY_FROM_VECTOR(X)
>>
>>
>> In the process I have found a bug in the Python std::vector
>> implementation.
>> The __getitem__ method has two versions, one takes an index, the other
>> takes
>> a slice object. The version that takes a slice object returns a new
>> std::vector
>> but does not use SWIG_POINTER_OWN and therefore leaks memory.
> Which version of SWIG are you using because I can't see this problem.
> This overload of __getitem__:
>
> std::vector<_Tp,_Alloc >* __getitem__(PySliceObject *slice);
>
> has a %feature("new") for ensuring SWIG_POINTER_OWN gets generated to
> take ownership of the memory. And indeed the generated code does have
> SWIG_POINTER_OWN in the wrappers, so there is no memory leak.
Not to worry, I found the bug report and I am looking at the version 
which includes the already committed fix.

William