numpy-discussion

[Numpy-discussion] memory corruption bug

[Sven S. <sve...@gm...>]

Hi,
I experienced this strange bug which caused a totally unrelated variable
to be overwritten (no exception or error was raised, so it took me while
to rule out any errors of my own).

The context where this is in is a method of a class (Vecm.getSW()), and
the instance of Vecm is created within a different class (GG.__init__).
Now, the affected variable is in the namespace of GG (it's GG.urate),
and so I would think that anything local in Vecm.getSW() should not
affect GG.urate, right?

Originally I did:

glx[lag:, :] -= temp

But that caused the described problem. Then I tried:

glx[lag:, :] = glx[lag:, :] - temp

But the same problem remains. Then I worked around the slice assignment
like this:

temp4 = r_[zeros([lag, n_y]), temp]
glx = glx - temp4

And everything is ok! However, when I alter the second line of this
workaround to:

glx -= temp4

The problem reappears! So I'm not even sure whether this is one or two
bugs...

This is with yesterday's numpy svn on windows, but the same thing
happens with an earlier svn (~b2) as well. If you need further info,
please tell me how to provide it.

Thanks,
Sven

Re: [Numpy-discussion] memory corruption bug

[Bill B. <wb...@gm...>]

You're sure it's not just pass-by-reference semantics biting you?
If you make an array and pass it to another class or function, by default
they just get a reference to the same array.
so e.g.:

a = numpy.array([1,2,3])
some_class.set_array(a)
a[1] = 10

Then both the local 'a' and the 'a' that some_class has are now [1,10,3].
If you don't want that sharing then you need to make an explicit copy of a
by calling a.copy().
Watch out for lists or dicts of arrays too.   The python idom for copying a
list:  new_list = list_orig[:], won't copy the contents of elements that are
array.  If you want to be sure to make complete copies of complex data
structures, there's the deepcopy method of the copy module.  new_list =
copy.deepcopy(list_orig).

I found a bunch of these sorts of bugs in some code I ported over from
Matlab last week.  Matlab uses copy semantics for everything, so if you pass
a matrix A to a function in Matlab you can always treat it as a fresh local
copy inside the function.  Not so with Python.  I found that locating and
fixing those bugs was the most difficult thing about porting Matlab code to
Numpy (that and the lack of some major toolkit or function you use in Matlab
doesn't have an equivalent in Numpy... like eigs()).

--bb



On 8/26/06, Sven Schreiber <sve...@gm...> wrote:
>
> Hi,
> I experienced this strange bug which caused a totally unrelated variable
> to be overwritten (no exception or error was raised, so it took me while
> to rule out any errors of my own).
>
> The context where this is in is a method of a class (Vecm.getSW()), and
> the instance of Vecm is created within a different class (GG.__init__).
> Now, the affected variable is in the namespace of GG (it's GG.urate),
> and so I would think that anything local in Vecm.getSW() should not
> affect GG.urate, right?
>
> Originally I did:
>
> glx[lag:, :] -= temp
>
> But that caused the described problem. Then I tried:
>
> glx[lag:, :] = glx[lag:, :] - temp
>
> But the same problem remains. Then I worked around the slice assignment
> like this:
>
> temp4 = r_[zeros([lag, n_y]), temp]
> glx = glx - temp4
>
> And everything is ok! However, when I alter the second line of this
> workaround to:
>
> glx -= temp4
>
> The problem reappears! So I'm not even sure whether this is one or two
> bugs...
>
> This is with yesterday's numpy svn on windows, but the same thing
> happens with an earlier svn (~b2) as well. If you need further info,
> please tell me how to provide it.
>
> Thanks,
> Sven
>
>
>
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: [Numpy-discussion] memory corruption bug

[Sven S. <sve...@gm...>]

I appreciate your warnings, thanks. However, they don't seem to apply
here, or why would my described workaround work at all in that case?
Also, afaict, the affected variable is not even passed to the class
where the problematic assignment happens.
-sven

Bill Baxter schrieb:
> You're sure it's not just pass-by-reference semantics biting you?
> If you make an array and pass it to another class or function, by
> default they just get a reference to the same array.
> so e.g.:
> 
> a = numpy.array ([1,2,3])
> some_class.set_array(a)
> a[1] = 10
> 
> Then both the local 'a' and the 'a' that some_class has are now [1,10,3].
> If you don't want that sharing then you need to make an explicit copy of
> a by calling a.copy ().
> Watch out for lists or dicts of arrays too.   The python idom for
> copying a list:  new_list = list_orig[:], won't copy the contents of
> elements that are array.  If you want to be sure to make complete copies
> of complex data structures, there's the deepcopy method of the copy
> module.  new_list = copy.deepcopy(list_orig).
> 
> I found a bunch of these sorts of bugs in some code I ported over from
> Matlab last week.  Matlab uses copy semantics for everything, so if you
> pass a matrix A to a function in Matlab you can always treat it as a
> fresh local copy inside the function.  Not so with Python.  I found that
> locating and fixing those bugs was the most difficult thing about
> porting Matlab code to Numpy (that and the lack of some major toolkit or
> function you use in Matlab doesn't have an equivalent in Numpy... like
> eigs()).
> 
> --bb
> 
> 
> 
> On 8/26/06, *Sven Schreiber* <sve...@gm...
> <mailto:sve...@gm...>> wrote:
> 
>     Hi,
>     I experienced this strange bug which caused a totally unrelated variable
>     to be overwritten (no exception or error was raised, so it took me while
>     to rule out any errors of my own).
> 
>     The context where this is in is a method of a class ( Vecm.getSW()), and
>     the instance of Vecm is created within a different class (GG.__init__).
>     Now, the affected variable is in the namespace of GG (it's GG.urate),
>     and so I would think that anything local in Vecm.getSW () should not
>     affect GG.urate, right?
> 
>     Originally I did:
> 
>     glx[lag:, :] -= temp
> 
>     But that caused the described problem. Then I tried:
> 
>     glx[lag:, :] = glx[lag:, :] - temp
> 
>     But the same problem remains. Then I worked around the slice assignment
>     like this:
> 
>     temp4 = r_[zeros([lag, n_y]), temp]
>     glx = glx - temp4
> 
>     And everything is ok! However, when I alter the second line of this
>     workaround to:
> 
>     glx -= temp4
> 
>     The problem reappears! So I'm not even sure whether this is one or two
>     bugs...
> 
>     This is with yesterday's numpy svn on windows, but the same thing
>     happens with an earlier svn (~b2) as well. If you need further info,
>     please tell me how to provide it.
> 
>     Thanks,
>     Sven
> 

Re: [Numpy-discussion] memory corruption bug

[Charles R H. <cha...@gm...>]

Hi,

On 8/26/06, Bill Baxter <wb...@gm...> wrote:
>
> You're sure it's not just pass-by-reference semantics biting you?
> If you make an array and pass it to another class or function, by default
> they just get a reference to the same array.
> so e.g.:
>
> a = numpy.array ([1,2,3])
> some_class.set_array(a)
> a[1] = 10
>
> Then both the local 'a' and the 'a' that some_class has are now [1,10,3].
> If you don't want that sharing then you need to make an explicit copy of a
> by calling a.copy ().
> Watch out for lists or dicts of arrays too.   The python idom for copying
> a list:  new_list = list_orig[:], won't copy the contents of elements that
> are array.  If you want to be sure to make complete copies of complex data
> structures, there's the deepcopy method of the copy module.  new_list =
> copy.deepcopy(list_orig).
>
> I found a bunch of these sorts of bugs in some code I ported over from
> Matlab last week.  Matlab uses copy semantics for everything,
>

Matlab does copy on write, so it maintains a reference until an element is
modified, at which point it makes a copy. I believe it does this for
efficiency and memory conservation, probably the latter because it doesn't
seem to have garbage collection. I could be wrong about that, though.

Chuck

Re: [Numpy-discussion] memory corruption bug

[Albert S. <fu...@gm...>]

A complete code snippet that reproduces the bug would be most helpful.

If there is a memory corruption problem, it might show up if we run the
problematic code under Valgrind.

Regards,

Albert

> -----Original Message-----
> From: num...@li... [mailto:numpy-
> dis...@li...] On Behalf Of Sven Schreiber
> Sent: 26 August 2006 14:14
> To: numpy-discussion
> Subject: [Numpy-discussion] memory corruption bug
> 
> Hi,
> I experienced this strange bug which caused a totally unrelated variable
> to be overwritten (no exception or error was raised, so it took me while
> to rule out any errors of my own).

Re: [Numpy-discussion] memory corruption bug

[Charles R H. <cha...@gm...>]

Hi,

On 8/26/06, Albert Strasheim <fu...@gm...> wrote:
>
> A complete code snippet that reproduces the bug would be most helpful.


+1. I too suspect that what you have here is a reference/copy problem. The
only thing that is local to the class is the reference (pointer), the data
is global.

Chuck

Re: [Numpy-discussion] memory corruption bug

[Sven S. <sve...@gm...>]

Charles R Harris schrieb:
> +1. I too suspect that what you have here is a reference/copy problem.
> The only thing that is local to the class is the reference (pointer),
> the data is global.
> 
> Chuck

Ok, so you guys were right, turns out that my problem was caused by the
fact that a local assignment like x = y is also by reference only, which
I wasn't really aware of. (Of course, it's explained in Travis' book...)
So that behavior is different from standard python assignments, isn't it?

Sorry for the noise.

-Sven

Re: [Numpy-discussion] memory corruption bug

[Bill B. <wb...@gm...>]

Nope, that's the way python works in general for any type other than
basic scalar types.
>>> a = [1,2,3,4]
>>> b = a
>>> b[1] = 99
>>> print a
[1, 99, 3, 4]
>>> print b
[1, 99, 3, 4]

Also the issue never comes up for types like tuples or strings because
they aren't mutable.

--bb

On 8/28/06, Sven Schreiber <sve...@gm...> wrote:
> Charles R Harris schrieb:
> > +1. I too suspect that what you have here is a reference/copy problem.
> > The only thing that is local to the class is the reference (pointer),
> > the data is global.
> >
> > Chuck
>
> Ok, so you guys were right, turns out that my problem was caused by the
> fact that a local assignment like x = y is also by reference only, which
> I wasn't really aware of. (Of course, it's explained in Travis' book...)
> So that behavior is different from standard python assignments, isn't it?
>
> Sorry for the noise.
>
> -Sven
>