Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - rcols issue

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Karl,

Thank you for both reporting this issue, then doing this deep investigation as well.

In order to track down exactly when this changed, I stripped down your repro code to:
use PDL::LiteF; # not full PDL so can just build “make core”
$x = sequence(4)+1;
$x = $x->mv(-1,0)->slice("0:2");
$x *= 100;
$y=&PDL::_clump_int($x,-1);
print "x=$x y=$y\n";
$wrong = "$y" eq "[0 0 0]";
exit $wrong;

As you can see in that comment, if one comments out the multiplication, it all still works correctly. The “sequence” can be replaced with hardcoded “pdl [1..5]”. But if one changes either the “mv”, or the “slice”, or the inplace “*=” (including replacing with “$x = $x * 100” ), or the “_clump_int”, it doesn’t misbehave.

Anyway, using the above code to tell me whether I’d found where it started failing, I did “git bisect” as follows (noted here to help anyone who wants to do this themselves; I’d forgotten and had to look up how):
git bisect start
git bisect bad # current “master” is bad
git bisect good 2.025 # tell it it was working as of 2.025
perl Makefile.PL && time make core && perl -Mblib repro-script; echo $? # kept running this, then:
git bisect bad # if failing
git bisect good # if working correctly
git bisect reset # when finished, to close down the bisect
Note the use of “make core” which takes about 2 mins from scratch on my system, vs about 6 to “make” everything, saving lots of time.

It turns out it was this, which was released with 2.078:
commit a4678091acf7e450c02a7b0feaf3c7578f37e53f
Author: Ed J <mo...@us...>
Date:   Sun Apr 3 22:27:28 2022 +0100

    parents of non-flowing trans also track trans_children so can de-register on destroy

Basic/Core/pdlapi.c | 46 +++++++++++++++++++---------------------------
1 file changed, 19 insertions(+), 27 deletions(-)

I am now investigating a fix, which given how specific it is to trigger, will probably be small, and is surely related to book-keeping of parents vs children, and flowing transformations.

Best regards,
Ed

From: Karl Glazebrook via pdl-general<mailto:pdl...@li...>
Sent: 07 January 2024 06:30
To: perldl<mailto:pdl...@li...>
Subject: Re: [Pdl-general] Changes I noted PDL2.025 -> PDL2.084 - rcols issue

Seems I can’t help myself! I have now found the offending line in rcols() and can now reproduce this without rcols. That should at least make it easier to track down.

use PDL;
$x = sequence(100)+1; # This works
$x = $x->mv(-1,0)->slice("0:3");
print $x, "\n";
$x *= 100;
print $x, "\n";
$y=&PDL::_clump_int($x,-1);
print $y, "\n"; # prints [0 0 0 0]
1;

It seems to be the particular combination of ->mv and ->slice (that was on line 693 of misc.pd). This is a 1D ndarray so the ->mv(-1,0) should do nothing. Removing it however makes the problem go away.

Works fine in PDL-2.025. Some bug that has been introduced in ->mv ?

Sorry to the stream of consciousness series of emails. I will stop looking now...

Karl

On 7 Jan 2024, at 4:32 pm, Karl Glazebrook <kar...@ma...<mailto:kar...@ma...>> wrote:

OK here is some deeper diving in to the problem

use PDL;
$x = rcols 'tmp.dat'; # This does causes the error
#$x = sequence(4)+1; # This works
print $x, "\n";
$x *= 100;
print $x, "\n";
$y=&PDL::_clump_int($x,-1);
print $y, "\n"; # prints [0 0 0 0]
;

 looks like the error is happening in the internal pp routine _clump_int in slices.pd.

I also found ->sever() had the same behaviour with rcols:

pdl> $x = rcols 'tmp.dat'
Reading data into ndarrays of type: [ Double ]
Read in  4  elements.

pdl> $x *= 100

pdl> p $x
[100 200 300 400]
pdl> p $x->sever
[0 0 0 0]

Note this is not a general problem with dataflow, if I make a sequence and slice or index is then the ops work fine. It is just something weird on the ndarray produced by rcols.

Karl

On 7 Jan 2024, at 3:41 pm, Karl Glazebrook via pdl-general <pdl...@li...<mailto:pdl...@li...>> wrote:

Ah! I believe the difference between medover and median is a clump(-1) to collapse the dimensions

There does it indeed to be something wrong with clump too, so that is probably the underlying cause

pdl> $x = rcols 'tmp.dat'
Reading data into ndarrays of type: [ Double ]
Read in  4  elements.

pdl> p $x
[1 2 3 4]
pdl> $x *= 100

pdl> p $x
[100 200 300 400]
pdl> p $x->clump(-1)
[0 0 0 0]

What could be happening in rcols() that produces an ndarray that behaves like that?

Karl

On 7 Jan 2024, at 12:48 pm, Luis Mochan <mo...@ic...<mailto:mo...@ic...>> wrote:

I noticed that medover and maxover do work as expected in this case.

On Sun, Jan 07, 2024 at 11:26:56AM +1100, Karl Glazebrook via pdl-general wrote:

Hi all,

This dinosaur just upgraded from PDL v2.025 to v.2.084 (yes, I know that is lame)

I noticed a few things when running one of my complicated codes, I will start seperate email threads

First there seems to be a serious rcols bug:

e.g. create a file

# tmp.dat
1
2
3
4

Loaded PDL v2.084 (supports bad values)
pdl> $x = rcols 'tmp.dat'
Reading data into ndarrays of type: [ Double ]
Read in  4  elements.

pdl> p $x
[1 2 3 4]
pdl> $x *= 100

pdl> p $x
[100 200 300 400]
pdl> p median($x)
0
pdl> p $x
[100 200 300 400]

It seems the median function sees the values BEFORE the inplace multiplacation, whereas print does not. This is very bad. min() and max() are similar. No idea what is going on here! The behaviour or absent from v2.025

Notes
- making a $x->copy() removes the effect
- creating $x using sequence also removes, so it is something to do with rcols() and not inplace in general?

I’d be interested to know if others can reproduce this. It definitely needs a fix

best

Karl

_______________________________________________
pdl-general mailing list
pdl...@li...<mailto:pdl...@li...>
https://lists.sourceforge.net/lists/listinfo/pdl-general

--

                                                                 o
W. Luis Mochán,                      | tel:(52)(777)329-1734     /<(*)
Instituto de Ciencias Físicas, UNAM  | fax:(52)(777)317-5388     `>/   /\
Av. Universidad s/n CP 62210         |                           (*)/\/  \
Cuernavaca, Morelos, México          | mo...@fi...<mailto:mo...@fi...>   /\_/\__/
GPG: 791EB9EB, C949 3F81 6D9B 1191 9A16  C2DF 5F0A C52B 791E B9EB

_______________________________________________
pdl-general mailing list
pdl...@li...<mailto:pdl...@li...>
https://lists.sourceforge.net/lists/listinfo/pdl-general