Re: [Numpy-discussion] Back to numexpr

[Francesc A. <fa...@ca...>]

A Dimarts 13 Juny 2006 20:46, Tim Hochberg va escriure:
> >Uh, I'm afraid that yes. In PyTables, int64, while being a bit bizarre f=
or
> >some users (specially in 32-bit platforms), is a type with the same righ=
ts
> >than the others and we would like to give support for it in numexpr. In
> > fact, Ivan Vilata already has implemented this suport in our local copy
> > of numexpr, so perhaps (I say perhaps because we are in the middle of a
> > big project now and are a bit scarce of time resources) we can provide
> > the patch against the latest version of David for your consideration.
> > With this we can solve the problem with int64 support in 32-bit platfor=
ms
> > (although addmittedly, the VM gets a bit more complicated, I really thi=
nk
> > that this is worth the effort)
>
> In addition to complexity, I worry that we'll overflow the code cache at
> some point and slow everything down. To be honest I have no idea at what
> point that is likely to happen, but I know they worry about it with the
> Python interpreter mainloop.

That's true. I didn't think about this :-/

> Also, it becomes much, much slower to=20
> compile past a certain number of case statements under VC7, not sure
> why. That's mostly my problem though.

No, this is a general problem (I'd say much more in GCC, because the optimi=
zer=20
runs so slooooow). However, this should only affect to poor developers, not=
=20
users and besides, we should find a solution for int64 in 32-bit platforms.

> One idea that might be worth trying for int64 is to special case them
> using functions. That is using OP_FUNC_LL and OP_FUNC_LLL and some
> casting opcodes. This could support int64 with relatively few new
> opcodes. There's obviously some exta overhead introduced here by the
> function call. How much this matters is probably a function of how well
> the compiler / hardware supports int64 to begin with.

Mmm, in my experience int64 operations are reasonable well supported by mod=
ern=20
32-bit processors (IIRC they normally take twice of the time than int32 ops=
).

The problem with using a long for representing ints in numexpr is that we h=
ave=20
the duality of being represented differently in 32/64-bit platforms and tha=
t=20
could a headache in the long term (int64 support in 32-bit platforms is onl=
y=20
one issue, but there should be more). IMHO, it is much better to assign the=
=20
role for ints in numexpr to a unique datatype, and this should be int64, fo=
r=20
the sake of wide int64 support, but also for future (and present!) 64-bit=20
processors. The problem would be that operations with 32-bit ints in 32-bit=
=20
processors can be slowed-down by a factor 2x (or more, because there is a=20
casting now), but in exchange, whe have full portable code and int64 suppor=
t.=20
In case we consider entering this way, we have two options here: keep VM=20
simple and advertise that int32 arithmetic in numexpr in 32-bit platforms=20
will be sub-optimal, or, as we already have done, add the proper machinery =
to=20
support both integer separately (at the expense of making the VM more=20
complex).  Or perhaps David can come with a better solution (vmgen from=20
gforth? no idea what this is, but the name sounds sexy;-)=20

>
> That brings up another point. We probably don't want to have casting
> opcodes from/to everything. Given that there are 8 types on the table
> now, if we support every casting opcode we're going to have 56(?)
> opcodes just for casting. I imagine what we'll have to do is write a
> cast from int16 to float as OP_CAST_Ii; OP_CAST_FI; trading an extra
> step in these cases for keeping the number of casting opcodes under
> control. Once again, int64 is problematic since you lose precision
> casting to int. I guess in this case you could get by with being able to
> cast back and forth to float and int. No need to cast directly to
> booleans, etc as two stage casting should suffice for this.

Well, we already thought about this. Not only you can't safely cast an int6=
4=20
to an int32 without loosing precistion, but what is worse, you can't even=20
cast it to any other commonly available datatype (casting to a float64 will=
=20
also loose precision). And, although you can afford loosing precision when=
=20
dealing with floating data in some scenarios (but not certainly with a=20
general-purpose library like numexpr tries to be), it is by any means=20
unacceptable loosing 'precision' in ints. So, to my mind, the only solution=
=20
is completely avoiding casting int64 to any type.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"