numpy-discussion

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Tim Hochberg <tim...@ie...> wrote:
>
> Charles R Harris wrote:
> >
> >
> > On 10/18/06, *Tim Hochberg* <tim...@ie...
> > <mailto:tim...@ie...>> wrote:
> >
> >     Charles R Harris wrote:
> >
> >     [SNIP]
> >     >
> >     > I'm not talking about the keyword in the ravel call, I'm talking
> >     about
> >     > the flag in a. The question is: do we *need* a fortran flag. I am
> >     > argueing not, because the only need is for fortran contiguous
> >     arrays
> >     > to pass to fortran function, or translation from fortran
> contiguous
> >     > arrays to numpy arrays. What I am saying is that things are
> >     > unnecessarily complicated. None of the LaPack stuff seems to use
> >     the
> >     > Fortran stuff, they just transpose and copy. I don't even think
> >     I want
> >     > to change that, because it is *clear* what is going on.
> >     Interfacing to
> >     > fortran is all about memory layout, nothing more or less.
> >     >
> >
> >     Chuck,
> >
> >     There are two things here. One is the order keyword and one is the
> >     FORTRAN flag. The latter is mainly an optimization for use at the
> >     C-level so that one doesn't have to check whether a given array is
> in
> >     contiguous FORTRAN order by examining the strides, in the same way
> >     that
> >     the CONTIGUOUS flag allows you to skip examining the strides when
> you
> >     need a contiguous C-order matrix.
> >
> >
> > That sounds like the two flags should be named f-contiguous and
> > c-contiguous. Then they would be orthogonal and one could have all
> > four combinations. Is that the case now? Perhaps I am misunderstanding
> > the meaning of the flags.
> That is the case now. The flag names simply mirror their values in C.
> Why they have those names in something of a historical accident I
> believe. Take a look at this:


OK, that is good. I no longer have any objection to the flags, I just wish
the names were more descriptive of what they mean. In fact, it looks like
the following sort of construction will be useful in the linalg module.

In [17]:a = array([[1,2],[3,4]], dtype=int)

In [18]:b = array(a, dtype=double, order='f')

In [19]:b.flags
Out[19]:
  CONTIGUOUS : False
  FORTRAN : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

I've been a pain in the a** because I really want to know what is going on
down in the boiler room.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ie...>]

>
> I'm not talking about the keyword in the ravel call, I'm talking about 
> the flag in a.
Ah.  Yes, I see.  I misunderstood.  Of course ravel ignores the FORTRAN 
flag (actually it doesn't because if a copy is not necessary it doesn't 
make one).    The key is that the Python user doesn't need to care about 
the array flag unless they are interfacing to compiled code.  That's the 
point of the flag.  It's actually redundant because it could be checked 
every time it's needed.  But, right now, it's kept updated so that the 
check is simple.   The same is true with the C-CONTIGUOUS flag (called 
contiguous).

> The question is: do we *need* a fortran flag.
No, you don't *need* the flag.  But, it saves copying data to check it 
(look how many times ISFORTRAN is called in the code).  Without the flag 
all of those cases would need to do a strides-check which is done in the 
UpdateFlags code.
> I am argueing not, because the only need is for fortran contiguous 
> arrays to pass to fortran function, or translation from fortran 
> contiguous arrays to numpy arrays. What I am saying is that things are 
> unnecessarily complicated.
I disagree.  It's actually not that complicated.   Even if it was 
compilcated to implement, the point is that it is now done.  There is no 
sense ripping it out (that would be a huge pain and for what purpose?)  
The FORTRAN flag gives us a lot more flexibility when it comes to 
copying data or not. 

I think part of the complication is that you are misunderstanding some 
of the terms and the purposes of the keywords.
> None of the LaPack stuff seems to use the Fortran stuff, they just 
> transpose and copy.
It doesn't now only because I haven't had time to go through and change 
it, but it should.   Look at scipy's LaPack interface.  It (through 
f2py) uses the FORTRAN stuff extensively (much was borrowed from there 
in the first place).

-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ie...>]

Tim Hochberg wrote:
> One thing that may be confusing the issue is that, as I understand it, 
> FORTRAN and CONTIGUOUS together represent three states which I'll call 
> FORTRAN_ORDER, C_ORDER and DISCONTIGUOUS. 

Yep,  that's what they mean.  CONTIGUOUS is the name Numeric gave it and 
it meant C-order contiguous.  We have kept the same meaning.  All we've 
done is selected out from the class of arrays that Numeric called 
DISTCONTIGUOUS, arrays that are FORTRAN-order (and so still 
single-segment), but discontiguous in the sense that Numeric had.

> I periodically wonder if it 
> would be valuable to have a way to query the order directly: the result 
> would be "C", "F" or None, just like the order keyword that is passed 
> in. 
You an do it with the flags

a.flags.contiguous
a.flags.fortran

Discontiguous is when both of these are false.  Note that for a.ndim < 
2, both a.flags.contiguous and a.flags.fortran are true if one of them 
is true.

This is all explained in the first chapters of my book.  You have to 
understand CONTIGUOUS == C-order contiguous and FORTRAN == Fortran-order 
contiguous.


-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[George N. <gn...@go...>]

> > None of the LaPack stuff seems to use the Fortran stuff, they just
> > transpose and copy.

You've got me worried here. I have assumed that when you start with a
c-contiguous array, a, with say,a.shape = (m,n), if you use the
transpose as an argument to a fortran routine which requires an mxn
size array, then no copying is required.

This seems to work for me -- the transpose *does* have fortran order.
Also, in f2py, if I use -DF2PY_REPORT_ON_ARRAY_COPY=1 I receive no
alert of any copy.

 Apologies if these are simply confused ravings.

George Nurser.

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, George Nurser <gn...@go...> wrote:
>
> > > None of the LaPack stuff seems to use the Fortran stuff, they just
> > > transpose and copy.
>
> You've got me worried here. I have assumed that when you start with a
> c-contiguous array, a, with say,a.shape = (m,n), if you use the
> transpose as an argument to a fortran routine which requires an mxn
> size array, then no copying is required.
>
> This seems to work for me -- the transpose *does* have fortran order.


Nope. The result is an (n,m) array in fortran order, not an (m,n) array in
fortran order.

In [52]:a = array([[1,2,3],[4,5,6]])

In [53]:a.transpose().flags
Out[53]:
  CONTIGUOUS : False
  FORTRAN : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [54]:a.transpose().shape
Out[54]:(3, 2)

Looks like what you want is either fastcopyandtranspose or the order flag.

In [56]:fastCopyAndTranspose(a).flags
Out[56]:
  CONTIGUOUS : True
  FORTRAN : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

which is a (n,m) array in c order, i.e., an (m,n) array in fortran order. Or

In [57]:array(a, order='F').flags
Out[57]:
  CONTIGUOUS : False
  FORTRAN : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

which is a (m,n) array in fortran order.

Also, in f2py, if I use -DF2PY_REPORT_ON_ARRAY_COPY=1 I receive no
> alert of any copy.


f2py takes care of the ordering, which is one reason why it is so useful.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[George N. <gn...@go...>]

On 18/10/06, Charles R Harris <cha...@gm...> wrote:
>
>
> On 10/18/06, George Nurser <gn...@go...> wrote:
> > > > None of the LaPack stuff seems to use the Fortran stuff, they just
> > > > transpose and copy.
> >
> > You've got me worried here. I have assumed that when you start with a
> > c-contiguous array, a, with say, a.shape = (m,n), if you use the
> > transpose as an argument to a fortran routine which requires an mxn
> > size array, then no copying is required.
> >
> > This seems to work for me -- the transpose *does* have fortran order.
>
> Nope. The result is an (n,m) array in fortran order, not an (m,n) array in
> fortran order.

Presumably that's because it's a view of the original array.

>> > Also, in f2py, if I use -DF2PY_REPORT_ON_ARRAY_COPY=1 I receive no
> > alert of any copy.
>
> f2py takes care of the ordering, which is one reason why it is so useful.

Yes, when I first used it, I assumed that the fortran routine had to
use an nxm array. But f2py is clever enough to make the above work.

George.

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ee...>]

Charles R Harris wrote:

>
> Could we make a few changes ;)
>
> For printing the flags I would suggest using C-Contiguous and 
> F-Contiguous so folks don't have to read the book. And at the c level 
> define alternates, i.e, #define c-contiguous contiguous or whatever. 
> That way backward compatibility would be maintained but more 
> descriptive names would be available.

Printing the flags is not intended for the casual user.  So, I'd like to 
keep consistent with C-level names and the names that are printed.

CONTIGUOUS is the old name Numeric used.  It always meant C-CONTIGUOUS 
and so that meaning is preserved.  FORTRAN is the new one flag and it 
means FORTRAN CONTIGUOUS.

So, you want something like?

#define NPY_C_CONTIGUOUS NPY_CONTIGUOUS
#define NPY_F_CONTIGUOUS NPY_FORTRAN

and to have C_CONTIGUOUS and F_CONTIGUOUS print for the flags description?

I'm not opposed to it, but I don't really see the need.  It's just a 
semantic question.  Given the history of CONTIGUOUS in Numeric I thought 
it was clear that CONTIGUOUS always meant C-contiguous. 

-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Stefan v. d. W. <st...@su...>]

On Tue, Oct 17, 2006 at 07:53:11PM -0600, Travis Oliphant wrote:
> Stefan van der Walt wrote:
> > Hi all,
> >
> > Some of you may have seen the interesting thread on Fortran-ordering
> > earlier.  I thought it might be fun to set up a short quiz which test=
s
> > your knowledge on the topic.
> >
> > If you're up for the challenge, take a look at
> >
> > http://mentat.za.net/numpy/quiz
> >
> > I won't be held liable for any emotional trauma caused, self-inflicte=
d
> > wounds or brain-core meltdowns.
> >  =20
>=20
> Cute (especially the comment if you get them all right).   I'm not sure=
=20
> if this quiz is a veiled complaint about the rules for Fortran ordering=
=20
> or not ;-)

If it is, it is very tactfully disguised ;) Actually, I was just
trying to figure out how Fortran ordering works and thought others
might be interested.

I reckoned you would be the only one to get 100%, although now you've
explained the rules so clearly a couple of other might grasp for the
holy grail too.

> In my mind the Fortran ordering rules are consistent (if not completely=
=20
> bug-free).   You just have to get the right idea of what is meant by th=
e=20
> order argument when you use it.   If you think you are having trouble=20
> figuring out the rules, think of the trouble it was to figure out what=20
> they should be and then to code them up.

My sympathies (no, really).

> 2) On reshaping, the order argument specifies how you think the array i=
s=20
> organized.  Whenever you make a significant reshape you are telling the=
=20
> computer to re-interpret the chunk of data in a different way, it makes=
=20
> a big difference as to how you think about that chunk of data.  Do you=20
> think of it as organized rows-first (C-order) or columns-first=20
> (Fortran-order).   The order argument allows you to specify how you=20
> think about it and indicates the 1-d indexing order of the array.  It=20
> also fills in the newly-shaped array in exactly that same order.  =20
> Semantically, one could technically separate those two concepts and hav=
e=20
> one order argument that specifies how you think about the input and=20
> another that specifies how you think about the output.   But, I really=20
> didn't want to go there and couldn't see a real advantage to that.  So,=
=20
> the single order argument specifies how you think about both.

Thanks for the detailed overview (we should put it on the wiki).
Another thing I'm wondering about: when exactly does reshape need to
make copies?

One last case, which confuses me still (probably because it is
04:16am):

In [41]: x =3D N.array([[0,1,2],[3,4,5]],order=3D'F')

In [42]: x
Out[42]:=20
array([[0, 1, 2],
       [3, 4, 5]])

I assume the data is now stored in memory as

[0 3 1 4 2 5] (column-wise)

If I now do

x.reshape((3,2),order=3D'C')

i.e. take that block of memory, assume it is in 'C' order, and make
its shape (3,2), I expect

[[0 3]
 [1 4]
 [2 5]]

but get

[[1 2]
 [3 4]
 [5 6]]

I'm obviously missing something trivial -- I'll try again tomorrow.

Cheers
St=E9fan

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ie...>]

Stefan van der Walt wrote:
> One last case, which confuses me still (probably because it is
> 04:16am):
>
> In [41]: x = N.array([[0,1,2],[3,4,5]],order='F')
>
> In [42]: x
> Out[42]: 
> array([[0, 1, 2],
>        [3, 4, 5]])
>
> I assume the data is now stored in memory as
>
> [0 3 1 4 2 5] (column-wise)
>
> If I now do
>
> x.reshape((3,2),order='C')
>
> i.e. take that block of memory, assume it is in 'C' order, and make
> its shape (3,2), I expect
>
> [[0 3]
>  [1 4]
>  [2 5]]
>
> but get
>
> [[1 2]
>  [3 4]
>  [5 6]]
>
> I'm obviously missing something trivial -- I'll try again tomorrow.
>   

I think I see what is going on and where people are getting tripped up.  
  You have to remember, that to NumPy it doesn't semantically matter 
what the "ordering" of the array is.  There is no guarantee that C- 
order or Fortran-order is *ever* preserved through an operation.    
Because, in fact the general memory model of the array has no defined 
"order".  It's defined by the strides array.  It just so happens that 
two special-cases are tracked so that we can call out to compiled 
routines that expect contiguous arrays more easily.  

So, your mistake is trying to think that the "block" of memory is [0, 3, 
1, 4, 2, 5]  when you pass the Fortran-order array to the reshape 
method.    While this is true, and it means that you will save a copy if 
you passed this off to a Fortran routine, the reshape command does not 
use this information in determining how to "think-about" the input 
array.  In fact,  the reshape method does not allow any way to specify 
the order of the "input" array (self) separately from the order of the 
output array.   The order argument indicates the defined order of both 
input and output.  You might think that the order of self should be used 
as the order of  the input array.  The problem with this is, again, that 
a general array does not have a defined "order".  What should be used as 
the assumed "order" for an un-strided array?  You're left with an 
unresolved question.

To avoid two input order arguments, I just let order indicate the order 
for both the input and the output arrays.  We could provide both, but 
this seems a bit over-done as the same could be accomplished by 
separately raveling the input to 1-d and then specifying the order 
argument on reshape.

Please continue to question.  All the code needs as much review as it 
can get.

Best regards,


-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/17/06, Travis Oliphant <oli...@ie...> wrote:
>
> Stefan van der Walt wrote:
> > One last case, which confuses me still (probably because it is
> > 04:16am):
>
> Please continue to question.  All the code needs as much review as it
> can get.


I am really starting to wonder why we need an order keyword at all except
when order matters for interfacing, i.e., in contiguos arrays where one
wants fortran contiguous or c contiguous. For output type stuff:

tofile
tostring
ravel
flatten

These all need the keyword so that they can be used to produce files and
arrays to interface to fortran. Right now the fortran interface can be
achieved by:

a.T.tofile
a.T.tostring
a.T.ravel
a.T.flatten

The order keyword is just syntatic sugar that makes the intent clearer. For
the input type stuff

fromfile
fromstring
reshape-1d-array (unravel?)

Currently, the key operation is reshape, which only needs to return a view
in fortran order and doesn't even need to mark the resulting array as
fortran order because, well, because it works just fine in numpy as is, it
just isn't contiguous. If the other functions took shape and order, reshape
wouldn't even need the order keyword.

I don't see why the array constructor needs the order keyword, it doesn't
*do* anything. For instance

a = array([[1,2,3],[4,5,6]], order='F')

doesn't produce a fortran contiguous array, it produces the same array as
the 'C' form, just sets the fortran flag and marks contiguous as False. What
is the use of that? It is just a generic non-contiguous numpy array. And

In [131]: ascontiguousarray(array([[1,2,3],[4,5,6]], dtype=int8,
order='F')).flags
Out[131]:
  CONTIGUOUS : True
  FORTRAN : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Doesn't produce a fortran contiguous array, so what use was the flag? And

In [141]: array([1,2,3,4,5,6], dtype=int8).reshape((2,3),
order='F').astype(int16).flags
Out[141]:
  CONTIGUOUS : True
  FORTRAN : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

reorders stuff in memory, so is a bug looking to happen in a fortran
interface.

mmapped files are the only thing I can think of where one might want vary an
operation depending on Fortran ordering because seeking out of order is very
expensive. But that means adapting algorithms depending on order type,
better I think to just stick to using the small strided dimensions when
appropriate.

It would be helpful in debugging all this order stuff if it was clear what
was supposed to happen in every case. Ravel, for instance, ignores the
FORTRAN flag, again begging the question as to why we *have* the flag.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Tim H. <tim...@ie...>]

One thing that may be confusing the issue is that, as I understand it, 
FORTRAN and CONTIGUOUS together represent three states which I'll call 
FORTRAN_ORDER, C_ORDER and DISCONTIGUOUS. I periodically wonder if it 
would be valuable to have a way to query the order directly: the result 
would be "C", "F" or None, just like the order keyword that is passed 
in. This might well eliminate sine confusion. However, 99% of the time 
the order just doesn't matter, so it's probably pointless.


-tim

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Travis Oliphant <oli...@ee...> wrote:
>
> Charles R Harris wrote:
>
> > Well, I knew that for numeric, but it was a good deal less obvious in
> > combo with the order keyword.  For instance, contiguous could change
> > its meaning to match up with FORTRAN, so that FORTRAN=True and
> > CONTIGUOUS=True meant Fortran contiguous, which was sort of what I was
> > thinking. Explicit never hurts.
>
>
> Ahh, so you were confused by looking at flags for 1-d arrays (where


More confused by what seemed to make sense to me. Tim was the one who
actually ran the experiment to see what was going on, and he wasn't sure
what FORTRAN meant either. Now if I had accessed the full set of offsets,
strides, and counts, it would all have become clear. Numpy tries to hide the
nasty details, which is good until you really have to know what it happening
underneath.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Travis Oliphant <oli...@ie...> wrote:
>
>
> >
> > Currently, the key operation is reshape, which only needs to return a
> > view in fortran order and doesn't even need to mark the resulting
> > array as fortran order because, well, because it works just fine in
> > numpy as is, it just isn't contiguous. If the other functions took
> > shape and order, reshape wouldn't even need the order keyword.
> The flag is the there as a quick check for interfacing.  The order
> keyword grew because it was useful to avoid the arbitrariness of
> C-contiguous order for those who prefer to think of it differently.
> Remember the .T attribute for .transpose() was a recent addition and
> sticking .transpose() everywhere is a lot more ugly.   But, yes, many
> uses of the order keyword could be replaced by preceding with
> .transpose() --- this is not without cost, however.
>
> >
> > I don't see why the array constructor needs the order keyword, it
> > doesn't *do* anything. For instance
> >
> > a = array([[1,2,3],[4,5,6]], order='F')
> >
> > doesn't produce a fortran contiguous array, it produces the same array
> > as the 'C' form, just sets the fortran flag and marks contiguous as
> > False. What is the use of that? It is just a generic non-contiguous
> > numpy array.
>
> What?   You're not understanding something.  The order flag definitely
> does something here.   First of all it seems like you are not
> understanding the meaning of the CONTIGUOUS flag.   CONTIGUOUS means
> "C-order contiguous"  while FORTRAN means "FORTRAN-order contiguous".
> That's why I use the word single-segment to talk about FORTRAN-order or
> C-contiguous order.  For Numeric, CONTIGUOUS always meant C-order
> contiguous and we are continuing that tradition.  All we've done is
> notice that there is such a think as FORTRAN-order contiguous and copies
> do not need to be made in all circumstances when you have FORTRAN-order.
>
> Look at the difference between:
>
> a = array([[1,2,3],[4,5,6]],order='F').data[:]
>
> b = array([[1,2,3],[4,5,6]]).data[:]
>
> Notice the layout is definitely different between a and b.
>
> > And
> >
> > In [131]: ascontiguousarray(array([[1,2,3],[4,5,6]], dtype=int8,
> > order='F')).flags
> > Out[131]:
> >   CONTIGUOUS : True
> >   FORTRAN : False
> >   OWNDATA : True
> >   WRITEABLE : True
> >   ALIGNED : True
> >   UPDATEIFCOPY : False
> >
> > Doesn't produce a fortran contiguous array, so what use was the flag?
> And
>
> Because you requested a C-contiguous array --- that's what contiguous
> means in NumPy (exactly what it meant in Numeric).
>
> >
> > In [141]: array([1,2,3,4,5,6], dtype=int8).reshape((2,3),
> > order='F').astype(int16).flags
> > Out[141]:
> >   CONTIGUOUS : True
> >   FORTRAN : False
> >   OWNDATA : True
> >   WRITEABLE : True
> >   ALIGNED : True
> >   UPDATEIFCOPY : False
> >
> > reorders stuff in memory, so is a bug looking to happen in a fortran
> > interface.
>
> Yes, like I said before, all kinds of operations alter the "layout" of
> data.  You can't assume all operations will preserve FORTRAN ordering.
> FORTRAN-order has meaning beyond how the data is actually set out in
> memory.  Sometimes it indicates how you think it is layed out when you
> are doing re-shaping operations.
>
> >
> > mmapped files are the only thing I can think of where one might want
> > vary an operation depending on Fortran ordering because seeking out of
> > order is very expensive. But that means adapting algorithms depending
> > on order type, better I think to just stick to using the small strided
> > dimensions when appropriate.
> >
> > It would be helpful in debugging all this order stuff if it was clear
> > what was supposed to happen in every case. Ravel, for instance,
> > ignores the FORTRAN flag, again begging the question as to why we
> > *have* the flag.
> No it doesn't.  Please show your evidence.  Look:
>
> a = array([[1,2,3],[4,5,6]])
>
> print a.ravel()
> [1 2 3 4 5 6]
>
> print a.ravel('F')
> [1 4 2 5 3 6]


I'm not talking about the keyword in the ravel call, I'm talking about the
flag in a. The question is: do we *need* a fortran flag. I am argueing not,
because the only need is for fortran contiguous arrays to pass to fortran
function, or translation from fortran contiguous arrays to numpy arrays.
What I am saying is that things are unnecessarily complicated. None of the
LaPack stuff seems to use the Fortran stuff, they just transpose and copy. I
don't even think I want to change that, because it is *clear* what is going
on. Interfacing to fortran is all about memory layout, nothing more or less.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ie...>]

>
> I'm not talking about the keyword in the ravel call, I'm talking about 
> the flag in a.
Ah.  Yes, I see.  I misunderstood.  Of course ravel ignores the FORTRAN 
flag (actually it doesn't because if a copy is not necessary it doesn't 
make one).    The key is that the Python user doesn't need to care about 
the array flag unless they are interfacing to compiled code.  That's the 
point of the flag.  It's actually redundant because it could be checked 
every time it's needed.  But, right now, it's kept updated so that the 
check is simple.   The same is true with the C-CONTIGUOUS flag (called 
contiguous).

> The question is: do we *need* a fortran flag.
No, you don't *need* the flag.  But, it saves copying data to check it 
(look how many times ISFORTRAN is called in the code).  Without the flag 
all of those cases would need to do a strides-check which is done in the 
UpdateFlags code.
> I am argueing not, because the only need is for fortran contiguous 
> arrays to pass to fortran function, or translation from fortran 
> contiguous arrays to numpy arrays. What I am saying is that things are 
> unnecessarily complicated.
I disagree.  It's actually not that complicated.  The FORTRAN flag gives 
us a lot more flexibility when it comes to copying data or not.  I think 
part of the complication is that you are misunderstanding some of the 
terms and the purposes of the keywords.
> None of the LaPack stuff seems to use the Fortran stuff, they just 
> transpose and copy.
It doesn't now only because I haven't had time to go through and change 
it, but it should.   Look at scipy's LaPack interface.  It (through 
f2py) uses the FORTRAN stuff extensively (much was borrowed from there 
in the first place).

-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Tim Hochberg <tim...@ie...> wrote:
>
> Charles R Harris wrote:
>
> [SNIP]
> >
> > I'm not talking about the keyword in the ravel call, I'm talking about
> > the flag in a. The question is: do we *need* a fortran flag. I am
> > argueing not, because the only need is for fortran contiguous arrays
> > to pass to fortran function, or translation from fortran contiguous
> > arrays to numpy arrays. What I am saying is that things are
> > unnecessarily complicated. None of the LaPack stuff seems to use the
> > Fortran stuff, they just transpose and copy. I don't even think I want
> > to change that, because it is *clear* what is going on. Interfacing to
> > fortran is all about memory layout, nothing more or less.
> >
>
> Chuck,
>
> There are two things here. One is the order keyword and one is the
> FORTRAN flag. The latter is mainly an optimization for use at the
> C-level so that one doesn't have to check whether a given array is in
> contiguous FORTRAN order by examining the strides, in the same way that
> the CONTIGUOUS flag allows you to skip examining the strides when you
> need a contiguous C-order matrix.


That sounds like the two flags should be named f-contiguous and
c-contiguous. Then they would be orthogonal and one could have all four
combinations. Is that the case now? Perhaps I am misunderstanding the
meaning of the flags.


I believe it is the former that you
> are objecting to, but it would help if you could specify whether you are
> talking about the order keyword or whether you are talking about the
> FORTRAN flag.


Both. I was argueing against the FORTRAN flag, and of limiting the order
keyword to those cases where f or c contiguous arrays were the output or
input.

I'll also note that the order keyword could probably have been used to
> fix a performance problem someone was having a few weeks ago. We ended
> up transposing the data, but the individual felt that obscured the
> intent of the algorithm. I believe the same effect could probably have
> been been achieved without re jiggering the algorithm by using the order
> parameter.


Some more details would be helpful. It would be good to know what problem
the order keyword should solve.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

Travis,

On 10/18/06, Travis Oliphant <oli...@ie...> wrote:
>
> Tim Hochberg wrote:
> > One thing that may be confusing the issue is that, as I understand it,
> > FORTRAN and CONTIGUOUS together represent three states which I'll call
> > FORTRAN_ORDER, C_ORDER and DISCONTIGUOUS.
>
> Yep,  that's what they mean.  CONTIGUOUS is the name Numeric gave it and
> it meant C-order contiguous.  We have kept the same meaning.  All we've
> done is selected out from the class of arrays that Numeric called
> DISTCONTIGUOUS, arrays that are FORTRAN-order (and so still
> single-segment), but discontiguous in the sense that Numeric had.
>
> > I periodically wonder if it
> > would be valuable to have a way to query the order directly: the result
> > would be "C", "F" or None, just like the order keyword that is passed
> > in.
> You an do it with the flags
>
> a.flags.contiguous
> a.flags.fortran
>
> Discontiguous is when both of these are false.  Note that for a.ndim <
> 2, both a.flags.contiguous and a.flags.fortran are true if one of them
> is true.
>
> This is all explained in the first chapters of my book.  You have to
> understand CONTIGUOUS == C-order contiguous and FORTRAN == Fortran-order
> contiguous.


Could we make a few changes ;)

For printing the flags I would suggest using C-Contiguous and F-Contiguous
so folks don't have to read the book. And at the c level define alternates,
i.e, #define c-contiguous contiguous or whatever. That way backward
compatibility would be maintained but more descriptive names would be
available.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Travis Oliphant <oli...@ee...> wrote:
>
> Charles R Harris wrote:
>
> >
> > Could we make a few changes ;)
> >
> > For printing the flags I would suggest using C-Contiguous and
> > F-Contiguous so folks don't have to read the book. And at the c level
> > define alternates, i.e, #define c-contiguous contiguous or whatever.
> > That way backward compatibility would be maintained but more
> > descriptive names would be available.
>
> Printing the flags is not intended for the casual user.  So, I'd like to
> keep consistent with C-level names and the names that are printed.
>
> CONTIGUOUS is the old name Numeric used.  It always meant C-CONTIGUOUS
> and so that meaning is preserved.  FORTRAN is the new one flag and it
> means FORTRAN CONTIGUOUS.
>
> So, you want something like?
>
> #define NPY_C_CONTIGUOUS NPY_CONTIGUOUS
> #define NPY_F_CONTIGUOUS NPY_FORTRAN
>
> and to have C_CONTIGUOUS and F_CONTIGUOUS print for the flags description?


Yes, I think that would be more informative.

I'm not opposed to it, but I don't really see the need.  It's just a
> semantic question.  Given the history of CONTIGUOUS in Numeric I thought
> it was clear that CONTIGUOUS always meant C-contiguous.


Well, I knew that for numeric, but it was a good deal less obvious in combo
with the order keyword.  For instance, contiguous could change its meaning
to match up with FORTRAN, so that FORTRAN=True and CONTIGUOUS=True meant
Fortran contiguous, which was sort of what I was thinking. Explicit never
hurts.

Chuck

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ee...>]

Charles R Harris wrote:

> Well, I knew that for numeric, but it was a good deal less obvious in 
> combo with the order keyword.  For instance, contiguous could change 
> its meaning to match up with FORTRAN, so that FORTRAN=True and 
> CONTIGUOUS=True meant Fortran contiguous, which was sort of what I was 
> thinking. Explicit never hurts.


Ahh, so you were confused by looking at flags for 1-d arrays (where 
indeed both CONTIGUOUS and FORTRAN can be True --- 1-d arrays that are 
CONTIGUOUS are also FORTRAN CONTIGUOUS).

-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Travis O. <oli...@ee...>]

Charles R Harris wrote:

>
>
> On 10/18/06, *Travis Oliphant* <oli...@ee... 
> <mailto:oli...@ee...>> wrote:
>
>     Charles R Harris wrote:
>
>     > Well, I knew that for numeric, but it was a good deal less
>     obvious in
>     > combo with the order keyword.  For instance, contiguous could change
>     > its meaning to match up with FORTRAN, so that FORTRAN=True and
>     > CONTIGUOUS=True meant Fortran contiguous, which was sort of what
>     I was
>     > thinking. Explicit never hurts.
>
>
>     Ahh, so you were confused by looking at flags for 1-d arrays (where
>
>
> More confused by what seemed to make sense to me. Tim was the one who 
> actually ran the experiment to see what was going on, and he wasn't 
> sure what FORTRAN meant either. Now if I had accessed the full set of 
> offsets, strides, and counts, it would all have become clear. Numpy 
> tries to hide the nasty details, which is good until you really have 
> to know what it happening underneath.

I've done as you requested.  I've added F_CONTIGUOUS as an alias to 
FORTRAN and C_CONTIGUOUS as an alias to CONTIGUOUS.  These are the names 
that show up when you print the flags and you can use them whenever you 
used the other names.  The other names are still everywhere in the code, 
but perhaps the existence of these aliases will help people understand 
what is meant better.

-Travis

Re: [Numpy-discussion] The NumPy Fortran-ordering quiz

[Charles R H. <cha...@gm...>]

On 10/18/06, Travis Oliphant <oli...@ee...> wrote:
>
> Charles R Harris wrote:
>
> >
> >
> > On 10/18/06, *Travis Oliphant* <oli...@ee...
> > <mailto:oli...@ee...>> wrote:
> >
> >     Charles R Harris wrote:
> >
> >     > Well, I knew that for numeric, but it was a good deal less
> >     obvious in
> >     > combo with the order keyword.  For instance, contiguous could
> change
> >     > its meaning to match up with FORTRAN, so that FORTRAN=True and
> >     > CONTIGUOUS=True meant Fortran contiguous, which was sort of what
> >     I was
> >     > thinking. Explicit never hurts.
> >
> >
> >     Ahh, so you were confused by looking at flags for 1-d arrays (where
> >
> >
> > More confused by what seemed to make sense to me. Tim was the one who
> > actually ran the experiment to see what was going on, and he wasn't
> > sure what FORTRAN meant either. Now if I had accessed the full set of
> > offsets, strides, and counts, it would all have become clear. Numpy
> > tries to hide the nasty details, which is good until you really have
> > to know what it happening underneath.
>
> I've done as you requested.  I've added F_CONTIGUOUS as an alias to
> FORTRAN and C_CONTIGUOUS as an alias to CONTIGUOUS.  These are the names
> that show up when you print the flags and you can use them whenever you
> used the other names.  The other names are still everywhere in the code,
> but perhaps the existence of these aliases will help people understand
> what is meant better.


Thanks, Travis. I really appreciate your willingness to make such
modifications.

Chuck