numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[Eric M. <e.m...@po...>]

On Wednesday 24 July 2002 17:59, Chris Barker wrote:
> Just to add my $.02:
>
> I disagree with Eric about what the default behaviour should be. Every
> programming language/environment I have ever used uses some kind of
> "random" seed by default. When I want reproducible results (which I
> often do for testing) I can specify a seed. I find the the most useful
> behaviour. As Eric points out, it is not trivial to generate a "random"
> seed (from the time, or whatever), so it doesn't make sense to burdon
> the nieve user with this chore.
>
> Therefore, I strongly support keeping the default behaviour of a
> "random" seed.

In that case, and if that is the general consensus, RNG should be adapted=
:
it now uses a fixed seed by default (and not a clock generated one).

> Eric Maryniak wrote:
> > then I'd urge at least to use a better initial seed.
> > In certain applications, e.g. generating session id's in crypto progr=
ams,
> > non-predictability of initial seeds is crucial. But if you have a loo=
k
> > at GPG's or OpenSSL's source for a PRNG (especially for Windows), it
> > looks like an art in itself. So perhaps RNG's 'clock code' should rep=
lace
> > RandomArray2's: it uses microseconds (in gettimeofday), too, and thus
> > will not have the 1-second problem.
>
> This I agree with: a better default initial seed would be great. As
> someone said, "show me the code!". I don't imagine anyone would object
> to improving this.

The source is in Mixranf(), file Numerical/Packages/RNG/Src/ranf.c
(when checked out with CVS), but it may be a good idea to check it
with Python's own random/whrandom code (which I don't have at hand
-- it may be more recent and/or portable for other OSes).

By the way, I realized in my code 'fix' for RandomArray2.seed(x=3DNone,y=3D=
None)
that I already anticipated this and that the default behavior is _not_
to use a fixed seed ;-)  :

  if either the x or y seed:

    seed  < 0     =3D=3D>  Use the default initial seed value.
    seed  =3D None  =3D=3D>  Set a "random" value for the seed from clock=
 (default)
    seeds >=3D 0    =3D=3D>  Set seed directly (32 bits only).

and en-passant do a better job than clock-based seeding:

---cut---
def seed(x=3DNone,y=3DNone):
    """seed(x, y), set the seed using the integers x, y;
    ...
    """
    if (x !=3D None and type (x) !=3D IntType) or
       (y !=3D None and type (y) !=3D IntType) :
        raise ArgumentError, "seed requires integer arguments (or None)."
    if x =3D=3D None or y =3D=3D None:
        # This would be the best, but is problematic under Windows/Mac.
        import dev_random_device  # uses /dev/random or equivalent
        x =3D dev_random_device.nextvalue()   # egd.sf.net is a user spac=
e
        y =3D dev_random_device.nextvalue()   # alternative
        # So best is to use Mixranf() from RNG/Src/ranf.c here.
    elif x < 0 or y < 0:
        x =3D 1234567890L
        y =3D 123456789L
    ranlib.set_seeds(x,y)
---cut---

Bye-bye,

Eric
--=20
Eric Maryniak <e.m...@po...>
WWW homepage: http://pobox.com/~e.maryniak/
Mobile phone: +31 6 52047532, or (06) 520 475 32 in NL.

Unix was a trademark of AT&T.
AT&T is a modem test command.

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[Chris B. <Chr...@no...>]

Just to add my $.02:

I disagree with Eric about what the default behaviour should be. Every
programming language/environment I have ever used uses some kind of
"random" seed by default. When I want reproducible results (which I
often do for testing) I can specify a seed. I find the the most useful
behaviour. As Eric points out, it is not trivial to generate a "random"
seed (from the time, or whatever), so it doesn't make sense to burdon
the nieve user with this chore.

Therefore, I strongly support keeping the default behaviour of a
"random" seed.

Eric Maryniak wrote:
> then I'd urge at least to use a better initial seed.
> In certain applications, e.g. generating session id's in crypto programs,
> non-predictability of initial seeds is crucial. But if you have a look
> at GPG's or OpenSSL's source for a PRNG (especially for Windows), it looks
> like an art in itself. So perhaps RNG's 'clock code' should replace
> RandomArray2's: it uses microseconds (in gettimeofday), too, and thus will
> not have the 1-second problem.

This I agree with: a better default initial seed would be great. As
someone said, "show me the code!". I don't imagine anyone would object
to improving this.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[Eric M. <e.m...@po...>]

On Tuesday 23 July 2002 22:15, pa...@pf... wrote:
> RandomArray got a "special" position as part of Numeric simply by
> historical accident in being there first. I think in the conversion to
> Numarray we will be able to remove such things from the "core" and make
> more of a marketplace of equals for the "addons". As it is now there is
> some implication that somehow one is "better" than the other, which is
> unjustified either mathematically or in the sense of design.
>
> RNG's design is based on my experience with large codes needing many
> independent streams. The mathematics is from a well-tested Cray algorit=
hm.
> I'm sure it could use fluffing up but a good case can be made for it.

A famous quote from Linus is "Nice idea. Now show me the code."

Perhaps a detailed example makes my problem clearer, because as it is now=
,
RNG and RandomArray2 are not orthogonal in design, in the sense that RNG'=
s
default seed is fixed and RandomArray's is automagical (clock), not
reproducible and mathematically suspect, which I think is not good for th=
e
more naive Python user.

Below I will give intended usage in a provocative way, but please don't
take me too seriously (I know, I don't ;-)

Let's say you have a master shell script that runs a neural net paradigm
(size 20x20) 10 times, each time with the same parameters, to see if it's
stable or chaotic, i.e. does not 'converge' c.q. outcome depends on initi=
al
values (it should not be chaotic, but this should always be checked).

    run10.sh
       tracelink.py 20 20 inputpat.dat > hippocamp01.out
       ... 8 more ...
       tracelink.py 20 20 inputpat.dat > hippocamp10.out

    tracelink.py
       ... import numarray, RandomArray2 _or_ RNG ...
       # Case 1: RandomArray2
       # User uses default clock seed, which is the same
       # during 1 second (see my previous posting).
       # ignlgi(void)'s seeds 1234567890L,123456789L
       # are _not_ used (see com.c).
       RandomArray2.seed()
       # But if omitted, RandomArray2.py does it, too.
       ... calculations
       ... other program outcome _only_ if program runs > 1 second,
       ... otherwise the others will have the same result.
       # Case 2: RNG
       # A 'standard_generator =3D CreateGenerator(-1)' is automatically =
done.
       #   seed < 0  =3D=3D>  Use the default initial seed value.
       #   seed =3D 0  =3D=3D>  Set a "random" value for the seed from sy=
stem clock.
       #   seed > 0  =3D=3D>  Set seed directly (32 bits only).
       # Thus, the fixed seeds used are 0,0 (see Mixranf() in ranf.c).
       ... calculations
       ... all 10 programs have the same outcome when using ranf(),
       ... because it always starts the same seed, the sequence is always=
:
       ... 0.58011364857958725, 0.95051273498076583, 0.78637142533060356 =
etc.
      =20
The problem with RandomArray's seed is, that it is not truly random itsel=
f.
In it's current (time.time based) implementation it is linearly auto
incrementing every second, and therefore suffers from auto-correlation.
Moreover, in the above example, if 10 separate .py runs complete in 1 sec=
ond
they'll all have the same seed (and outcome). This is not what the user,
if accustomed to clock seeding, would expect.
But if the seed is different each time, a problem is that runs are not
reproducible. Let's say that run hippocamp06.out produced some strange
output: now unless the user saved the seed (with get_seed), it can never
be reproduced.

Therefore, I think RNG's design is better and should be applied to
RandomArray2, too, because RandomArray2's seeding is flawed anyways.
A user should be aware of proper seeding, agreed, and now will be:
when doing multiple identical runs, the same (and thus reproducible)
output will result and so the user is made aware of the fact that,
as an example, he or she should seed or pickle it between runs.
So my suggestion would be to re-implement RandomArray2.seed(x=3D0,y=3D0)
as follows:

  if either the x or y seed:

    seed  < 0     =3D=3D>  Use the default initial seed value.
    seed  =3D None  =3D=3D>  Set a "random" value for the seed from the s=
ystem clock.
    seeds >=3D 0    =3D=3D>  Set seed directly (32 bits only).

and en-passant do a better job than clock-based seeding:

---cut---
def seed(x=3DNone,y=3DNone):
    """seed(x, y), set the seed using the integers x, y;
    ...
    """
    if (x !=3D None and type (x) !=3D IntType) or
       (y !=3D None and type (y) !=3D IntType) :
        raise ArgumentError, "seed requires integer arguments (or None)."
    if x =3D=3D None or y =3D=3D None:
        import dev_random_device  # uses /dev/random or equivalent
        x =3D dev_random_device.nextvalue()   # egd.sf.net is a user spac=
e
        y =3D dev_random_device.nextvalue()   # alternative
    elif x < 0 or y < 0:
        x =3D 1234567890L
        y =3D 123456789L
    ranlib.set_seeds(x,y)
---cut---

But: I realize that this is different behavior from Python's standard
random and whrandom, where no arg or None uses the clock. But, if that
behavior is kept for RandomArray2 (and RNG should then be adapted, too)
then I'd urge at least to use a better initial seed.
In certain applications, e.g. generating session id's in crypto programs,
non-predictability of initial seeds is crucial. But if you have a look
at GPG's or OpenSSL's source for a PRNG (especially for Windows), it look=
s
like an art in itself. So perhaps RNG's 'clock code' should replace
RandomArray2's: it uses microseconds (in gettimeofday), too, and thus wil=
l
not have the 1-second problem.

Bye-bye, Eric
--=20
Eric Maryniak <e.m...@po...>
WWW homepage: http://pobox.com/~e.maryniak/
Mobile phone: +31 6 52047532, or (06) 520 475 32 in NL.

Just because you're not paranoid, that doesn't mean that they're not
after you.

Re: [Numpy-discussion] Bug in Numpy FFT reference?

[Paul D. <du...@ll...>]

The person who wrote the manual cut and pasted from running the code. I
think there was a bug in FFT at the time. (:->

On Tue, 2002-07-23 at 14:23, Gary Bishop wrote:
> The example given for real_fft in the FFT section of the Sept 7, 2001 
> Numpy manual makes no sense to me. The text says
> 
> >>> x = cos(arange(30.0)/30.0*2*pi)
> >>> print real_fft(x)
> [ -1. +0.j 13.69406641+2.91076367j
> -0.91354546-0.40673664j -0.80901699-0.58778525j
> -0.66913061-0.74314483j -0.5 -0.8660254j
> -0.30901699-0.95105652j -0.10452846-0.9945219j
> 0.10452846-0.9945219j 0.30901699-0.95105652j
> 0.5 -0.8660254j 0.66913061-0.74314483j
> 0.80901699-0.58778525j 0.91354546-0.40673664j
> 0.9781476 -0.20791169j 1. +0.j ]
> 
> But surely x is a single cycle of a cosine wave and should have a very 
> sensible and simple FT. Namely [0, 1, 0, 0, 0, ...]
> 
> Indeed, running the example using Numeric and FFT produces, within 
> rounding error, exactly what I would expect.
> 
> Why the non-intuitive (and wrong) result in the example text?
> 
> gb
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] Bug in Numpy FFT reference?

[Gary B. <gb...@cs...>]

The example given for real_fft in the FFT section of the Sept 7, 2001 
Numpy manual makes no sense to me. The text says

>>> x = cos(arange(30.0)/30.0*2*pi)
>>> print real_fft(x)
[ -1. +0.j 13.69406641+2.91076367j
-0.91354546-0.40673664j -0.80901699-0.58778525j
-0.66913061-0.74314483j -0.5 -0.8660254j
-0.30901699-0.95105652j -0.10452846-0.9945219j
0.10452846-0.9945219j 0.30901699-0.95105652j
0.5 -0.8660254j 0.66913061-0.74314483j
0.80901699-0.58778525j 0.91354546-0.40673664j
0.9781476 -0.20791169j 1. +0.j ]

But surely x is a single cycle of a cosine wave and should have a very 
sensible and simple FT. Namely [0, 1, 0, 0, 0, ...]

Indeed, running the example using Numeric and FFT produces, within 
rounding error, exactly what I would expect.

Why the non-intuitive (and wrong) result in the example text?

gb

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[<pa...@pf...>]

RandomArray got a "special" position as part of Numeric simply by histori=
cal
accident in being there first. I think in the conversion to Numarray we
will be able to remove such things from the "core" and make more of a mar=
ketplace
of equals for the "addons". As it is now there is some implication that
somehow one is "better" than the other, which is unjustified either mathe=
matically
or in the sense of design.

RNG's design is based on my experience with large codes needing many inde=
pendent
streams. The mathematics is from a well-tested Cray algorithm. I'm sure
it could use fluffing up but a good case can be made for it.

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[Eric M. <e.m...@po...>]

On Tuesday 23 July 2002 20:54, Todd Miller wrote:
> Eric Maryniak wrote:
> >...
> That should not, however, discourage you from writing a new and improve=
d
> random number package for numarray.

Yes, thank you :-)

> >...
> >3. I wonder what the design philosophy is behind the decision
> >   to use 'mathematically suspect' seeding as default behavior.
>
> Using time for a seed is fairly common.   Since it's an implementation
> detail,  I doubt anyone would object if you can suggest a better defaul=
t
> seed.

Well, as said, a fixed seed, provided by the class implementation
and therefore 'good', instead of a not-so-random 'random' seed.
And imho it would be better not to (only) use the clock, but a /dev/rando=
m
kinda thing.
Personally, I find the RNG setup much more appealing: there the default i=
s:

    standard_generator =3D CreateGenerator(-1)

where

    seed < 0  =3D=3D>  Use the default initial seed value.
    seed =3D 0  =3D=3D>  Set a "random" value for the seed from the syste=
m clock.
    seed > 0  =3D=3D>  Set seed directly (32 bits only).

And indeed 'void Mixranf(int *s,u32 s48[2])' uses a built-in constant
as initial seed value (actually, two).

>...
> >     If you use default seed()'ing now and re-run your program/model
> >     later with identical parameters, you will get different output.
>
> When you care about this, you need to set the seed to something
> deterministic.

Naturally, but how do I know what a 'good' seed is (or indeed it's type,
range, etc.)? I just would like, as e.g. RNG does, let the number generat=
or
take care of this... (or at least provide the option to)

>...

In the programs I've seen so far, including a lot of ours ahem, usually a
program (simulation) is run multiple times with the same parameters and,
in our case for neural nets, seeded each time with a clock generated seed
and then the different simulations are compared and checked if they are
similar or sensitive to chaotic influences.
But I don't think this is the proper way to do this.
My point is, I guess, that the sequence of these clock-generated seeds
itself is not random, because (as for RandomArray) the generated numbers
are clearly not random.
Better, and reproducible, would be to start the first simulation with a
supplied seed, get the seed and pickle after the first run and use the
pickled seed for run 2 etc. or indeed have a kind of master script
(as you suggest) that manages this.
That way you would start with one seed only and are not re-seeding
for each run. Because if the clock-seeds are not truly random, you
will a much greater change of cycles in your overall sequence of numbers.

Bye-bye, Eric
--=20
Eric Maryniak <e.m...@po...>
WWW homepage: http://pobox.com/~e.maryniak/
Mobile phone: +31 6 52047532, or (06) 520 475 32 in NL.

VME ERROR 37022: Hierarchic name syntax invalid taking
into account starting points defined by initial context.

Re: [Numpy-discussion] Numarray: question on RandomArray2.seed(x=0, y=0) system clock default and possible bug

[Todd M. <jm...@st...>]

Eric Maryniak wrote:

>Dear crunchers,
>
>According to the _Numpy_ manual for RandomArray.seed(x=0, y=0)
>(with /my/ emphasis):
>
>  The seed() function takes two integers and sets the two seeds
>  of the random number generator to those values. If the default
>  values of 0 are used for /both/ x and y, then a seed is generated
>  from the current time, providing a /pseudo-random/ seed.
>
>Note: in numarray, the RandomArray2 package is provided but it's
>description is not (yet) included in the numarray manual.
>
>I have some questions about this:
>
>1. The implementation of seed(), which is, by the way, identical
>   both in Numeric's RandomArray.py and numarray's RandomArray2.py
>   seems to contradict it's usage description:
>
The 2 in RandomArray2 is there to support side-by-side testing with 
Numeric, not to imply something new and improved.  The point of 
providing RandomArray2 is to provide a migration path for current 
Numeric users.  To that end, RandomArray2 should be functionally 
identical to RandomArray.  

That should not, however, discourage you from writing a new and improved 
random number package for numarray.

>
>
>---cut---
>def seed(x=0,y=0):
>    """seed(x, y), set the seed using the integers x, y;
>    Set a random one from clock if  y == 0
>    """
>    if type (x) != IntType or type (y) != IntType :
>        raise ArgumentError, "seed requires integer arguments."
>    if y == 0:
>        import time
>        t = time.time()
>        ndigits = int(math.log10(t))
>        base = 10**(ndigits/2)
>        x = int(t/base)
>        y = 1 + int(t%base)
>    ranlib.set_seeds(x,y)
>---cut---
>
>   Shouldn't the second 'if' be:
>
>    if x == 0 and y == 0:
>
>  With the current implementation:
>
>  - 'seed(3)' will actually use the clock for seeding
>  - it is impossible to specify 0's (0,0) as seed: it might be
>    better to use None as default values?
>
>2. With the current time.time() based default seeding, I wonder
>   if you can call that, from a mathematical point of view,
>   pseudo-random:
>
>---cut---
>$ python
>Python 2.2.1 (#1, Jun 25 2002, 20:45:02)
>[GCC 2.95.3 20010315 (SuSE)] on linux2
>Type "help", "copyright", "credits" or "license" for more information.
>
>>>>from numarray import *
>>>>from RandomArray2 import *
>>>>import time
>>>>numarray.__version__
>>>>
>'0.3.5'
>
>>>>for i in range(5):
>>>>
>...     time.time()
>...     RandomArray2.seed()
>...     RandomArray2.get_seed()
>...     time.sleep(1)
>...     print
>...
>1027434978.406238
>(102743, 4979)
>
>1027434979.400319
>(102743, 4980)
>
>1027434980.400316
>(102743, 4981)
>
>1027434981.40031
>(102743, 4982)
>
>1027434982.400308
>(102743, 4983)
>---cut---
>
>   It is incremental, and if you use default seeding within
>   one (1) second, you get the same seed:
>
>---cut---
>
>>>>for i in range(5):
>>>>
>...     time.time()
>...     RandomArray2.seed()
>...     RandomArray2.get_seed()
>...     time.sleep(0.1)
>...     print
>...
>1027436537.066677
>(102743, 6538)
>
>1027436537.160303
>(102743, 6538)
>
>1027436537.260363
>(102743, 6538)
>
>1027436537.360299
>(102743, 6538)
>
>1027436537.460363
>(102743, 6538)
>---cut---
>
>3. I wonder what the design philosophy is behind the decision
>   to use 'mathematically suspect' seeding as default behavior.
>
Using time for a seed is fairly common.   Since it's an implementation 
detail,  I doubt anyone would object if you can suggest a better default 
seed.

>
>   Apart from the fact that it can hardly be called 'random', I also
>   have the following problems with it:
>
>   - The RandomArray2 module initializes with 'seed()' itself, too.
>     Reload()'s of RandomArray2, which might occur outside the
>     control of the user, will thus override explicit user's seeding.
>     Or am I seeing ghosts here?
>
Overriding a user's explicit seed as a result of a reload sounds correct 
to me.   All of the module's top level statements are re-executed during 
a reload.

>
>   - When doing repeated run's of one's neural net simulations that
>     each take less than a second, one will get identical streams of
>     random numbers, despite seed()'ing each time.
>     Not quite what you would expect or want.
>
This is easy enough to work around: don't seed or re-seed.     If you 
then need to make multiple simulation runs, make a separate module and 
call your simulation like:

import simulation

RandomArray2.seed(something_deterministic, something_else_deterministic)
for i in range(number_of_runs):
    simulation.main()

>
>   - From a purist software engineering point of view, I don't think
>     automagical default behavior is desirable: one wants programs to
>     be deterministic and produce reproducible behavior/output.
>
I don't know.  I think by default,  random numbers *should be* random.

>
>     If you use default seed()'ing now and re-run your program/model
>     later with identical parameters, you will get different output.
>
When you care about this, you need to set the seed to something 
deterministic.

>
>     In Eiffel, object attributes are always initialized, and you will
>     almost never have irreproducible runs. I found that this is a good
>     thing for reproducing ones bugs, too ;-)
>
This sounds like a good design principle, but I don't see anything in 
RandomArray2 which is keeping you from doing this now.

>
>To summarize, my recommendation would be to use None default arguments
>and use, when no user arguments are supplied, a hard (built-in) seed
>tuple, like (1,1) or whatever.
>
Unless there is a general outcry from the rest of the community,  I 
think the (existing) numarray extensions (RandomArray2, LinearAlgebra2, 
FFT2) should try to stay functionally identical with Numeric.

>
>Sometimes a paper on a random number generator suggests seeds (like 4357
>for the MersenneTwister), but of course, a good random number generator
>should behave well independently of the initial seed/seed-tuple.
>I may be completely mistaken here (I'm not an expert on random number
>theory), but the random number generators (Ahrens, et. al) seem 'old'?
>After some studying, we decided to use the Mersenne Twister:
>
An array enabled version might make a good add-on package for numarray.

>
>
>    http://www-personal.engin.umich.edu/~wagnerr/MersenneTwister.html
>    http://www.math.keio.ac.jp/~matumoto/emt.html
>
>PDF article:
>
>    http://www.math.keio.ac.jp/~nisimura/random/doc/mt.pdf
>
>    M. Matsumoto and T. Nishimura,
>        "Mersenne Twister: A 623-dimensionally equidistributed uniform
>         pseudorandom number generator",
>         ACM Trans. on Modeling and Computer Simulation Vol. 8, No. 1,
>         January pp.3-30 1998
>
>There are some Python wrappers and it has good performance as well.
>
>Bye-bye,
>
>Eric
>
Bye,
Todd

[Numpy-discussion] Summary to NumPy on Mac OS 10.1.5

[Thomas B. <bie...@ya...>]

Cc: Jack Jansen <Jac...@or...>, Greg Welch <we...@cs...>, 
    
                                  Kelvin Chu <kel...@uv...>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Dear All,

thank you for the swift responses and the help!
NumPy now works for me on OSX. Let me summarize:

1. Python 2.2.1 compiles without errors.

2. NumPy 20 does, too - special thanks to Jack!

3. Fink does not have a good UI, I dont't use it.

4. gcc is a hassle to get.

The next step ;-) is now to compile gnuplot in Darwin
which fails. I do have a running version that works
through the OSX interface but do not know how to call
this from the the terminal. If someone has any hints,
I would appreciate an e-mail message. Thanks again!

Best regards,
Thomas

__________________________________________________
Do You Yahoo!?
Yahoo! Health - Feel better, live better
http://health.yahoo.com

Re: [Numpy-discussion] Numarray design announcement

[Pete S. <pe...@sh...>]

Paul F Dubois wrote:
 > Numarray's Design
 > Paul F. Dubois and Perry Greenfield

a very nice design, for a lot of challenging decisions


 > Numarray's current view of arrays in C, using either native or
 > emulation C-APIs, is that array data can be mutated, but array
 > properties cannot. Thus, an existing Numeric extension function
 > which tries to change the shape or strides of an array in C is
 > more of a porting challenge, possibly requiring a python wrapper.

i have a c extension that does this, but only during "creation time" 
of the array. i'm hoping there can be some way to do this from C. i 
need to create a new array from a block of numbers that aren't 
contiguous...

/* roughly snipped code */
dim[0] = myimg->w;
dim[1] = myimg->h;
dim[2] = 3; /*r,g,b*/
array = PyArray_FromDimsAndData(3, dim, PyArray_UBYTE, startpixel);
array->flags = OWN_DIMENSIONS|OWN_STRIDES;
array->strides[2] = pixelstep;
array->strides[1] = myimg->pitch;
array->strides[0] = myimg->format->BytesPerPixel;
array->base = myimg_object;


note this data is image data, and i am "reorienting" it so that the 
first index is X and the second index is Y. plus i need to account 
for an image pitch, where the rows are not exactly the same width as 
the number of pixels.

also, i am also changing the "base" field, since the data for this 
array lives inside another image object

of course, once the array is created, i pass it off to the user and 
never touch these fields again, so perhaps something like this will 
work in the new numarray?

if not, i'm eager to start my petition for a 
"PyArray_FromDimsAndDataAndStrides" function, and also a way to 
assign the "base" as well.


i'm looking forward to the new numarray, looks very exciting.

[Numpy-discussion] Numarray design announcement

[Paul F D. <pa...@pf...>]

At numpy.sf.net you will find a posting from Perry Greenfield and I
detailing the design decisions we have taken with respect to Numarray.
What follows is the text of that message without the formatting. We ask
for your understanding about those decisions that differ from the ones
you might prefer. 

Numarray's Design
Paul F. Dubois and Perry Greenfield

Numarray is the new implementation of the Numeric Python extension. It
is our intention that users will change as rapidly as possible to the
new  module when we decide it is ready. The present Numeric Python team
will cease supporting Numeric after a short transition period.

During recent months there has been a lot of discussion about Numarray
and whether or not it should differ from Numeric in certain ways. We
have reviewed this lengthy discussion and come to some conclusions about
what we plan to do. The discussion has been valuable in that it took a
whole new "generation" back through the considerations that the
"founding fathers" debated when Numeric Python was designed.

There are literally tens of thousands of Numerical Python users. These
users may represent only a tiny percentage of potential users but they
are real users today with real code that they have written, and breaking
that code would represent real harm to real people. Most of the issues
discussed recently were discussed at length when Numeric was first
designed. Some decisions taken then represent a choice that was simply a
choice among valid alternatives. Nevertheless, the choice was made, and
to arbitrarily now make a different choice would be difficult to
justify.

In arguing about Python's indentation, we often see heart-felt arguments
from opponents who have sincere reasons for feeling as they do. However,
many of the pitfalls they point to do not seem to actually occur in real
life very often. We feel the same way about many arguments about Numeric
Python. The view / copy argument, for example, claims that beginners
will make errors with view semantics. Well, some do, but not very often,
and not twice.  It is just one of many differences that users need to
adapt to when learning an entity-object model such as Python's when they
are used to variable semantics such as in Fortran or C. Similarly, we do
not receive massive reports of confusion about differing default values
for the axis keyword -- there was a rationale for the way it is now, and
although one could propose a different rationale for a different choice,
it would be just a choice.

Decisions

Numarray will have the same Python interface as Numeric except for the
exceptions discussed below. 

1. The Numarray C API includes a compatibility layer consisting of some
of the members of the Numeric C API. For details on compatibility at the
C level see
http://telia.dl.sourceforge.net/sourceforge/numpy/numarray.pdf , pdf
pages 78-81. Since no formal decision was ever made about what parts of
the Numeric C header file were actually intended to be publicly
available, do not expect complete emulation. 

Numarray's current view of arrays in C, using either native or emulation
C-APIs, is that array data can be mutated, but array properties cannot.
Thus, an existing Numeric extension function which tries to change the
shape or strides of an array in C is more of a porting challenge,
possibly requiring a python wrapper. Depending on what kind of
optimization we do, this restriction might be lifted. For the Numeric
extensions already ported to Numarray (RandomArray, LinearAlgebra, FFT),
none of this was an issue.

2. Currently, if the result of an index operation x[i] results in a
scalar result, the result is converted to a similar Python type. For
example, the result of array([1,2,3])[1] is the Python integer 2. This
will be changed so that the result of an index operation on a Numarray
array is always a Numarray array. Scalar results will become rank-zero
arrays (i.e., shape () ).

3. Currently, binary operations involving Numeric arrays and Python
scalars uses the precision of the Python scalar to help determine the
precision of the result. In Numarray, the precision of the array will
have precedence in determining the precision of the outcome. Full
details are available in the Numarray documention.

4. The Numarray version of MA will no longer have copy semantics on
indexing but instead will be consistent with Numarray. (The decision to
make MA differ in this regards was due to a need for CDAT to be backward
compatible with a local variant of Numeric; the CDAT user community no
longer feels this was necessary).

Some explanation about the scalar change is in order. Currently, much
coding in Numeric-based applications must be devoted to handling the
fact that after an index operation, the programmer can not assume that
the result is an array. So, what are the consequences of change? A
rank-zero array will interact as expected with most other parts of
Python. When it does not, the most likely result is a type error. For
example, let x = array([1,2,3]). Then [1,2,3][x[0]] currently produces
the result 2. With the change, it would produce a type error unless a
change is made to the Python core (currently under discussion). But
x[x[0]] would still work because we have control of that.  In short, we
do not think this change will break much code and it will prevent the
writing of more code that is either broken or difficult to write
correctly.

Re: [Numpy-discussion] NumPy on Mac OS 10.1.5

[Jack J. <Jac...@or...>]

On vrijdag, juli 19, 2002, at 10:12 , Thomas Biesinger wrote:

> Hi,
>
> when I try to install NumPy on Mac OS X.1.5, it fails on this error:
>
> ....
> cc -bundle -undefined suppress build/temp.darwin-5.5-Power Macintosh-
> 2.1/_numpymodule.o build/temp.darwin-5.5-Power Macintosh-2.1/
> arrayobject.o build/temp.darwin-5.5-Power Macintosh-2.1/ufuncobject.o -
> o build/lib.darwin-5.5-Power Macintosh-2.1/_numpy.so
> /usr/bin/ld: -undefined error must be used when -twolevel_namespace is
> in effect

Thomas,
as of MacOSX 10.1 the link step needs either the -flat_namespace 
option, or the -bundle_loader option.

But: this has been fixed in both Python 2.2.1 and Python 2.3a0 
(the CVS tree). Are you by any chance still running Python 2.2 
(which predates OSX 10.1, and therefore two-level namespaces, 
and therefore the right linker invocations, which distutils 
reads from Python's own Makefile).

If you're running 2.2: please upgrade and try again. If you're 
running 2.2.1 or later: let me know and I'll try and think of 
what questions I should ask you to debug this:-)
--
- Jack Jansen        <Jac...@or...>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- 
Emma Goldman -

FW: [Numpy-discussion] NumPy on Mac OS 10.1.5

[Greg W. <we...@cs...>]

Thomas, I have (recently) built Numeric 21.3 on multiple OS X 10.1.5
platforms, and have had no problems that I know of. I am using Python 2.3a0
but had also built Numeric w/ earlier versions of Python too. All platforms
have the April 2002 developer tools update.

I just noticed that your compile line shows the use of cc, as opposed to
gcc. Here is the corresponding compile line for 21.3 on my powerbook (Python
2.3a0):

gcc -bundle -bundle_loader /usr/local/bin/python build/temp.darwin-5.5-Power
Macintosh-2.3/_numpymodule.o build/temp.darwin-5.5-Power
Macintosh-2.3/arrayobject.o
build/temp.darwin-5.5-PowerMacintosh-2.3/ufuncobject.o -o
build/lib.darwin-5.5-Power Macintosh-2.3/_numpy.so

--Greg

Re: [Numpy-discussion] NumPy on Mac OS 10.1.5

[rob <ro...@py...>]

Thomas Biesinger wrote:
> 
> Hi,
> 
> when I try to install NumPy on Mac OS X.1.5, it fails on this error:
> 
> ....
> cc -bundle -undefined suppress build/temp.darwin-5.5-Power Macintosh-
> 2.1/_numpymodule.o build/temp.darwin-5.5-Power Macintosh-2.1/
> arrayobject.o build/temp.darwin-5.5-Power Macintosh-2.1/ufuncobject.o -
> o build/lib.darwin-5.5-Power Macintosh-2.1/_numpy.so
> /usr/bin/ld: -undefined error must be used when -twolevel_namespace is
> in effect
> error: command 'cc' failed with exit status 1
> ~/Python/Numeric-21.3 % cc
> cc: No input files
> 
> I had thought to submit this to the developers section of the list
> but could not find the way to subscribe to it ;-)
> 
> If somehow had a running version of NumPy with for Mac OSX
> http://tony.lownds.com/macosx, I would appreciate it.
> 
> Thanks everyone for their help!
> 
> Regards,
> Thomas
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Autos - Get free new car price quotes
> http://autos.yahoo.com
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion


Hi Thomas, sorry I don't have the expertise to help you with your
question.  I am wondering if you are using one of Apple's new G4
machines?  I'm curious about the floating point performance of those 
chips.  If you ever get Numpy working, I have a routine that I use for a
benchmark, a Norton-Summerfeld ground (antenna) simulation routine that
I could send to you.  The record for me is 120s on a P4 1.8Ghz at work,
but I'm sure the new Xeons would beat that, and maybe the new Athlons. 
My 1.2Ghz DDR Athlon is much slower than the P4, but the clock speeds
are so much different.  Rob.

-- 
-----------------------------
The Numeric Python EM Project

www.pythonemproject.com

[Numpy-discussion] NumPy on Mac OS 10.1.5

[Thomas B. <bie...@ya...>]

Hi,

when I try to install NumPy on Mac OS X.1.5, it fails on this error:

....
cc -bundle -undefined suppress build/temp.darwin-5.5-Power Macintosh-
2.1/_numpymodule.o build/temp.darwin-5.5-Power Macintosh-2.1/
arrayobject.o build/temp.darwin-5.5-Power Macintosh-2.1/ufuncobject.o -
o build/lib.darwin-5.5-Power Macintosh-2.1/_numpy.so
/usr/bin/ld: -undefined error must be used when -twolevel_namespace is
in effect
error: command 'cc' failed with exit status 1
~/Python/Numeric-21.3 % cc
cc: No input files

I had thought to submit this to the developers section of the list
but could not find the way to subscribe to it ;-)

If somehow had a running version of NumPy with for Mac OSX
http://tony.lownds.com/macosx, I would appreciate it.

Thanks everyone for their help!

Regards,
Thomas



__________________________________________________
Do You Yahoo!?
Yahoo! Autos - Get free new car price quotes
http://autos.yahoo.com

[Numpy-discussion] [ANNOUNCE] Pyfort 8.0

[Paul F D. <pa...@pf...>]

Pyfort 8.0 has been released at SourceForge (sf.net/projects/pyfortran)

Version 8
     This version contains a new facility for making and installing
projects.
     Old compile lines will still work, but will produce an equivalent
.pfp
     file that you could use in the future. Included is a Tkinter-based
GUI
     editor for the project files. However, the format of the files is
simple
     and they could be edited with a text editor as well.

     There is improved support for installing Pyfort and the modules it 
     creates in a location other than inside Python. See README.

     This version does change the installation location for an
extension. 
     Therefore, you should remove the files of any previous installation

     from your Python. Yes, this is annoying. That is why we are doing
it, 
     so that we can have an "uninstall" command.

     A new "windows" subdirectory has been added, containing an example
of how to
     use Pyfort on Windows with Visual Fortran. Thanks to Reinhold
Niesner. Testing
     of, and advice about, this are needed from Windows users. The
pyfort script itself
     is also now installed as a .bat script for win32.
 
     Support for Mac OSX (Darwin) added.

Re: [Numpy-discussion] Optimization advice

[John E. <ja...@zh...>]

After getting some advice off the pygame list I think I have a pretty
good version of my influence map now. I thought someone on this list
might be interested or at least someone checking the archives.

The new and improved code is around 6-7x faster. The main gain was
obtained by converting all the array functions to slice notation and
eliminating most of the needless copying of arrays. 

The new version is attached and is much better commented. It is also
unabridged, as it was pointed out that it wasn't entirely clear what was
going on in the last (edited) version. Hopefully things are more obvious
in this one.

Anyways... I just hating leaving a thread without a conclusion. Hope
someone finds this useful.

-- 

John Eikenberry
[ja...@zh... - http://zhar.net]
______________________________________________________________
"They who can give up essential liberty to purchase a little temporary
 safety, deserve neither liberty nor safety."
                                          --B. Franklin

RE: [Numpy-discussion] Caution -- // not standard

[Paul F D. <pa...@pf...>]

Thank you for the clarification.

Unfortunately, "my" compiler vendor is the set of all compiler vendors
that users of Numeric have, and we have to restrict ourselves to what
works. I misspoke when I said it was "not standard"; I should have said,
"doesn't work everywhere".

> -----Original Message-----
> From: Andrew P. Lentvorski [mailto:bs...@ma...] 
> Sent: Monday, July 08, 2002 12:02 PM
> To: Paul Dubois
> Cc: num...@li...
> Subject: Re: [Numpy-discussion] Caution -- // not standard
> 
> 
> Actually, // is standard C99 released December 1, 1999 as 
> ISO/IEC 9899:1999.
> 
> It also has support for variable length arrays, a complex 
> number type and a bunch of *portable* stuff for getting at 
> numerical information (limits, floating-point environment) 
> rather than nasty compiler specific hacks. ( See: 
> http://std.dkuug.dk/JTC1/SC22/WG14/www/newinc9> x.htm )
> 
> Many 
> of these extensions are specifically for the 
> numerical community.
> 
> I would recommend taking up the issue of non-standards 
> compliance with your compiler vendor.
> 
> -a
> 
> On 8 Jul 2002, Paul Dubois wrote:
> 
> > I have run into several cases of this on different open-source 
> > projects, the latest being an incorrect change in Numeric's 
> > arrayobject.c: the use of // to start a comment. Many 
> contributors who 
> > work only with Linux have come to believe that this works 
> with other C 
> > compilers, which is not true. This construct comes from C++. Please 
> > avoid this construct when contributing changes or patches 
> to Numeric.
> 
> 

Re: [Numpy-discussion] Caution -- // not standard

[Andrew P. L. <bs...@ma...>]

Actually, // is standard C99 released December 1, 1999 as
ISO/IEC 9899:1999.

It also has support for variable length arrays, a complex number type and
a bunch of *portable* stuff for getting at numerical information (limits,
floating-point environment) rather than nasty compiler specific hacks.
( See: http://std.dkuug.dk/JTC1/SC22/WG14/www/newinc9x.htm )

Many of these extensions are specifically for the numerical community.

I would recommend taking up the issue of non-standards compliance with
your compiler vendor.

-a

On 8 Jul 2002, Paul Dubois wrote:

> I have run into several cases of this on different open-source projects,
> the latest being an incorrect change in Numeric's arrayobject.c: the use
> of // to start a comment. Many contributors who work only with Linux
> have come to believe that this works with other C compilers, which is
> not true. This construct comes from C++. Please avoid this construct
> when contributing changes or patches to Numeric.

[Numpy-discussion] Caution -- // not standard

[Paul D. <du...@ll...>]

I have run into several cases of this on different open-source projects,
the latest being an incorrect change in Numeric's arrayobject.c: the use
of // to start a comment. Many contributors who work only with Linux
have come to believe that this works with other C compilers, which is
not true. This construct comes from C++. Please avoid this construct
when contributing changes or patches to Numeric.

[Numpy-discussion] Optimization advice

[John E. <ja...@zh...>]

I'm working on an influence map [1] for game civil [2]. I have a working
version, but as a real numeric newbie I thought I'd bounce it off the
people here before calling it done. I'm basically looking for an easy to
understand but fast influence spreading algorithm. I've read that this
algorithm is similar to those used to predict fire spreading or heat
transfer in metal if that helps.

The attached code is setup for a hex based map and the functions to take
this into accounts (shift_hex_up,shift_hex_down) are probably the most
naive. The others being only slight modifications of those in the
life.py example. Its not really commented but its short and hopefully
should be readily understandable.

I've only included the base influence map class and its associated
functions. If you'd like a version you can run, I can send you a .tgz
setup to run in place (for *nix systems). 

Thanks in advance for any advice or opinions.


[1] An influence map is used commonly in strategic war games. It is a
simple means of capturing the areas on the game map that one side is
strong vs the other side. Read the first post in this thread for a good
description: 

    http://www.gameai.com/influ.thread.html

[2] Civil is a cross-platform, turn-based, networked strategy game,
developed using Python, PyGame and SDL--allowing players to take part in
scenarios set during the American Civil war.

    http://civil.sourceforge.net/

-- 

John Eikenberry
[ja...@zh... - http://zhar.net]
______________________________________________________________
"They who can give up essential liberty to purchase a little temporary
 safety, deserve neither liberty nor safety."
                                          --B. Franklin

[Numpy-discussion] words that must not be spoken

[Paul F D. <pa...@pf...>]

> [mailto:num...@li...] On 
> Behalf Of Eric Maryniak
In the midst of a discussion Eric wrote:
> > ...
> shouldn't Convolve, for 
> orthogonality, be named
>     Convolve2? (cuz who knows, numarray's Convolve may be backported
>     to Numeric in the future, for comparative testing etc.).

Use of the phrase "backported to Numeric" will result in your
subscription to numpy-discussion being cancelled. (:->

No backporting is ever going to happen. This is a short one-way street
or there is no purpose to travel on it. 

I am just back from Europython and had a chance to talk to a lot of
users and have some thoughts which I will share with all of you shortly.
However, since I just had to fill out a form and where it said "Date" I
looked at my watch and wrote the time 11/16, I conclude that I have jet
lag and can't trust myself to be lucid yet.

RE: [Numpy-discussion] Numarray: minor feature requests (setup.py and version info)

[Paul F D. <pa...@pf...>]

distutils installs into the python used to run the setup.py by using the
sys.exec_prefix and sys.prefix. You would not normally need to use any
option unless you are trying to install something "off to the side"
because, for example, you don't have write permission in that python's
site-packages directory.

[Numpy-discussion] Scientific Python 2.4

[Konrad H. <hi...@cn...>]

                       Scientific Python 2.4
                       ---------------------

Scientific Python is a module library for scientific computing. In
this collection you will find modules that cover basic geometry
(vectors, tensors, transformations, vector and tensor fields),
quaternions, automatic derivatives, (linear) interpolation,
polynomials, elementary statistics, nonlinear least-squares fits, unit
calculations and conversions, Fortran-compatible data formatting, 3D
visualization via VRML, two Tk widgets for simple line plots and 3D
wireframe models.

Scientific Python also contains Python interfaces to the netCDF
library (implementing a portable binary format for large arrays) and
the Message Passing Interface, the most widely used communications
library for parallel computers.

Version 2.4 of Scientific Python has just been released. In addition
to numerous small improvents and bug fixes, it contains

 - the high-level parallelization module Scientific.BSP

 - an interface to the parallelization library BSPlib
   (see www.bsp-worldwide.org for details)

 - autoregressive models for time series in Scientific.Signals.Models


The BSP parallelization module was designed to facilitate development
and testing of parallel programs. Its main features are:

 - communication can handle almost any Python object

 - deadlocks are impossible by design

 - possibility to implement distributed data classes that
   can be used transparently by parallel applications

 - an interactive parallel interpreter that can be used inside
   Emacs (and perhaps other Python development environments)
   in order to provide an interactive parallel programming
   environment

 - parallel programs run as serial monoprocessor code on any
   Python installation with no changes and usually negligeable
   loss of performance - no need to maintain a separate
   serial version

A tutorial on BSP programming with Python is available at the Web site
and included in the distribution.


For more information and for downloading, see

      http://dirac.cnrs-orleans.fr/ScientificPython

or

      http://starship.python.net/crew/hinsen/scientific.html

-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] Numarray: minor feature requests (setup.py and version info)

[Eric M. <e.m...@po...>]

On Sunday 30 June 2002 15:31, Todd Miller wrote:
> Perry Greenfield wrote:
> >...
> >>2. Because I'm running two versions of Python (because Zope
> >>   and a lot of Zope/C products depend on a particular version)
> >>   the 'development' Python is installed in /usr/local/bin
> >>   (whereas SuSE's python is in /usr/bin).
> >>   It probably wouldn't do any harm if the manual would include
> >>   a hint at the '--prefix' option and mention an alternative
> >>   Python installation like:
> >>
> >>       /usr/local/bin/python ./setup.py install --prefix=3D/usr/local
> >
> >Good idea.
>
> I'm actually surprised that this is necessary.  I was under the
> impression that the distutils pick reasonable defaults simply based on
> the python that is running.  In your case,  I would expect numarray to
> install to /usr/local/lib/pythonX.Y/site-packages without specifying an=
y
> prefix.  What happens on SuSE?

Yes, you're probably right.
On SuSE I tested it out on my own machine ('test server'), because I did
not want to do it on the production server. It run's Python 2.2.1 exclusi=
vely.
I remembered that I had to this in a previous Numeric installation, where
1.5.2 and 2.1 were running side-by-side (and at that time I also had to
install distutils manually).
So, yes, it may not be an issue (anymore) for at least recent Python's
if you call the Python explicitly like '/usr/local/bin/python ./setup.py'
and '/usr/bin/python ./setup' (on SuSE python goes to /usr/bin).

>
> >>...

Bye-bye,

Eric
--=20
Eric Maryniak <e.m...@po...>
WWW homepage: http://pobox.com/~e.maryniak/
Mobile phone: +31 6 52047532, or (06) 520 475 32 in NL.

It said 'Insert disk #3', but only two will fit.