numpy-discussion

[Numpy-discussion] package vs module

[Perry G. <pe...@st...>]

Back in December the issue of whether numarray should be a package
or set of modules came up. When I asked about the possibility
of making numarray a package (on the scipy mailing list but I
can't seem to find the thread where it was discussed), I got
only positive comments. The issue needs to be raised here also.

Is there any objection to making numarray package based?
The implications are that 3rd party modules (e.g. FFT)
will be imported as part of the package structure, i.e.,

  import numarray.FFT 

or

  from numarray.FFT import *

instead of 

  import FFT

As usual there are advantages and disadvantages. The advantages
are that we will not have name collisions with existing Numeric
modules (currently we name FFT as FFT2 for this reason). It also
potentially reduces name collision issues in general. Most feel
it is a cleaner way to organize the software (at least based on
the feedback so far).

The main disadvantages I see so far are:

1) One will either have to change import statements in old code
   to match the new style (a pain, but generally changing imports
   is not terribly difficult since they are easy to identify) or
   explicitly add the path to each 3rd party module to Python
   Path (or some equivalent).
2) If numarray were accepted into the Python Standard Library, it
   would be the first case (as far as I can tell) of a standard
   library package where we would expect to add sub modules to
   it (e.g., FFT)). Normally these would not be distributed with
   the standard library, so some general mechanism will be needed
   to allow numarray to find 3rd party packages outside of the
   Python directory structure. For example, I don't think we can
   require having people install FFT in the Standard Library 
   directory structure after Python is installed. Rather, we would
   probably have numarray look for extension modules in a standard
   named site-packages directory (or site-numarray?) or otherwise
   check a numarraypath environmental variable so that
   import numarray.FFT works properly. Perhaps others have ideas
   about how to best handle this.

Any other issues being overlooked?

Feedback?

Thanks, Perry

Re: [Numpy-discussion] package vs module

[Magnus L. H. <ma...@he...>]

Perry Greenfield <pe...@st...>:
>
> Back in December the issue of whether numarray should be a package
> or set of modules came up. When I asked about the possibility
> of making numarray a package (on the scipy mailing list but I
> can't seem to find the thread where it was discussed), I got
> only positive comments. The issue needs to be raised here also.
> 
> Is there any objection to making numarray package based?

I think this seems like a very good and natural thing to do. (Maybe
names like RandomArray2 etc. can be changed too, now... :)

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] package vs module

[Pearu P. <pe...@ce...>]

On Mon, 6 Jan 2003, Perry Greenfield wrote:

> The main disadvantages I see so far are:
> 
> 1) One will either have to change import statements in old code
>    to match the new style (a pain, but generally changing imports
>    is not terribly difficult since they are easy to identify) or
>    explicitly add the path to each 3rd party module to Python
>    Path (or some equivalent).
> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can
>    require having people install FFT in the Standard Library 
>    directory structure after Python is installed. Rather, we would
>    probably have numarray look for extension modules in a standard
>    named site-packages directory (or site-numarray?) or otherwise
>    check a numarraypath environmental variable so that
>    import numarray.FFT works properly. Perhaps others have ideas
>    about how to best handle this.
> 
> Any other issues being overlooked?

There is one, though not so critical at this point but I will raise
it anyway. In summary, I am +1 for making numarray a package.

The issue is releated to import time and memory usage: more extension
modules in a package increase both of them, even if users have no
indention to use these modules. On slower machines this may cause
inconvinieces, especially in applications that call Python multiple times
for short tasks containing numarray operation.

Let me repeat, currently this is not a problem neither with Numeric
(because it never imports its extension modules) or numarray until
numarray will contain a number of extension modules that
presumably are not small.

For a realistic example of this issue consider Scipy (as a sort of upper
bound what numarray may become one day). Scipy contains a linalg module
that is an (almost complete) wrapper to ATLAS/BLAS/LAPACK libraries and
therefore importing the corresponding extension modules can be both time
and memory consuming.  For example, importing scipy to Python may take 2-5
seconds on PII 400MHz, mainly because of loading the linalg extension
modules. This time may be annoying for small but frequent tasks.

I wish Python import mechanism would be a bit smarter or lazier in
loading extension modules that are never used...

Pearu

Re: [Numpy-discussion] package vs module

[Tim H. <tim...@ie...>]

Pearu Peterson wrote:

>On Mon, 6 Jan 2003, Perry Greenfield wrote:
>
>  
>
>>The main disadvantages I see so far are:
>>
>>1) One will either have to change import statements in old code
>>   to match the new style (a pain, but generally changing imports
>>   is not terribly difficult since they are easy to identify) or
>>   explicitly add the path to each 3rd party module to Python
>>   Path (or some equivalent).
>>2) If numarray were accepted into the Python Standard Library, it
>>   would be the first case (as far as I can tell) of a standard
>>   library package where we would expect to add sub modules to
>>   it (e.g., FFT)). Normally these would not be distributed with
>>   the standard library, so some general mechanism will be needed
>>   to allow numarray to find 3rd party packages outside of the
>>   Python directory structure. For example, I don't think we can
>>   require having people install FFT in the Standard Library 
>>   directory structure after Python is installed. Rather, we would
>>   probably have numarray look for extension modules in a standard
>>   named site-packages directory (or site-numarray?) or otherwise
>>   check a numarraypath environmental variable so that
>>   import numarray.FFT works properly. Perhaps others have ideas
>>   about how to best handle this.
>>
>>Any other issues being overlooked?
>>    
>>
>
>There is one, though not so critical at this point but I will raise
>it anyway. In summary, I am +1 for making numarray a package.
>
>The issue is releated to import time and memory usage: more extension
>modules in a package increase both of them, even if users have no
>indention to use these modules. On slower machines this may cause
>inconvinieces, especially in applications that call Python multiple times
>for short tasks containing numarray operation.
>  
>
That's not right, is it? I'm pretty certain that submodules in a package 
are not loaded until explicitly imported. I'm not sure why SciPy is 
slow, maybe the __init__ imports everything? I don't have a copy here so 
I can't check right now.

In any event I'm +1 for putting it in a package unless it interferes 
with it getting into the core. As Paul mentioned keeping it in a zip 
archive would be even cooler once that's an option.

-tim

Re: [Numpy-discussion] package vs module

[Francesc A. <fa...@op...>]

On Mon, Jan 06, 2003 at 07:29:15PM -0500, Perry Greenfield wrote:
> The main disadvantages I see so far are:
> 
> 1) One will either have to change import statements in old code
>    to match the new style (a pain, but generally changing imports
>    is not terribly difficult since they are easy to identify) or
>    explicitly add the path to each 3rd party module to Python
>    Path (or some equivalent).

I think this should be regarded as a minor annoyance compared with the
advantages of making numarray a package. In addition, the introduction of
numarray as substitute of Numeric can justify some re-code on existing
applications.

> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can
>    require having people install FFT in the Standard Library 
>    directory structure after Python is installed. Rather, we would
>    probably have numarray look for extension modules in a standard
>    named site-packages directory (or site-numarray?) or otherwise
>    check a numarraypath environmental variable so that
>    import numarray.FFT works properly. Perhaps others have ideas
>    about how to best handle this.
> 

Great. I would be glad to see a package containing numarray kernel in order
to allow aplications to use their core features, and have a mechanism to add
3rd party packages. In particular, having something similar to site-numarray
to install these packages can be quite neat. In fact, I was pondering to
include a subset of numarray in the PyTables package (it only needs the
numarray core functionality), but if this reorganization takes place, I
would not need to do that anymore.

> Any other issues being overlooked?

Yeah. In case you decide to break numarray in several modules, which would
be the granularity of the separation. My opinion goes to have a reduced core
with basic functionality (to maximize the chances to be included in the
Pyhton Standard Library, but also to allow an easy entry for people who may
wish to use this functionality) and then different, small, 3rd party
packages, but perhaps this is also the most laborious solution.

-- 
Francesc Alted                            PGP KeyID:      0x61C8C11F

Re: [Numpy-discussion] package vs module

[<pa...@pf...>]

1. I favor the package approach.

2. I don't care if FFT is numarray.FFT or numpy.FFT (i.e., in a separate
place). However, see (3).

3. Extensions built with one version of Python/numarray may not work with=

a different version. This means the safer approach is to have all addons
inside the same directory, so that you can blow away just one directory
and be sure that no 'old' packages remain. 

Some new stuff being put into Python also envisions being able to add var=
ious
zipped files to the Python path as places to be searched. Perhaps this re=
presents
a packaging opportunity. I haven't paid enough attention to be sure.

While we are on the subject of packaging, the current distribution places=

all sorts of extraneous test and installation-related files in the Lib di=
rectory.
This makes it harder to work with the source when you are new to it.

Re: [Numpy-discussion] package vs module

[Konrad H. <hi...@cn...>]

Perry Greenfield <pe...@st...> writes:

> Back in December the issue of whether numarray should be a package
> potentially reduces name collision issues in general. Most feel
> it is a cleaner way to organize the software (at least based on
> the feedback so far).

I agree. We have discussed converting NumPy into a package a few times
in the past, the major argument against it was compatibility issues.
Numarray will require some changes to import statements anyway, so
this seems the right time to make the change.

> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can

If you plan to unbundle FFT etc. from numarray, then I would prefer a
different naming scheme: numarray being just numarray, and some other
package name grouping together the other modules. That is not only a
question of installation, but also of general maintenance and of
clarity for users. I see the Python package system as a tree:
everything inside a package belongs together, is distributed together
and is maintained by the same people.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------