py2exe-users

[Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

I'm still battling with numpy/Numeric issues.  The basic
problem is that py2exe places all of Numeric's files at the
"root" of the library (directly in the 'dist' directory) rather
than in a Numeric directory (dist/Numeric).  The result is that
this changes Numeric's modules into global modules (what's
supposed to be Numeric.foobar is now just foobar) causing them
to override modules with the same name in other packages.

The solution I found in the list is to create a directory named
Numeric and move the conflicting Numeric files into it.  That
keeps them from overriding other package modules, but it also
hides them from use by the Numeric package.

In the "normal" library tree, all of Numeric's modules are in
<library>/site-packages/Numeric. Why does py2exe place them in
the library "root" directory?

Why not put them in a Numeric directory like they are in the
normal library tree?  Wouldn't that avoid all these namespace
collision problems?

-- 
Grant Edwards                   grante             Yow! I know how to do
                                  at               SPECIAL EFFECTS!!
                               visi.com            

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
> I'm still battling with numpy/Numeric issues.  The basic
> problem is that py2exe places all of Numeric's files at the
> "root" of the library (directly in the 'dist' directory) rather
> than in a Numeric directory (dist/Numeric).

It appears to do the same thing for many of numarray's files.
The .pyc files are in the correct spots, but the .pyd files are
erroneously put in the root directory instead of in the
directory where the rest of numarray's files are.

Why is py2exe putting all these files in the library's root
directory when they belong elsewhere?

-- 
Grant Edwards                   grante             Yow! I just went below the
                                  at               poverty line!
                               visi.com            

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> I'm still battling with numpy/Numeric issues.  The basic
>> problem is that py2exe places all of Numeric's files at the
>> "root" of the library (directly in the 'dist' directory) rather
>> than in a Numeric directory (dist/Numeric).
> 
> It appears to do the same thing for many of numarray's files.
> The .pyc files are in the correct spots, but the .pyd files are
> erroneously put in the root directory instead of in the
> directory where the rest of numarray's files are.
> 
> Why is py2exe putting all these files in the library's root
> directory when they belong elsewhere?
> 

See my answer to the previous post.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Grant Edwards <gr...@vi...> wrote:

> Why not put them in a Numeric directory like they are in the
> normal library tree?  Wouldn't that avoid all these namespace
> collision problems?

Rats.  I tried moving the Numeric files into a Numeric
directory and then creating a Numeric.pth file in the library root
directory (same layout as <...>/site-packages.  It didn't work:

  Traceback (most recent call last):
    File "tflowfit.py", line 4, in ?
    File "Scientific\Functions\LeastSquares.pyc", line 8, in ?
    
  ImportError: No module named Numeric

Any ideas why Numeric.pth doesn't work?
  
-- 
Grant Edwards                   grante             Yow! The PILLSBURY DOUGHBOY
                                  at               is CRYING for an END to
                               visi.com            BURT REYNOLDS movies!!

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
> 
>> Why not put them in a Numeric directory like they are in the
>> normal library tree?  Wouldn't that avoid all these namespace
>> collision problems?
> 
> Rats.  I tried moving the Numeric files into a Numeric
> directory and then creating a Numeric.pth file in the library root
> directory (same layout as <...>/site-packages.  It didn't work:
> 
>   Traceback (most recent call last):
>     File "tflowfit.py", line 4, in ?
>     File "Scientific\Functions\LeastSquares.pyc", line 8, in ?
>     
>   ImportError: No module named Numeric
> 
> Any ideas why Numeric.pth doesn't work?
>   

IIRC .pth files are processed in normal Python by the site.py file,
this mechanism isn't used in py2exe.  You could, however, add some logic
to your script that does this.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> I'm still battling with numpy/Numeric issues.  The basic
> problem is that py2exe places all of Numeric's files at the
> "root" of the library (directly in the 'dist' directory) rather
> than in a Numeric directory (dist/Numeric).

You mean .pyd's, do you?

>  The result is that
> this changes Numeric's modules into global modules (what's
> supposed to be Numeric.foobar is now just foobar) causing them
> to override modules with the same name in other packages.
> 
> The solution I found in the list is to create a directory named
> Numeric and move the conflicting Numeric files into it.  That
> keeps them from overriding other package modules, but it also
> hides them from use by the Numeric package.
> 
> In the "normal" library tree, all of Numeric's modules are in
> <library>/site-packages/Numeric. Why does py2exe place them in
> the library "root" directory?

Because people don't like deep directory trees: they want a single directory,
or, better yet, a single file.  Usually.

> Why not put them in a Numeric directory like they are in the
> normal library tree?  Wouldn't that avoid all these namespace
> collision problems?
> 

That may be.  A better approach for py2exe might be to rename them
into 'Numeric.foobar.pyd'.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Thomas Heller <th...@ct...> wrote:
> Grant Edwards schrieb:
>> I'm still battling with numpy/Numeric issues.  The basic
>> problem is that py2exe places all of Numeric's files at the
>> "root" of the library (directly in the 'dist' directory) rather
>> than in a Numeric directory (dist/Numeric).
>
> You mean .pyd's, do you?

Yes.  The problem is that py2exe will try to put multiple
different files with the same name in the library root
directory, so the package that owns the "last one written" wins
and the package the owns the first one written losses.

>> The result is that this changes Numeric's modules into global
>> modules (what's supposed to be Numeric.foobar is now just
>> foobar) causing them to override modules with the same name in
>> other packages.
>> 
>> The solution I found in the list is to create a directory
>> named Numeric and move the conflicting Numeric files into it.
>> That keeps them from overriding other package modules, but it
>> also hides them from use by the Numeric package.
>> 
>> In the "normal" library tree, all of Numeric's modules are in
>> <library>/site-packages/Numeric. Why does py2exe place them in
>> the library "root" directory?
>
> Because people don't like deep directory trees:

Why?  They don't really const anything, and there's information
in that directory structure that py2exe is throwing away.  That
information is required so that the right files get loaded by
the right packages. Throwing that information is breaking
packages.

> they want a single directory, or, better yet, a single file.
> Usually.

Wanting a shallow directory tree is fine except that I don't
think it should take precidence over having a working directory
tree.  py2exe is causing breaking in two related ways:

 1) By "deleting" when it overwrites them with a second file of
    the same name.

 2) By placing files at a different place in the search order
    than they need to be.

>> Why not put them in a Numeric directory like they are in the
>> normal library tree?  Wouldn't that avoid all these namespace
>> collision problems?
>
> That may be.  A better approach for py2exe might be to rename
> them into 'Numeric.foobar.pyd'.

How would the file that's loading the .pyd files know they have
been renamed by py2exe?

-- 
Grant Edwards                   grante             Yow! I want to read my new
                                  at               poem about pork brains and
                               visi.com            outer space ...

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>> Grant Edwards schrieb:
>>> I'm still battling with numpy/Numeric issues.  The basic
>>> problem is that py2exe places all of Numeric's files at the
>>> "root" of the library (directly in the 'dist' directory) rather
>>> than in a Numeric directory (dist/Numeric).
>>
>> You mean .pyd's, do you?
> 
> Yes.  The problem is that py2exe will try to put multiple
> different files with the same name in the library root
> directory, so the package that owns the "last one written" wins
> and the package the owns the first one written losses.
> 
>>> The result is that this changes Numeric's modules into global
>>> modules (what's supposed to be Numeric.foobar is now just
>>> foobar) causing them to override modules with the same name in
>>> other packages.
>>> 
>>> The solution I found in the list is to create a directory
>>> named Numeric and move the conflicting Numeric files into it.
>>> That keeps them from overriding other package modules, but it
>>> also hides them from use by the Numeric package.
>>> 
>>> In the "normal" library tree, all of Numeric's modules are in
>>> <library>/site-packages/Numeric. Why does py2exe place them in
>>> the library "root" directory?
>>
>> Because people don't like deep directory trees:
> 
> Why?  They don't really const anything, and there's information
> in that directory structure that py2exe is throwing away.  That
> information is required so that the right files get loaded by
> the right packages. Throwing that information is breaking
> packages.
> 
>> they want a single directory, or, better yet, a single file.
>> Usually.
> 
> Wanting a shallow directory tree is fine except that I don't
> think it should take precidence over having a working directory
> tree.

Sure.  ;-)

>  py2exe is causing breaking in two related ways:
> 
>  1) By "deleting" when it overwrites them with a second file of
>     the same name.
> 
>  2) By placing files at a different place in the search order
>     than they need to be.
> 
>>> Why not put them in a Numeric directory like they are in the
>>> normal library tree?  Wouldn't that avoid all these namespace
>>> collision problems?
>>
>> That may be.  A better approach for py2exe might be to rename
>> them into 'Numeric.foobar.pyd'.

> How would the file that's loading the .pyd files know they have
> been renamed by py2exe?
> 

Loading .pyds from the dist directory does not use the normal mechanism
anyway - the dist directory is not on sys.path (not by default).

Loading pyds works by loader modules that py2exe creates.
If you open the library.zip file with winzip or a similar utility
you'll see that it has a xxx.pyc or xxx.pyo file in it (in the correct
location relative to the package) for each xxx.pyd file that is in the
dist directory.  These xxx.py[co] files contain this code:

<loader code>
# This loader locates extension modules relative to the library.zip
# file when an archive is used (i.e., skip_archive is not used), otherwise
# it locates extension modules relative to sys.prefix.
def __load():
    import imp, os, sys
    try:
        dirname = os.path.dirname(__loader__.archive)
    except NameError:
        dirname = sys.prefix
    path = os.path.join(dirname, '%s')
    #print "py2exe extension module", __name__, "->", path
    mod = imp.load_dynamic(__name__, path)
##    mod.frozen = 1
__load()
del __load
<loader code/>

In other words, the pyds are loaded with a call to 'imp.load_dynamic',
passing the module name and the .pyd file path.

This code can be found in Lib/site-packages/py2exe/build_exe.py, near line 68.
Just to explain how it works.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Thomas Heller <th...@ct...> wrote:

> As I said, I would prefer the second solution; to rename the pyds
> from 'package/name.pyd' to 'package.name.pyd'.
>
> Would you like to work on a patch?

Yes.  I've just started on it...

-- 
Grant Edwards                   grante             Yow! ... or were you
                                  at               driving the PONTIAC that
                               visi.com            HONKED at me in MIAMI last
                                                   Tuesday?

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-09-06, Thomas Heller <th...@ct...> wrote:

>>> OK, I've got a patch working.  
>> 
>> The patch against 0.6.6 is at
>> 
>> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch
>
> Thanks for the patch, I'm currently looking at it.  Can you please also
> provide a short script (some Numeric or numpy operations, maybe) that
> demonstrates the problem in 0.6.6 and that the patch fixes it?

Sure.  IIRC, the following will do it, but I'll verify that

#import Numeric
#import numpy

-- 
Grant Edwards                   grante             Yow! I'm encased in the
                                  at               lining of a pure pork
                               visi.com            sausage!!

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-09-06, Thomas Heller <th...@ct...> wrote:

> What I do not like in your patch is that it creates files with these names:
>
> c_.python24.DLLs.bz2.pyd
> c_.python24.DLLs.select.pyd
> c_.python24.DLLs.unicodedata.pyd
> c_.python24.DLLs.zlib.pyd
> c_.python24.DLLs._socket.pyd
> c_.python24.DLLs._ssl.pyd
> c_.python24.DLLs._tkinter.pyd
> c_.python24.lib.site-packages.numarray.libnumarray.pyd
> c_.python24.lib.site-packages.numarray.libnumeric.pyd
> c_.python24.lib.site-packages.numarray.memory.pyd
> c_.python24.lib.site-packages.numarray._bytes.pyd
> c_.python24.lib.site-packages.numarray._chararray.pyd
> c_.python24.lib.site-packages.numarray._conv.pyd
> c_.python24.lib.site-packages.numarray._converter.pyd
> c_.python24.lib.site-packages.numarray._ndarray.pyd
> c_.python24.lib.site-packages.numarray._numarray.pyd
> c_.python24.lib.site-packages.numarray._numerictype.pyd
> c_.python24.lib.site-packages.numarray._objectarray.pyd
>
> I have changed your patch a little bit so that instead the filenames
> are derived from the package names (maybe I even mentioned the idea
> in one of the previous posts already):

Ah right. I misread your suggestion and got 'path' instead of
'module' stuck in my head.

> bz2.pyd
> multiarray.pyd
> numarray.libnumarray.pyd
> numarray.libnumeric.pyd
> numarray.memory.pyd
> numarray._bytes.pyd
> numarray._chararray.pyd
> numarray._conv.pyd
> numarray._converter.pyd
> numarray._ndarray.pyd
> numarray._numarray.pyd
> numarray._numerictype.pyd
> numarray._objectarray.pyd
> numarray._operator.pyd
> numarray._sort.pyd
> numarray._ufunc.pyd
> numarray._ufuncBool.pyd
> numarray._ufuncComplex32.pyd
> numpy.core.multiarray.pyd
> numpy.core.scalarmath.pyd
> numpy.core.umath.pyd
> numpy.core._dotblas.pyd
> numpy.core._sort.pyd
> numpy.fft.fftpack_lite.pyd
> numpy.lib._compiled_base.pyd
> numpy.linalg.lapack_lite.pyd
> numpy.random.mtrand.pyd
> select.pyd
> umath.pyd
> unicodedata.pyd
> win32api.pyd
> win32pdh.pyd
> zlib.pyd
> _dotblas.pyd
> _numpy.pyd
> _socket.pyd
> _ssl.pyd
> _tkinter.pyd
>
> IMO this looks much nicer.  Can you think of any reason why this would not work?

No, I think that should still result in unique names, and it is
much nicer to look at.  I suppose Just to be paranoid, one could
add a test when the files are being copied to see if there are
any collisions, but I can't how there could be.

-- 
Grant Edwards                   grante             Yow! ... My pants just went
                                  at               on a wild rampage through a
                               visi.com            Long Island Bowling Alley!!

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Thomas Heller <th...@ct...> wrote:
> Grant Edwards schrieb:
>> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> 
>>> Why not put them in a Numeric directory like they are in the
>>> normal library tree?  Wouldn't that avoid all these namespace
>>> collision problems?
>> 
>> Rats.  I tried moving the Numeric files into a Numeric
>> directory and then creating a Numeric.pth file in the library root
>> directory (same layout as <...>/site-packages.  It didn't work:
>> 
>>   Traceback (most recent call last):
>>     File "tflowfit.py", line 4, in ?
>>     File "Scientific\Functions\LeastSquares.pyc", line 8, in ?
>>     
>>   ImportError: No module named Numeric
>> 
>> Any ideas why Numeric.pth doesn't work?
>>   
>
> IIRC .pth files are processed in normal Python by the site.py
> file, this mechanism isn't used in py2exe.  

I also tried setting PYTHONPATH to no avail.

> You could, however, add some logic to your script that does
> this.

I presume you mean by modifying sys.path in my setup script?

-- 
Grant Edwards                   grante             Yow! Remember, in 2039,
                                  at               MOUSSE & PASTA will
                               visi.com            be available ONLY by
                                                   prescription!!

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>> Grant Edwards schrieb:

>>> Any ideas why Numeric.pth doesn't work?
>>>   
>>
>> IIRC .pth files are processed in normal Python by the site.py
>> file, this mechanism isn't used in py2exe.  
> 
> I also tried setting PYTHONPATH to no avail.

The py2exe generated executables don't care about PYTHONPATH.

>> You could, however, add some logic to your script that does
>> this.
> 
> I presume you mean by modifying sys.path in my setup script?
> 

Yes.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Thomas Heller <th...@ct...> wrote:

>>> You could, however, add some logic to your script that does
>>> this.
>> 
>> I presume you mean by modifying sys.path in my setup script?
>
> Yes.

I tried appending 'Numeric' to sys.path in my setup script, but
it didn't help.  import Numeric still fails because it can't
find the module if I move Numeric.pyc into dist/Numeric.  

What is used as a base for relative paths in sys.path?

-- 
Grant Edwards                   grante             Yow! I love ROCK 'N ROLL!
                                  at               I memorized the all WORDS
                               visi.com            to "WIPE-OUT" in 1965!!

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
> 
>>>> You could, however, add some logic to your script that does
>>>> this.
>>> 
>>> I presume you mean by modifying sys.path in my setup script?
>>
>> Yes.
> 
> I tried appending 'Numeric' to sys.path in my setup script, but
> it didn't help.  import Numeric still fails because it can't
> find the module if I move Numeric.pyc into dist/Numeric.  
> 
> What is used as a base for relative paths in sys.path?
> 

I would think the current directory.  IIRC, you should prepend
sys.prefix which is the absolute path of the dist directory.

The py2exe distribution contains a folder with samples in
Lib/site-packages/py2exe/samples/...,  the simple subdirectory
contains a hello.py script that prints some info which may be useful
to you, and can also be used to check all these variables.

Then, to debug the loading of modules py2exe reacts to an environent
variable named PY2EXE_VERBOSE.  This play for the py2exe scripts the same
role as PYTHONVERBOSE for Python: setting it to '1' will print info about
where modules were loaded from, and setting it to '2' will also print
all the paths that are tried for importing.  Of course, you need to build
a console version of your script to see this output.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Thomas Heller <th...@ct...> wrote:

>>>> Why not put them in a Numeric directory like they are in the
>>>> normal library tree?  Wouldn't that avoid all these namespace
>>>> collision problems?
>>>
>>> That may be.  A better approach for py2exe might be to rename
>>> them into 'Numeric.foobar.pyd'.
>>
>> How would the file that's loading the .pyd files know they
>> have been renamed by py2exe?
>
> Loading .pyds from the dist directory does not use the normal
> mechanism anyway - the dist directory is not on sys.path (not
> by default).
>
> Loading pyds works by loader modules that py2exe creates. If
> you open the library.zip file with winzip

I'm not using a zipped library, since doing so breaks the scipy
package. :)_

> or a similar utility you'll see that it has a xxx.pyc or
> xxx.pyo file in it (in the correct location relative to the
> package) for each xxx.pyd file that is in the dist directory.

I see.  Since py2exe is creating xxx.pyc, it can know that the
.pyd file has been named module.path.xxx.pyd instead of just
xxx.pyd.

It looks like one or the other needs to be done:

  1) Locate the .pyd files in the same directory as the
     corresponding .py[do] file.

  2) Make sure the .pyd file names are unique (e.g. by
     prepending the original object module's path).

The current assumption that all object modules found in the
python library tree have unique filenames is clearly not valid.

-- 
Grant Edwards                   grante             Yow! With YOU, I can be
                                  at               MYSELF ...  We don't NEED
                               visi.com            Dan Rather ...

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
> 
>>>>> Why not put them in a Numeric directory like they are in the
>>>>> normal library tree?  Wouldn't that avoid all these namespace
>>>>> collision problems?
>>>>
>>>> That may be.  A better approach for py2exe might be to rename
>>>> them into 'Numeric.foobar.pyd'.
>>>
>>> How would the file that's loading the .pyd files know they
>>> have been renamed by py2exe?
>>
>> Loading .pyds from the dist directory does not use the normal
>> mechanism anyway - the dist directory is not on sys.path (not
>> by default).
>>
>> Loading pyds works by loader modules that py2exe creates. If
>> you open the library.zip file with winzip
> 
> I'm not using a zipped library, since doing so breaks the scipy
> package. :)_
> 
>> or a similar utility you'll see that it has a xxx.pyc or
>> xxx.pyo file in it (in the correct location relative to the
>> package) for each xxx.pyd file that is in the dist directory.
> 
> I see.  Since py2exe is creating xxx.pyc, it can know that the
> .pyd file has been named module.path.xxx.pyd instead of just
> xxx.pyd.

Correct.

> It looks like one or the other needs to be done:
> 
>   1) Locate the .pyd files in the same directory as the
>      corresponding .py[do] file.
> 
>   2) Make sure the .pyd file names are unique (e.g. by
>      prepending the original object module's path).
> 
> The current assumption that all object modules found in the
> python library tree have unique filenames is clearly not valid.
> 

As I said, I would prefer the second solution; to rename the pyds
from 'package/name.pyd' to 'package.name.pyd'.

Would you like to work on a patch?

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>
>> As I said, I would prefer the second solution; to rename the pyds
>> from 'package/name.pyd' to 'package.name.pyd'.
>>
>> Would you like to work on a patch?
>
> Yes.  I've just started on it...

Two quick questions before I start making changes:

 1) It looks like all I need to modify is Python code in
    build_exe.py (specifically create_loader).  Is that
    correct?  I don't have any way to compile extensions for
    Win32, so I can't work on C code.

 2) Assuming the answer to 1) is yes, is it OK if I work off of
    0.6.6 sources, since that's what I've got installed?  It
    doesn't appear that the code I would be modifying has
    changed between 0.6.6 and HEAD, so merging my changes into
    CVS shouldn't be a problem .

-- 
Grant Edwards                   grante             Yow! Alright, you!!
                                  at               Imitate a WOUNDED SEAL
                               visi.com            pleading for a PARKING
                                                   SPACE!!

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>>
>>> As I said, I would prefer the second solution; to rename the pyds
>>> from 'package/name.pyd' to 'package.name.pyd'.
>>>
>>> Would you like to work on a patch?
>>
>> Yes.  I've just started on it...
> 
> Two quick questions before I start making changes:
> 
>  1) It looks like all I need to modify is Python code in
>     build_exe.py (specifically create_loader).  Is that
>     correct?  I don't have any way to compile extensions for
>     Win32, so I can't work on C code.
> 
>  2) Assuming the answer to 1) is yes, is it OK if I work off of
>     0.6.6 sources, since that's what I've got installed?  It
>     doesn't appear that the code I would be modifying has
>     changed between 0.6.6 and HEAD, so merging my changes into
>     CVS shouldn't be a problem .
> 

Yes to both questions.

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>
>> As I said, I would prefer the second solution; to rename the pyds
>> from 'package/name.pyd' to 'package.name.pyd'.
>>
>> Would you like to work on a patch?
>
> Yes.  I've just started on it...

OK, I've got a patch working.  

Am I correct in my understanding that I only need to worry
about the case where bundle_files > 2 is true?  It appears that
for the false case, the pyd files are left in the original file
heirarchy.

BTW, this comment in copy_extensions made me smile:

        # copy the extensions to the target directory
        for item in extensions:
            src = item.__file__
            if self.bundle_files > 2: # don't bundle pyds and dlls
                # XXX all dlls are copied into the same directory - a flat name space.
                # sooner or later that will give conflicts.

-- 
Grant Edwards                   grante             Yow! I just heard the
                                  at               SEVENTIES were over!!  And
                               visi.com            I was just getting in touch
                                                   with my LEISURE SUIT!!

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>>
>>> As I said, I would prefer the second solution; to rename the pyds
>>> from 'package/name.pyd' to 'package.name.pyd'.
>>>
>>> Would you like to work on a patch?
>>
>> Yes.  I've just started on it...
> 
> OK, I've got a patch working.  
> 
> Am I correct in my understanding that I only need to worry
> about the case where bundle_files > 2 is true?  It appears that
> for the false case, the pyd files are left in the original file
> heirarchy.

Yes, this seems so.

> BTW, this comment in copy_extensions made me smile:
> 
>         # copy the extensions to the target directory
>         for item in extensions:
>             src = item.__file__
>             if self.bundle_files > 2: # don't bundle pyds and dlls
>                 # XXX all dlls are copied into the same directory - a flat name space.
>                 # sooner or later that will give conflicts.
> 

;-)

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>>
>>> As I said, I would prefer the second solution; to rename the pyds
>>> from 'package/name.pyd' to 'package.name.pyd'.
>>>
>>> Would you like to work on a patch?
>>
>> Yes.  I've just started on it...
>
> OK, I've got a patch working.  

The patch against 0.6.6 is at

ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch

It's also included below:
                                                                            
---------------------------------8<----------------------------------
Index: build_exe.py
===================================================================
RCS file: /cvsroot/py2exe/py2exe/py2exe/build_exe.py,v
retrieving revision 1.72
diff -U10 -r1.72 build_exe.py
--- build_exe.py	22 Nov 2006 08:43:36 -0000	1.72
+++ build_exe.py	4 Sep 2007 19:20:13 -0000
@@ -400,21 +400,21 @@
         self.mkpath(self.lib_dir)
 
     def copy_extensions(self, extensions):
         print "*** copy extensions ***"
         # copy the extensions to the target directory
         for item in extensions:
             src = item.__file__
             if self.bundle_files > 2: # don't bundle pyds and dlls
                 # XXX all dlls are copied into the same directory - a flat name space.
                 # sooner or later that will give conflicts.
-                dst = os.path.join(self.lib_dir, os.path.basename(item.__file__))
+                dst = os.path.join(self.lib_dir, (item.__pydfile__))
                 self.copy_file(src, dst, preserve_mode=0)
                 self.lib_files.append(dst)
             else:
                 # we have to preserve the packages
                 package = "\\".join(item.__name__.split(".")[:-1])
                 if package:
                     dst = os.path.join(package, os.path.basename(src))
                 else:
                     dst = os.path.basename(src)
                 self.copy_file(src, os.path.join(self.collect_dir, dst), preserve_mode=0)
@@ -1071,25 +1071,35 @@
             else:
                 fname = os.path.join(os.path.dirname(sys.executable), "w9xpopen.exe")
             # Don't copy w9xpopen.exe if it doesn't exist (64-bit
             # Python build, for example)
             if os.path.exists(fname):
                 dlls.add(fname)
 
     def create_loader(self, item):
         # Hm, how to avoid needless recreation of this file?
         pathname = os.path.join(self.temp_dir, "%s.py" % item.__name__)
+        if self.bundle_files > 2: # don't bundle pyds and dlls
+            # all dlls are copied into the same directory, so
+            # modify names to include original path to avoid 
+            # name conflicts and tuck it away for future reference
+            fname = item.__file__
+            fname = fname.replace(":","_")
+            fname = fname.replace("\\",".")
+            item.__pydfile__ = fname
+        else:
+            fname = os.path.basename(item.__file__)
+            
         # and what about dry_run?
         if self.verbose:
-            print "creating python loader for extension '%s'" % item.__name__
+            print "creating python loader for extension '%s' (%s -> %s)" % (item.__name__,item.__file__,fname)
 
-        fname = os.path.basename(item.__file__)
         source = LOADER % fname
         if not self.dry_run:
             open(pathname, "w").write(source)
         else:
             return None
         from modulefinder import Module
         return Module(item.__name__, pathname)
 
     def plat_prepare(self):
         self.includes.append("warnings") # needed by Python itself
---------------------------------8<----------------------------------

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>> On 2007-08-29, Grant Edwards <gr...@vi...> wrote:
>>> On 2007-08-29, Thomas Heller <th...@ct...> wrote:
>>>
>>>> As I said, I would prefer the second solution; to rename the pyds
>>>> from 'package/name.pyd' to 'package.name.pyd'.
>>>>
>>>> Would you like to work on a patch?
>>>
>>> Yes.  I've just started on it...
>>
>> OK, I've got a patch working.  
> 
> The patch against 0.6.6 is at
> 
> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch

Thanks for the patch, I'm currently looking at it.  Can you please also
provide a short script (some Numeric or numpy operations, maybe) that
demonstrates the problem in 0.6.6 and that the patch fixes it?

Thomas

Re: [Py2exe-users] Numeric file location issues

[Grant E. <gr...@vi...>]

On 2007-09-06, Grant Edwards <gr...@vi...> wrote:
> On 2007-09-06, Thomas Heller <th...@ct...> wrote:
>
>>>> OK, I've got a patch working.  
>>> 
>>> The patch against 0.6.6 is at
>>> 
>>> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch
>>
>> Thanks for the patch, I'm currently looking at it.  Can you please also
>> provide a short script (some Numeric or numpy operations, maybe) that
>> demonstrates the problem in 0.6.6 and that the patch fixes it?
>
> Sure.  IIRC, the following will do it, but I'll verify that
>
> #import Numeric
> #import numpy

Except without the #'s.  Apparently I was partially stuck in C
mode.

-- 
Grant Edwards                   grante             Yow! My haircut is totally
                                  at               traditional!
                               visi.com            

Re: [Py2exe-users] Numeric file location issues

[Thomas H. <th...@ct...>]

Grant Edwards schrieb:
> On 2007-09-06, Grant Edwards <gr...@vi...> wrote:
>> On 2007-09-06, Thomas Heller <th...@ct...> wrote:
>>
>>>>> OK, I've got a patch working.  
>>>> 
>>>> The patch against 0.6.6 is at
>>>> 
>>>> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch
>>>
>>> Thanks for the patch, I'm currently looking at it.  Can you please also
>>> provide a short script (some Numeric or numpy operations, maybe) that
>>> demonstrates the problem in 0.6.6 and that the patch fixes it?
>>
>> Sure.  IIRC, the following will do it, but I'll verify that
>>
>> #import Numeric
>> #import numpy
> 
> Except without the #'s.  Apparently I was partially stuck in C
> mode.
> 

Ah - great.

What I do not like in your patch is that it creates files with these names:

c_.python24.DLLs.bz2.pyd
c_.python24.DLLs.select.pyd
c_.python24.DLLs.unicodedata.pyd
c_.python24.DLLs.zlib.pyd
c_.python24.DLLs._socket.pyd
c_.python24.DLLs._ssl.pyd
c_.python24.DLLs._tkinter.pyd
c_.python24.lib.site-packages.numarray.libnumarray.pyd
c_.python24.lib.site-packages.numarray.libnumeric.pyd
c_.python24.lib.site-packages.numarray.memory.pyd
c_.python24.lib.site-packages.numarray._bytes.pyd
c_.python24.lib.site-packages.numarray._chararray.pyd
c_.python24.lib.site-packages.numarray._conv.pyd
c_.python24.lib.site-packages.numarray._converter.pyd
c_.python24.lib.site-packages.numarray._ndarray.pyd
c_.python24.lib.site-packages.numarray._numarray.pyd
c_.python24.lib.site-packages.numarray._numerictype.pyd
c_.python24.lib.site-packages.numarray._objectarray.pyd

I have changed your patch a little bit so that instead the filenames
are derived from the package names (maybe I even mentioned the idea
in one of the previous posts already):

bz2.pyd
multiarray.pyd
numarray.libnumarray.pyd
numarray.libnumeric.pyd
numarray.memory.pyd
numarray._bytes.pyd
numarray._chararray.pyd
numarray._conv.pyd
numarray._converter.pyd
numarray._ndarray.pyd
numarray._numarray.pyd
numarray._numerictype.pyd
numarray._objectarray.pyd
numarray._operator.pyd
numarray._sort.pyd
numarray._ufunc.pyd
numarray._ufuncBool.pyd
numarray._ufuncComplex32.pyd
numpy.core.multiarray.pyd
numpy.core.scalarmath.pyd
numpy.core.umath.pyd
numpy.core._dotblas.pyd
numpy.core._sort.pyd
numpy.fft.fftpack_lite.pyd
numpy.lib._compiled_base.pyd
numpy.linalg.lapack_lite.pyd
numpy.random.mtrand.pyd
select.pyd
umath.pyd
unicodedata.pyd
win32api.pyd
win32pdh.pyd
zlib.pyd
_dotblas.pyd
_numpy.pyd
_socket.pyd
_ssl.pyd
_tkinter.pyd

IMO this looks much nicer.  Can you think of any reason why this would not work?

Thomas

PS: Here's the changed patch (it is against the CVS version of py2exe, but
will probably apply against 0.6.6 as well):

pcl-cvs: descending directory 
Index: build_exe.py
===================================================================
RCS file: /cvsroot/py2exe/py2exe/py2exe/build_exe.py,v
retrieving revision 1.74
diff -u -r1.74 build_exe.py
--- build_exe.py	6 Sep 2007 07:46:29 -0000	1.74
+++ build_exe.py	6 Sep 2007 08:11:45 -0000
@@ -412,9 +412,7 @@
         for item in extensions:
             src = item.__file__
             if self.bundle_files > 2: # don't bundle pyds and dlls
-                # XXX all dlls are copied into the same directory - a flat name space.
-                # sooner or later that will give conflicts.
-                dst = os.path.join(self.lib_dir, os.path.basename(item.__file__))
+                dst = os.path.join(self.lib_dir, (item.__pydfile__))
                 self.copy_file(src, dst, preserve_mode=0)
                 self.lib_files.append(dst)
             else:
@@ -1085,11 +1083,19 @@
     def create_loader(self, item):
         # Hm, how to avoid needless recreation of this file?
         pathname = os.path.join(self.temp_dir, "%s.py" % item.__name__)
+        if self.bundle_files > 2: # don't bundle pyds and dlls
+            # all dlls are copied into the same directory, so modify
+            # names to include the package name to avoid name
+            # conflicts and tuck it away for future reference
+            fname = item.__name__ + os.path.splitext(item.__file__)[1]
+            item.__pydfile__ = fname
+        else:
+            fname = os.path.basename(item.__file__)
+            
         # and what about dry_run?
         if self.verbose:
-            print "creating python loader for extension '%s'" % item.__name__
+            print "creating python loader for extension '%s' (%s -> %s)" % (item.__name__,item.__file__,fname)
 
-        fname = os.path.basename(item.__file__)
         source = LOADER % fname
         if not self.dry_run:
             open(pathname, "w").write(source)