From: Grant E. <gr...@vi...> - 2007-08-29 16:02:13
|
I'm still battling with numpy/Numeric issues. The basic problem is that py2exe places all of Numeric's files at the "root" of the library (directly in the 'dist' directory) rather than in a Numeric directory (dist/Numeric). The result is that this changes Numeric's modules into global modules (what's supposed to be Numeric.foobar is now just foobar) causing them to override modules with the same name in other packages. The solution I found in the list is to create a directory named Numeric and move the conflicting Numeric files into it. That keeps them from overriding other package modules, but it also hides them from use by the Numeric package. In the "normal" library tree, all of Numeric's modules are in <library>/site-packages/Numeric. Why does py2exe place them in the library "root" directory? Why not put them in a Numeric directory like they are in the normal library tree? Wouldn't that avoid all these namespace collision problems? -- Grant Edwards grante Yow! I know how to do at SPECIAL EFFECTS!! visi.com |
From: Grant E. <gr...@vi...> - 2007-08-29 16:16:55
|
On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > I'm still battling with numpy/Numeric issues. The basic > problem is that py2exe places all of Numeric's files at the > "root" of the library (directly in the 'dist' directory) rather > than in a Numeric directory (dist/Numeric). It appears to do the same thing for many of numarray's files. The .pyc files are in the correct spots, but the .pyd files are erroneously put in the root directory instead of in the directory where the rest of numarray's files are. Why is py2exe putting all these files in the library's root directory when they belong elsewhere? -- Grant Edwards grante Yow! I just went below the at poverty line! visi.com |
From: Thomas H. <th...@ct...> - 2007-08-29 17:35:12
|
Grant Edwards schrieb: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> I'm still battling with numpy/Numeric issues. The basic >> problem is that py2exe places all of Numeric's files at the >> "root" of the library (directly in the 'dist' directory) rather >> than in a Numeric directory (dist/Numeric). > > It appears to do the same thing for many of numarray's files. > The .pyc files are in the correct spots, but the .pyd files are > erroneously put in the root directory instead of in the > directory where the rest of numarray's files are. > > Why is py2exe putting all these files in the library's root > directory when they belong elsewhere? > See my answer to the previous post. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 17:09:31
|
On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > Why not put them in a Numeric directory like they are in the > normal library tree? Wouldn't that avoid all these namespace > collision problems? Rats. I tried moving the Numeric files into a Numeric directory and then creating a Numeric.pth file in the library root directory (same layout as <...>/site-packages. It didn't work: Traceback (most recent call last): File "tflowfit.py", line 4, in ? File "Scientific\Functions\LeastSquares.pyc", line 8, in ? ImportError: No module named Numeric Any ideas why Numeric.pth doesn't work? -- Grant Edwards grante Yow! The PILLSBURY DOUGHBOY at is CRYING for an END to visi.com BURT REYNOLDS movies!! |
From: Thomas H. <th...@ct...> - 2007-08-29 17:30:07
|
Grant Edwards schrieb: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > >> Why not put them in a Numeric directory like they are in the >> normal library tree? Wouldn't that avoid all these namespace >> collision problems? > > Rats. I tried moving the Numeric files into a Numeric > directory and then creating a Numeric.pth file in the library root > directory (same layout as <...>/site-packages. It didn't work: > > Traceback (most recent call last): > File "tflowfit.py", line 4, in ? > File "Scientific\Functions\LeastSquares.pyc", line 8, in ? > > ImportError: No module named Numeric > > Any ideas why Numeric.pth doesn't work? > IIRC .pth files are processed in normal Python by the site.py file, this mechanism isn't used in py2exe. You could, however, add some logic to your script that does this. Thomas |
From: Thomas H. <th...@ct...> - 2007-08-29 17:24:52
|
Grant Edwards schrieb: > I'm still battling with numpy/Numeric issues. The basic > problem is that py2exe places all of Numeric's files at the > "root" of the library (directly in the 'dist' directory) rather > than in a Numeric directory (dist/Numeric). You mean .pyd's, do you? > The result is that > this changes Numeric's modules into global modules (what's > supposed to be Numeric.foobar is now just foobar) causing them > to override modules with the same name in other packages. > > The solution I found in the list is to create a directory named > Numeric and move the conflicting Numeric files into it. That > keeps them from overriding other package modules, but it also > hides them from use by the Numeric package. > > In the "normal" library tree, all of Numeric's modules are in > <library>/site-packages/Numeric. Why does py2exe place them in > the library "root" directory? Because people don't like deep directory trees: they want a single directory, or, better yet, a single file. Usually. > Why not put them in a Numeric directory like they are in the > normal library tree? Wouldn't that avoid all these namespace > collision problems? > That may be. A better approach for py2exe might be to rename them into 'Numeric.foobar.pyd'. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 18:38:46
|
On 2007-08-29, Thomas Heller <th...@ct...> wrote: > Grant Edwards schrieb: >> I'm still battling with numpy/Numeric issues. The basic >> problem is that py2exe places all of Numeric's files at the >> "root" of the library (directly in the 'dist' directory) rather >> than in a Numeric directory (dist/Numeric). > > You mean .pyd's, do you? Yes. The problem is that py2exe will try to put multiple different files with the same name in the library root directory, so the package that owns the "last one written" wins and the package the owns the first one written losses. >> The result is that this changes Numeric's modules into global >> modules (what's supposed to be Numeric.foobar is now just >> foobar) causing them to override modules with the same name in >> other packages. >> >> The solution I found in the list is to create a directory >> named Numeric and move the conflicting Numeric files into it. >> That keeps them from overriding other package modules, but it >> also hides them from use by the Numeric package. >> >> In the "normal" library tree, all of Numeric's modules are in >> <library>/site-packages/Numeric. Why does py2exe place them in >> the library "root" directory? > > Because people don't like deep directory trees: Why? They don't really const anything, and there's information in that directory structure that py2exe is throwing away. That information is required so that the right files get loaded by the right packages. Throwing that information is breaking packages. > they want a single directory, or, better yet, a single file. > Usually. Wanting a shallow directory tree is fine except that I don't think it should take precidence over having a working directory tree. py2exe is causing breaking in two related ways: 1) By "deleting" when it overwrites them with a second file of the same name. 2) By placing files at a different place in the search order than they need to be. >> Why not put them in a Numeric directory like they are in the >> normal library tree? Wouldn't that avoid all these namespace >> collision problems? > > That may be. A better approach for py2exe might be to rename > them into 'Numeric.foobar.pyd'. How would the file that's loading the .pyd files know they have been renamed by py2exe? -- Grant Edwards grante Yow! I want to read my new at poem about pork brains and visi.com outer space ... |
From: Thomas H. <th...@ct...> - 2007-08-29 18:58:15
|
Grant Edwards schrieb: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: >> Grant Edwards schrieb: >>> I'm still battling with numpy/Numeric issues. The basic >>> problem is that py2exe places all of Numeric's files at the >>> "root" of the library (directly in the 'dist' directory) rather >>> than in a Numeric directory (dist/Numeric). >> >> You mean .pyd's, do you? > > Yes. The problem is that py2exe will try to put multiple > different files with the same name in the library root > directory, so the package that owns the "last one written" wins > and the package the owns the first one written losses. > >>> The result is that this changes Numeric's modules into global >>> modules (what's supposed to be Numeric.foobar is now just >>> foobar) causing them to override modules with the same name in >>> other packages. >>> >>> The solution I found in the list is to create a directory >>> named Numeric and move the conflicting Numeric files into it. >>> That keeps them from overriding other package modules, but it >>> also hides them from use by the Numeric package. >>> >>> In the "normal" library tree, all of Numeric's modules are in >>> <library>/site-packages/Numeric. Why does py2exe place them in >>> the library "root" directory? >> >> Because people don't like deep directory trees: > > Why? They don't really const anything, and there's information > in that directory structure that py2exe is throwing away. That > information is required so that the right files get loaded by > the right packages. Throwing that information is breaking > packages. > >> they want a single directory, or, better yet, a single file. >> Usually. > > Wanting a shallow directory tree is fine except that I don't > think it should take precidence over having a working directory > tree. Sure. ;-) > py2exe is causing breaking in two related ways: > > 1) By "deleting" when it overwrites them with a second file of > the same name. > > 2) By placing files at a different place in the search order > than they need to be. > >>> Why not put them in a Numeric directory like they are in the >>> normal library tree? Wouldn't that avoid all these namespace >>> collision problems? >> >> That may be. A better approach for py2exe might be to rename >> them into 'Numeric.foobar.pyd'. > How would the file that's loading the .pyd files know they have > been renamed by py2exe? > Loading .pyds from the dist directory does not use the normal mechanism anyway - the dist directory is not on sys.path (not by default). Loading pyds works by loader modules that py2exe creates. If you open the library.zip file with winzip or a similar utility you'll see that it has a xxx.pyc or xxx.pyo file in it (in the correct location relative to the package) for each xxx.pyd file that is in the dist directory. These xxx.py[co] files contain this code: <loader code> # This loader locates extension modules relative to the library.zip # file when an archive is used (i.e., skip_archive is not used), otherwise # it locates extension modules relative to sys.prefix. def __load(): import imp, os, sys try: dirname = os.path.dirname(__loader__.archive) except NameError: dirname = sys.prefix path = os.path.join(dirname, '%s') #print "py2exe extension module", __name__, "->", path mod = imp.load_dynamic(__name__, path) ## mod.frozen = 1 __load() del __load <loader code/> In other words, the pyds are loaded with a call to 'imp.load_dynamic', passing the module name and the .pyd file path. This code can be found in Lib/site-packages/py2exe/build_exe.py, near line 68. Just to explain how it works. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 20:13:20
|
On 2007-08-29, Thomas Heller <th...@ct...> wrote: > As I said, I would prefer the second solution; to rename the pyds > from 'package/name.pyd' to 'package.name.pyd'. > > Would you like to work on a patch? Yes. I've just started on it... -- Grant Edwards grante Yow! ... or were you at driving the PONTIAC that visi.com HONKED at me in MIAMI last Tuesday? |
From: Grant E. <gr...@vi...> - 2007-09-06 14:17:41
|
On 2007-09-06, Thomas Heller <th...@ct...> wrote: >>> OK, I've got a patch working. >> >> The patch against 0.6.6 is at >> >> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch > > Thanks for the patch, I'm currently looking at it. Can you please also > provide a short script (some Numeric or numpy operations, maybe) that > demonstrates the problem in 0.6.6 and that the patch fixes it? Sure. IIRC, the following will do it, but I'll verify that #import Numeric #import numpy -- Grant Edwards grante Yow! I'm encased in the at lining of a pure pork visi.com sausage!! |
From: Grant E. <gr...@vi...> - 2007-09-06 16:51:46
|
On 2007-09-06, Thomas Heller <th...@ct...> wrote: > What I do not like in your patch is that it creates files with these names: > > c_.python24.DLLs.bz2.pyd > c_.python24.DLLs.select.pyd > c_.python24.DLLs.unicodedata.pyd > c_.python24.DLLs.zlib.pyd > c_.python24.DLLs._socket.pyd > c_.python24.DLLs._ssl.pyd > c_.python24.DLLs._tkinter.pyd > c_.python24.lib.site-packages.numarray.libnumarray.pyd > c_.python24.lib.site-packages.numarray.libnumeric.pyd > c_.python24.lib.site-packages.numarray.memory.pyd > c_.python24.lib.site-packages.numarray._bytes.pyd > c_.python24.lib.site-packages.numarray._chararray.pyd > c_.python24.lib.site-packages.numarray._conv.pyd > c_.python24.lib.site-packages.numarray._converter.pyd > c_.python24.lib.site-packages.numarray._ndarray.pyd > c_.python24.lib.site-packages.numarray._numarray.pyd > c_.python24.lib.site-packages.numarray._numerictype.pyd > c_.python24.lib.site-packages.numarray._objectarray.pyd > > I have changed your patch a little bit so that instead the filenames > are derived from the package names (maybe I even mentioned the idea > in one of the previous posts already): Ah right. I misread your suggestion and got 'path' instead of 'module' stuck in my head. > bz2.pyd > multiarray.pyd > numarray.libnumarray.pyd > numarray.libnumeric.pyd > numarray.memory.pyd > numarray._bytes.pyd > numarray._chararray.pyd > numarray._conv.pyd > numarray._converter.pyd > numarray._ndarray.pyd > numarray._numarray.pyd > numarray._numerictype.pyd > numarray._objectarray.pyd > numarray._operator.pyd > numarray._sort.pyd > numarray._ufunc.pyd > numarray._ufuncBool.pyd > numarray._ufuncComplex32.pyd > numpy.core.multiarray.pyd > numpy.core.scalarmath.pyd > numpy.core.umath.pyd > numpy.core._dotblas.pyd > numpy.core._sort.pyd > numpy.fft.fftpack_lite.pyd > numpy.lib._compiled_base.pyd > numpy.linalg.lapack_lite.pyd > numpy.random.mtrand.pyd > select.pyd > umath.pyd > unicodedata.pyd > win32api.pyd > win32pdh.pyd > zlib.pyd > _dotblas.pyd > _numpy.pyd > _socket.pyd > _ssl.pyd > _tkinter.pyd > > IMO this looks much nicer. Can you think of any reason why this would not work? No, I think that should still result in unique names, and it is much nicer to look at. I suppose Just to be paranoid, one could add a test when the files are being copied to see if there are any collisions, but I can't how there could be. -- Grant Edwards grante Yow! ... My pants just went at on a wild rampage through a visi.com Long Island Bowling Alley!! |
From: Grant E. <gr...@vi...> - 2007-08-29 18:45:06
|
On 2007-08-29, Thomas Heller <th...@ct...> wrote: > Grant Edwards schrieb: >> On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> >>> Why not put them in a Numeric directory like they are in the >>> normal library tree? Wouldn't that avoid all these namespace >>> collision problems? >> >> Rats. I tried moving the Numeric files into a Numeric >> directory and then creating a Numeric.pth file in the library root >> directory (same layout as <...>/site-packages. It didn't work: >> >> Traceback (most recent call last): >> File "tflowfit.py", line 4, in ? >> File "Scientific\Functions\LeastSquares.pyc", line 8, in ? >> >> ImportError: No module named Numeric >> >> Any ideas why Numeric.pth doesn't work? >> > > IIRC .pth files are processed in normal Python by the site.py > file, this mechanism isn't used in py2exe. I also tried setting PYTHONPATH to no avail. > You could, however, add some logic to your script that does > this. I presume you mean by modifying sys.path in my setup script? -- Grant Edwards grante Yow! Remember, in 2039, at MOUSSE & PASTA will visi.com be available ONLY by prescription!! |
From: Thomas H. <th...@ct...> - 2007-08-29 18:49:50
|
Grant Edwards schrieb: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: >> Grant Edwards schrieb: >>> Any ideas why Numeric.pth doesn't work? >>> >> >> IIRC .pth files are processed in normal Python by the site.py >> file, this mechanism isn't used in py2exe. > > I also tried setting PYTHONPATH to no avail. The py2exe generated executables don't care about PYTHONPATH. >> You could, however, add some logic to your script that does >> this. > > I presume you mean by modifying sys.path in my setup script? > Yes. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 19:40:11
|
On 2007-08-29, Thomas Heller <th...@ct...> wrote: >>> You could, however, add some logic to your script that does >>> this. >> >> I presume you mean by modifying sys.path in my setup script? > > Yes. I tried appending 'Numeric' to sys.path in my setup script, but it didn't help. import Numeric still fails because it can't find the module if I move Numeric.pyc into dist/Numeric. What is used as a base for relative paths in sys.path? -- Grant Edwards grante Yow! I love ROCK 'N ROLL! at I memorized the all WORDS visi.com to "WIPE-OUT" in 1965!! |
From: Thomas H. <th...@ct...> - 2007-08-29 19:51:00
|
Grant Edwards schrieb: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: > >>>> You could, however, add some logic to your script that does >>>> this. >>> >>> I presume you mean by modifying sys.path in my setup script? >> >> Yes. > > I tried appending 'Numeric' to sys.path in my setup script, but > it didn't help. import Numeric still fails because it can't > find the module if I move Numeric.pyc into dist/Numeric. > > What is used as a base for relative paths in sys.path? > I would think the current directory. IIRC, you should prepend sys.prefix which is the absolute path of the dist directory. The py2exe distribution contains a folder with samples in Lib/site-packages/py2exe/samples/..., the simple subdirectory contains a hello.py script that prints some info which may be useful to you, and can also be used to check all these variables. Then, to debug the loading of modules py2exe reacts to an environent variable named PY2EXE_VERBOSE. This play for the py2exe scripts the same role as PYTHONVERBOSE for Python: setting it to '1' will print info about where modules were loaded from, and setting it to '2' will also print all the paths that are tried for importing. Of course, you need to build a console version of your script to see this output. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 19:50:17
|
On 2007-08-29, Thomas Heller <th...@ct...> wrote: >>>> Why not put them in a Numeric directory like they are in the >>>> normal library tree? Wouldn't that avoid all these namespace >>>> collision problems? >>> >>> That may be. A better approach for py2exe might be to rename >>> them into 'Numeric.foobar.pyd'. >> >> How would the file that's loading the .pyd files know they >> have been renamed by py2exe? > > Loading .pyds from the dist directory does not use the normal > mechanism anyway - the dist directory is not on sys.path (not > by default). > > Loading pyds works by loader modules that py2exe creates. If > you open the library.zip file with winzip I'm not using a zipped library, since doing so breaks the scipy package. :)_ > or a similar utility you'll see that it has a xxx.pyc or > xxx.pyo file in it (in the correct location relative to the > package) for each xxx.pyd file that is in the dist directory. I see. Since py2exe is creating xxx.pyc, it can know that the .pyd file has been named module.path.xxx.pyd instead of just xxx.pyd. It looks like one or the other needs to be done: 1) Locate the .pyd files in the same directory as the corresponding .py[do] file. 2) Make sure the .pyd file names are unique (e.g. by prepending the original object module's path). The current assumption that all object modules found in the python library tree have unique filenames is clearly not valid. -- Grant Edwards grante Yow! With YOU, I can be at MYSELF ... We don't NEED visi.com Dan Rather ... |
From: Thomas H. <th...@ct...> - 2007-08-29 20:01:17
|
Grant Edwards schrieb: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: > >>>>> Why not put them in a Numeric directory like they are in the >>>>> normal library tree? Wouldn't that avoid all these namespace >>>>> collision problems? >>>> >>>> That may be. A better approach for py2exe might be to rename >>>> them into 'Numeric.foobar.pyd'. >>> >>> How would the file that's loading the .pyd files know they >>> have been renamed by py2exe? >> >> Loading .pyds from the dist directory does not use the normal >> mechanism anyway - the dist directory is not on sys.path (not >> by default). >> >> Loading pyds works by loader modules that py2exe creates. If >> you open the library.zip file with winzip > > I'm not using a zipped library, since doing so breaks the scipy > package. :)_ > >> or a similar utility you'll see that it has a xxx.pyc or >> xxx.pyo file in it (in the correct location relative to the >> package) for each xxx.pyd file that is in the dist directory. > > I see. Since py2exe is creating xxx.pyc, it can know that the > .pyd file has been named module.path.xxx.pyd instead of just > xxx.pyd. Correct. > It looks like one or the other needs to be done: > > 1) Locate the .pyd files in the same directory as the > corresponding .py[do] file. > > 2) Make sure the .pyd file names are unique (e.g. by > prepending the original object module's path). > > The current assumption that all object modules found in the > python library tree have unique filenames is clearly not valid. > As I said, I would prefer the second solution; to rename the pyds from 'package/name.pyd' to 'package.name.pyd'. Would you like to work on a patch? Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 20:27:57
|
On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: > >> As I said, I would prefer the second solution; to rename the pyds >> from 'package/name.pyd' to 'package.name.pyd'. >> >> Would you like to work on a patch? > > Yes. I've just started on it... Two quick questions before I start making changes: 1) It looks like all I need to modify is Python code in build_exe.py (specifically create_loader). Is that correct? I don't have any way to compile extensions for Win32, so I can't work on C code. 2) Assuming the answer to 1) is yes, is it OK if I work off of 0.6.6 sources, since that's what I've got installed? It doesn't appear that the code I would be modifying has changed between 0.6.6 and HEAD, so merging my changes into CVS shouldn't be a problem . -- Grant Edwards grante Yow! Alright, you!! at Imitate a WOUNDED SEAL visi.com pleading for a PARKING SPACE!! |
From: Thomas H. <th...@ct...> - 2007-08-30 10:59:25
|
Grant Edwards schrieb: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> On 2007-08-29, Thomas Heller <th...@ct...> wrote: >> >>> As I said, I would prefer the second solution; to rename the pyds >>> from 'package/name.pyd' to 'package.name.pyd'. >>> >>> Would you like to work on a patch? >> >> Yes. I've just started on it... > > Two quick questions before I start making changes: > > 1) It looks like all I need to modify is Python code in > build_exe.py (specifically create_loader). Is that > correct? I don't have any way to compile extensions for > Win32, so I can't work on C code. > > 2) Assuming the answer to 1) is yes, is it OK if I work off of > 0.6.6 sources, since that's what I've got installed? It > doesn't appear that the code I would be modifying has > changed between 0.6.6 and HEAD, so merging my changes into > CVS shouldn't be a problem . > Yes to both questions. Thomas |
From: Grant E. <gr...@vi...> - 2007-08-29 21:28:42
|
On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > On 2007-08-29, Thomas Heller <th...@ct...> wrote: > >> As I said, I would prefer the second solution; to rename the pyds >> from 'package/name.pyd' to 'package.name.pyd'. >> >> Would you like to work on a patch? > > Yes. I've just started on it... OK, I've got a patch working. Am I correct in my understanding that I only need to worry about the case where bundle_files > 2 is true? It appears that for the false case, the pyd files are left in the original file heirarchy. BTW, this comment in copy_extensions made me smile: # copy the extensions to the target directory for item in extensions: src = item.__file__ if self.bundle_files > 2: # don't bundle pyds and dlls # XXX all dlls are copied into the same directory - a flat name space. # sooner or later that will give conflicts. -- Grant Edwards grante Yow! I just heard the at SEVENTIES were over!! And visi.com I was just getting in touch with my LEISURE SUIT!! |
From: Thomas H. <th...@ct...> - 2007-08-30 11:05:10
|
Grant Edwards schrieb: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> On 2007-08-29, Thomas Heller <th...@ct...> wrote: >> >>> As I said, I would prefer the second solution; to rename the pyds >>> from 'package/name.pyd' to 'package.name.pyd'. >>> >>> Would you like to work on a patch? >> >> Yes. I've just started on it... > > OK, I've got a patch working. > > Am I correct in my understanding that I only need to worry > about the case where bundle_files > 2 is true? It appears that > for the false case, the pyd files are left in the original file > heirarchy. Yes, this seems so. > BTW, this comment in copy_extensions made me smile: > > # copy the extensions to the target directory > for item in extensions: > src = item.__file__ > if self.bundle_files > 2: # don't bundle pyds and dlls > # XXX all dlls are copied into the same directory - a flat name space. > # sooner or later that will give conflicts. > ;-) Thomas |
From: Grant E. <gr...@vi...> - 2007-09-04 19:28:15
|
On 2007-08-29, Grant Edwards <gr...@vi...> wrote: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> On 2007-08-29, Thomas Heller <th...@ct...> wrote: >> >>> As I said, I would prefer the second solution; to rename the pyds >>> from 'package/name.pyd' to 'package.name.pyd'. >>> >>> Would you like to work on a patch? >> >> Yes. I've just started on it... > > OK, I've got a patch working. The patch against 0.6.6 is at ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch It's also included below: ---------------------------------8<---------------------------------- Index: build_exe.py =================================================================== RCS file: /cvsroot/py2exe/py2exe/py2exe/build_exe.py,v retrieving revision 1.72 diff -U10 -r1.72 build_exe.py --- build_exe.py 22 Nov 2006 08:43:36 -0000 1.72 +++ build_exe.py 4 Sep 2007 19:20:13 -0000 @@ -400,21 +400,21 @@ self.mkpath(self.lib_dir) def copy_extensions(self, extensions): print "*** copy extensions ***" # copy the extensions to the target directory for item in extensions: src = item.__file__ if self.bundle_files > 2: # don't bundle pyds and dlls # XXX all dlls are copied into the same directory - a flat name space. # sooner or later that will give conflicts. - dst = os.path.join(self.lib_dir, os.path.basename(item.__file__)) + dst = os.path.join(self.lib_dir, (item.__pydfile__)) self.copy_file(src, dst, preserve_mode=0) self.lib_files.append(dst) else: # we have to preserve the packages package = "\\".join(item.__name__.split(".")[:-1]) if package: dst = os.path.join(package, os.path.basename(src)) else: dst = os.path.basename(src) self.copy_file(src, os.path.join(self.collect_dir, dst), preserve_mode=0) @@ -1071,25 +1071,35 @@ else: fname = os.path.join(os.path.dirname(sys.executable), "w9xpopen.exe") # Don't copy w9xpopen.exe if it doesn't exist (64-bit # Python build, for example) if os.path.exists(fname): dlls.add(fname) def create_loader(self, item): # Hm, how to avoid needless recreation of this file? pathname = os.path.join(self.temp_dir, "%s.py" % item.__name__) + if self.bundle_files > 2: # don't bundle pyds and dlls + # all dlls are copied into the same directory, so + # modify names to include original path to avoid + # name conflicts and tuck it away for future reference + fname = item.__file__ + fname = fname.replace(":","_") + fname = fname.replace("\\",".") + item.__pydfile__ = fname + else: + fname = os.path.basename(item.__file__) + # and what about dry_run? if self.verbose: - print "creating python loader for extension '%s'" % item.__name__ + print "creating python loader for extension '%s' (%s -> %s)" % (item.__name__,item.__file__,fname) - fname = os.path.basename(item.__file__) source = LOADER % fname if not self.dry_run: open(pathname, "w").write(source) else: return None from modulefinder import Module return Module(item.__name__, pathname) def plat_prepare(self): self.includes.append("warnings") # needed by Python itself ---------------------------------8<---------------------------------- |
From: Thomas H. <th...@ct...> - 2007-09-06 08:14:27
|
Grant Edwards schrieb: > On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >> On 2007-08-29, Grant Edwards <gr...@vi...> wrote: >>> On 2007-08-29, Thomas Heller <th...@ct...> wrote: >>> >>>> As I said, I would prefer the second solution; to rename the pyds >>>> from 'package/name.pyd' to 'package.name.pyd'. >>>> >>>> Would you like to work on a patch? >>> >>> Yes. I've just started on it... >> >> OK, I've got a patch working. > > The patch against 0.6.6 is at > > ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch Thanks for the patch, I'm currently looking at it. Can you please also provide a short script (some Numeric or numpy operations, maybe) that demonstrates the problem in 0.6.6 and that the patch fixes it? Thomas |
From: Grant E. <gr...@vi...> - 2007-09-06 14:32:41
|
On 2007-09-06, Grant Edwards <gr...@vi...> wrote: > On 2007-09-06, Thomas Heller <th...@ct...> wrote: > >>>> OK, I've got a patch working. >>> >>> The patch against 0.6.6 is at >>> >>> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch >> >> Thanks for the patch, I'm currently looking at it. Can you please also >> provide a short script (some Numeric or numpy operations, maybe) that >> demonstrates the problem in 0.6.6 and that the patch fixes it? > > Sure. IIRC, the following will do it, but I'll verify that > > #import Numeric > #import numpy Except without the #'s. Apparently I was partially stuck in C mode. -- Grant Edwards grante Yow! My haircut is totally at traditional! visi.com |
From: Thomas H. <th...@ct...> - 2007-09-06 16:37:54
|
Grant Edwards schrieb: > On 2007-09-06, Grant Edwards <gr...@vi...> wrote: >> On 2007-09-06, Thomas Heller <th...@ct...> wrote: >> >>>>> OK, I've got a patch working. >>>> >>>> The patch against 0.6.6 is at >>>> >>>> ftp://ftp.visi.com/users/grante/python/py2exe-0.6.6-pyd-name-collision.patch >>> >>> Thanks for the patch, I'm currently looking at it. Can you please also >>> provide a short script (some Numeric or numpy operations, maybe) that >>> demonstrates the problem in 0.6.6 and that the patch fixes it? >> >> Sure. IIRC, the following will do it, but I'll verify that >> >> #import Numeric >> #import numpy > > Except without the #'s. Apparently I was partially stuck in C > mode. > Ah - great. What I do not like in your patch is that it creates files with these names: c_.python24.DLLs.bz2.pyd c_.python24.DLLs.select.pyd c_.python24.DLLs.unicodedata.pyd c_.python24.DLLs.zlib.pyd c_.python24.DLLs._socket.pyd c_.python24.DLLs._ssl.pyd c_.python24.DLLs._tkinter.pyd c_.python24.lib.site-packages.numarray.libnumarray.pyd c_.python24.lib.site-packages.numarray.libnumeric.pyd c_.python24.lib.site-packages.numarray.memory.pyd c_.python24.lib.site-packages.numarray._bytes.pyd c_.python24.lib.site-packages.numarray._chararray.pyd c_.python24.lib.site-packages.numarray._conv.pyd c_.python24.lib.site-packages.numarray._converter.pyd c_.python24.lib.site-packages.numarray._ndarray.pyd c_.python24.lib.site-packages.numarray._numarray.pyd c_.python24.lib.site-packages.numarray._numerictype.pyd c_.python24.lib.site-packages.numarray._objectarray.pyd I have changed your patch a little bit so that instead the filenames are derived from the package names (maybe I even mentioned the idea in one of the previous posts already): bz2.pyd multiarray.pyd numarray.libnumarray.pyd numarray.libnumeric.pyd numarray.memory.pyd numarray._bytes.pyd numarray._chararray.pyd numarray._conv.pyd numarray._converter.pyd numarray._ndarray.pyd numarray._numarray.pyd numarray._numerictype.pyd numarray._objectarray.pyd numarray._operator.pyd numarray._sort.pyd numarray._ufunc.pyd numarray._ufuncBool.pyd numarray._ufuncComplex32.pyd numpy.core.multiarray.pyd numpy.core.scalarmath.pyd numpy.core.umath.pyd numpy.core._dotblas.pyd numpy.core._sort.pyd numpy.fft.fftpack_lite.pyd numpy.lib._compiled_base.pyd numpy.linalg.lapack_lite.pyd numpy.random.mtrand.pyd select.pyd umath.pyd unicodedata.pyd win32api.pyd win32pdh.pyd zlib.pyd _dotblas.pyd _numpy.pyd _socket.pyd _ssl.pyd _tkinter.pyd IMO this looks much nicer. Can you think of any reason why this would not work? Thomas PS: Here's the changed patch (it is against the CVS version of py2exe, but will probably apply against 0.6.6 as well): pcl-cvs: descending directory Index: build_exe.py =================================================================== RCS file: /cvsroot/py2exe/py2exe/py2exe/build_exe.py,v retrieving revision 1.74 diff -u -r1.74 build_exe.py --- build_exe.py 6 Sep 2007 07:46:29 -0000 1.74 +++ build_exe.py 6 Sep 2007 08:11:45 -0000 @@ -412,9 +412,7 @@ for item in extensions: src = item.__file__ if self.bundle_files > 2: # don't bundle pyds and dlls - # XXX all dlls are copied into the same directory - a flat name space. - # sooner or later that will give conflicts. - dst = os.path.join(self.lib_dir, os.path.basename(item.__file__)) + dst = os.path.join(self.lib_dir, (item.__pydfile__)) self.copy_file(src, dst, preserve_mode=0) self.lib_files.append(dst) else: @@ -1085,11 +1083,19 @@ def create_loader(self, item): # Hm, how to avoid needless recreation of this file? pathname = os.path.join(self.temp_dir, "%s.py" % item.__name__) + if self.bundle_files > 2: # don't bundle pyds and dlls + # all dlls are copied into the same directory, so modify + # names to include the package name to avoid name + # conflicts and tuck it away for future reference + fname = item.__name__ + os.path.splitext(item.__file__)[1] + item.__pydfile__ = fname + else: + fname = os.path.basename(item.__file__) + # and what about dry_run? if self.verbose: - print "creating python loader for extension '%s'" % item.__name__ + print "creating python loader for extension '%s' (%s -> %s)" % (item.__name__,item.__file__,fname) - fname = os.path.basename(item.__file__) source = LOADER % fname if not self.dry_run: open(pathname, "w").write(source) |