From: <py...@ha...> - 2006-09-11 18:50:10
|
>py2exe@ha... wrote: >> When I run py2exe against the same input multiple times, I get different >> output each time. In particular, the file library.zip is (equivalent but) >> different. This can be verified with a example as simple as the Hello >> World sample from the Tutorial page. >> >> My automated build system could benefit from having py2exe produce >> identical output for the same input. >> >> Any suggestions? > >You probably should post a more detailed description of what you mean by >"equivalent but different". > >-Larry When I run and test "python setup.py py2exe" repeatedly for the hello.py sample, the resulting application always behaves as expected (i.e. every run is FUNCTIONALLY EQUIVALENT), but the files in the output (i.e. "library.zip") are BINARILY DIFFERENT each time. I suspect that one of the files inside the zip contains a time stamp or something that causes the zip to be different every time. |
From: Larry B. <lar...@we...> - 2006-09-11 23:55:25
|
> > When I run and test "python setup.py py2exe" repeatedly for the hello.py > sample, the resulting application always behaves as expected (i.e. every > run is FUNCTIONALLY EQUIVALENT), but the files in the output (i.e. > "library.zip") are BINARILY DIFFERENT each time. I suspect that one of the > files inside the zip contains a time stamp or something that causes the > zip to be different every time. Doesn't do that on my system. I saved library.zip that was generated on 31 Aug 2006 and then created new one just now 11 Sep 2006. Then ran FC (file compare) on them and they are the same. I suspect that you are changing something that causes python to recreate at least one of the .pyo files that are placed into library.zip. -Larry Bates |
From: <py...@ha...> - 2006-09-12 01:38:29
|
> Doesn't do that on my system. I saved library.zip that was generated > on 31 Aug 2006 and then created new one just now 11 Sep 2006. Then ran > FC (file compare) on them and they are the same. I suspect that you > are changing something that causes python to recreate at least one of > the .pyo files that are placed into library.zip. That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You? I looked deeper into the problem. Turns out the differences between my various library.zip files ARE timestamps, more precisely bytes 4 to 7 of the files bz2.pyc, unicodedata.pyc, and zlib.pyc inside the zips. (Same problem, but with .pyo files, when I ask py2exe to optimize the compiled files.) The sources for these files are regenerated on the fly (in the build/ subfolder) everytime I run py2exe on hello.py, as you can see at the start of the command output: running py2exe *** searching for required modules *** *** parsing results *** creating python loader for extension 'unicodedata' creating python loader for extension 'zlib' creating python loader for extension 'bz2' [...] Since the source files are always new, python recompiles the .pyc files each time and this changes the embedded timestamp. Can anyone give me more information on these files, and how could I ensure that they are always compiled with the same timestamp? Thanks. |
From: Larry B. <lar...@we...> - 2006-09-12 16:16:24
|
py...@ha... wrote: >> Doesn't do that on my system. I saved library.zip that was generated >> on 31 Aug 2006 and then created new one just now 11 Sep 2006. Then ran >> FC (file compare) on them and they are the same. I suspect that you >> are changing something that causes python to recreate at least one of >> the .pyo files that are placed into library.zip. > > That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You? > > I looked deeper into the problem. Turns out the differences between my > various library.zip files ARE timestamps, more precisely bytes 4 to 7 of > the files bz2.pyc, unicodedata.pyc, and zlib.pyc inside the zips. (Same > problem, but with .pyo files, when I ask py2exe to optimize the compiled > files.) The sources for these files are regenerated on the fly (in the > build/ subfolder) everytime I run py2exe on hello.py, as you can see at > the start of the command output: > > running py2exe > *** searching for required modules *** > *** parsing results *** > creating python loader for extension 'unicodedata' > creating python loader for extension 'zlib' > creating python loader for extension 'bz2' > [...] > > Since the source files are always new, python recompiles the .pyc files > each time and this changes the embedded timestamp. Can anyone give me more > information on these files, and how could I ensure that they are always > compiled with the same timestamp? Thanks. > > > That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You? I'm using 2.4.1 on Windows XP Pro SP2. When I run I see: Creating ZBKUP binary using py2exe ------------------------------------------------------------------- running py2exe *** searching for required modules *** *** parsing results *** *** finding dlls needed *** *** create binaries *** *** byte compile python files *** writing byte-compilation script 'c:\temp\tmp_oohre.py' c:\python25\python.exe -OO c:\temp\tmp_oohre.py skipping byte-compilation of F:\Larry\Python\Library\Utils\__init__.py to Librar y\Utils\__init__.pyo skipping byte-compilation of F:\Larry\Python\Library\Utils\fmt_wcommas.py to Lib rary\Utils\fmt_wcommas.pyo . . Lots more of these for other files. . I don't know why it would redo byte-compilation of files that aren't changing. You probably need to post your setup.py script also. -Larry |
From: <py...@ha...> - 2006-09-12 17:09:15
|
> You probably need to post your setup.py script also. It's the sample used in the py2exe tutorial (http://www.py2exe.org/index.cgi/Tutorial). I attached it for your convenience. I just run "python setup.py py2exe", rename the resulting folders (dist/ and build/), run again, and compare the two outputs. Thanks for your help. |
From: John M. <sjm...@le...> - 2006-09-12 21:24:44
|
On 13/09/2006 3:09 AM, py...@ha... wrote: >> You probably need to post your setup.py script also. > > It's the sample used in the py2exe tutorial > (http://www.py2exe.org/index.cgi/Tutorial). I attached it for your > convenience. > > I just run "python setup.py py2exe", rename the resulting folders (dist/ > and build/), run again, and compare the two outputs. > Maybe I've missed something earlier in the thread, or just plain ol' lost the plot, but: When it's run the second time, there are no dist and build folders, so the setup has to create everything afresh, yes/no? Maybe you should *copy* the contents of the dist and build folders before the second run, instead of renaming. HTH, John |
From: <py...@ha...> - 2006-09-13 02:05:07
|
When it's run the second time, there are no dist and build folders, so > the setup has to create everything afresh, yes/no? Maybe you should > *copy* the contents of the dist and build folders before the second run, > instead of renaming. John, Even if I don't rename or delete the build/ and dist/ folders, py2exe regenerates the bz2.py, unicodedata.py and zlib.py files (in build/), which updates their timestamps, which changes the compiled versions (*.pyc) included in library.zip. Same result: library.zip is different every time I run py2exe, even if the sources haven't changed. |
From: Alexander B. <bi...@uk...> - 2006-09-13 07:12:24
|
py...@ha... пишет: > When it's run the second time, there are no dist and build folders, so >> the setup has to create everything afresh, yes/no? Maybe you should >> *copy* the contents of the dist and build folders before the second run, >> instead of renaming. > > John, > > Even if I don't rename or delete the build/ and dist/ folders, py2exe > regenerates the bz2.py, unicodedata.py and zlib.py files (in build/), > which updates their timestamps, which changes the compiled versions > (*.pyc) included in library.zip. Same result: library.zip is different > every time I run py2exe, even if the sources haven't changed. I confirm this effect. My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5. I think there is something straightforward in py2exe build process: it does not check that some files may exist and have the same content but rebuild it every time from scratch. I think it's possible to fix this behaviour but it's not trivial fix. I can propose you to check library.zip not at binary level as one file, but checking content of each file inside zip. Per example by calculating md5 sum for all files inside zip. It's looks somewhat ugly, but it at least simple solution that does not require to hack py2exe itself. -- Alexander |
From: Larry B. <lar...@we...> - 2006-09-13 22:56:11
|
Alexander Belchenko wrote: > py...@ha... пишет: >> When it's run the second time, there are no dist and build folders, so >>> the setup has to create everything afresh, yes/no? Maybe you should >>> *copy* the contents of the dist and build folders before the second run, >>> instead of renaming. >> John, >> >> Even if I don't rename or delete the build/ and dist/ folders, py2exe >> regenerates the bz2.py, unicodedata.py and zlib.py files (in build/), >> which updates their timestamps, which changes the compiled versions >> (*.pyc) included in library.zip. Same result: library.zip is different >> every time I run py2exe, even if the sources haven't changed. > > I confirm this effect. > My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5. > > I think there is something straightforward in py2exe build process: it > does not check that some files may exist and have the same content but > rebuild it every time from scratch. I think it's possible to fix this > behaviour but it's not trivial fix. > > I can propose you to check library.zip not at binary level as one file, > but checking content of each file inside zip. Per example by calculating > md5 sum for all files inside zip. It's looks somewhat ugly, but it at > least simple solution that does not require to hack py2exe itself. > > -- > Alexander > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Py2exe-users mailing list > Py2...@li... > https://lists.sourceforge.net/lists/listinfo/py2exe-users Whatever is going on is subtle....My unicodedata.pyd does not change. Hey wait, you said "...bz2.py, unicodedata.py and zlib.py files..." My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or bz2.py what's up with that? On my system these are all .pyd files and they are not generated as they are binary extensions. -Larry |
From: John M. <sjm...@le...> - 2006-09-14 01:16:10
|
On 14/09/2006 8:55 AM, Larry Bates wrote: > Alexander Belchenko wrote: >> py...@ha... =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>> When it's run the second time, there are no dist and build folders,= so >>>> the setup has to create everything afresh, yes/no? Maybe you should = =20 >>>> *copy* the contents of the dist and build folders before the second = run, =20 >>>> instead of renaming. >>> John, >>> >>> Even if I don't rename or delete the build/ and dist/ folders, py2exe= =20 >>> regenerates the bz2.py, unicodedata.py and zlib.py files (in build/),= =20 >>> which updates their timestamps, which changes the compiled versions =20 >>> (*.pyc) included in library.zip. Same result: library.zip is differen= t =20 >>> every time I run py2exe, even if the sources haven't changed. >> I confirm this effect. >> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5. >> >> I think there is something straightforward in py2exe build process: it >> does not check that some files may exist and have the same content but >> rebuild it every time from scratch. I think it's possible to fix this >> behaviour but it's not trivial fix. >> >> I can propose you to check library.zip not at binary level as one file= , >> but checking content of each file inside zip. Per example by calculati= ng >> md5 sum for all files inside zip. It's looks somewhat ugly, but it a= t >> least simple solution that does not require to hack py2exe itself. >> >> -- >> Alexander >> > Whatever is going on is subtle....My unicodedata.pyd does not change. > Hey wait, you said >=20 > "...bz2.py, unicodedata.py and zlib.py files..." >=20 > My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or > bz2.py what's up with that? On my system these are all .pyd files and > they are not generated as they are binary extensions. >=20 > -Larry The library.zip includes (at the end) 3 files bz2.pyc etc. The dist=20 folder includes 3 files bz2.pyd etc. My guess is that py2exe=20 manufactures a "wrapper" foo.py on the fly for each foo.pyd. I guess the OP will just have to stop pressing the build button when=20 nothing has changed :-) |
From: Larry B. <lar...@we...> - 2006-09-14 02:23:37
|
John Machin wrote: > On 14/09/2006 8:55 AM, Larry Bates wrote: >> Alexander Belchenko wrote: >>> py...@ha... =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>>> When it's run the second time, there are no dist and build >>>> folders, so >>>>> the setup has to create everything afresh, yes/no? Maybe you >>>>> should *copy* the contents of the dist and build folders before >>>>> the second run, instead of renaming. >>>> John, >>>> >>>> Even if I don't rename or delete the build/ and dist/ folders, >>>> py2exe regenerates the bz2.py, unicodedata.py and zlib.py files >>>> (in build/), which updates their timestamps, which changes the >>>> compiled versions (*.pyc) included in library.zip. Same result: >>>> library.zip is different every time I run py2exe, even if the >>>> sources haven't changed. >>> I confirm this effect. >>> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5. >>> >>> I think there is something straightforward in py2exe build process: i= t >>> does not check that some files may exist and have the same content bu= t >>> rebuild it every time from scratch. I think it's possible to fix this >>> behaviour but it's not trivial fix. >>> >>> I can propose you to check library.zip not at binary level as one fil= e, >>> but checking content of each file inside zip. Per example by >>> calculating >>> md5 sum for all files inside zip. It's looks somewhat ugly, but it = at >>> least simple solution that does not require to hack py2exe itself. >>> >>> --=20 >>> Alexander >>> > >> Whatever is going on is subtle....My unicodedata.pyd does not change. >> Hey wait, you said >> >> "...bz2.py, unicodedata.py and zlib.py files..." >> >> My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or >> bz2.py what's up with that? On my system these are all .pyd files and >> they are not generated as they are binary extensions. >> >> -Larry > > The library.zip includes (at the end) 3 files bz2.pyc etc. The dist > folder includes 3 files bz2.pyd etc. My guess is that py2exe > manufactures a "wrapper" foo.py on the fly for each foo.pyd. > > I guess the OP will just have to stop pressing the build button when > nothing has changed :-) > > > > I don't have any .pyc files in my library file. I do however use "bundle_files": 2 in my options in the setup.py file. This places the .pyd files inside the library file.=20 Maybe that's the "secret". -Larry |
From: John M. <sjm...@le...> - 2006-09-14 02:36:42
|
On 14/09/2006 12:23 PM, Larry Bates wrote: > John Machin wrote: >> On 14/09/2006 8:55 AM, Larry Bates wrote: >>> Alexander Belchenko wrote: >>>> py...@ha... =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>>>> When it's run the second time, there are no dist and build >>>>> folders, so >>>>>> the setup has to create everything afresh, yes/no? Maybe you >>>>>> should *copy* the contents of the dist and build folders before >>>>>> the second run, instead of renaming. >>>>> John, >>>>> >>>>> Even if I don't rename or delete the build/ and dist/ folders, >>>>> py2exe regenerates the bz2.py, unicodedata.py and zlib.py files >>>>> (in build/), which updates their timestamps, which changes the >>>>> compiled versions (*.pyc) included in library.zip. Same result: >>>>> library.zip is different every time I run py2exe, even if the >>>>> sources haven't changed. >>>> I confirm this effect. >>>> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5. >>>> >>>> I think there is something straightforward in py2exe build process: = it >>>> does not check that some files may exist and have the same content b= ut >>>> rebuild it every time from scratch. I think it's possible to fix thi= s >>>> behaviour but it's not trivial fix. >>>> >>>> I can propose you to check library.zip not at binary level as one fi= le, >>>> but checking content of each file inside zip. Per example by >>>> calculating >>>> md5 sum for all files inside zip. It's looks somewhat ugly, but it= at >>>> least simple solution that does not require to hack py2exe itself. >>>> >>>> --=20 >>>> Alexander >>>> >>> Whatever is going on is subtle....My unicodedata.pyd does not change. >>> Hey wait, you said >>> >>> "...bz2.py, unicodedata.py and zlib.py files..." >>> >>> My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or >>> bz2.py what's up with that? On my system these are all .pyd files an= d >>> they are not generated as they are binary extensions. >>> >>> -Larry >> The library.zip includes (at the end) 3 files bz2.pyc etc. The dist >> folder includes 3 files bz2.pyd etc. My guess is that py2exe >> manufactures a "wrapper" foo.py on the fly for each foo.pyd. >> >> I guess the OP will just have to stop pressing the build button when >> nothing has changed :-) >> >> >> >> > I don't have any .pyc files in my library file. I do however use > "bundle_files": 2 in my options in the > setup.py file. This places the .pyd files inside the library file.=20 > Maybe that's the "secret". >=20 Could well be; I was emulating the OP: bog-standard no-options hello.py=20 and setup.py as per the tutorial. |
From: <py...@ha...> - 2006-09-13 21:35:12
Attachments:
build.exe.py.patch
|
Attached is a quick hack I wrote so that py2exe wouldn't recreate its "loader" files if their contents haven't changed since the last time. This ensures that py2exe always creates the same output for identical intput. |