py2exe

> 
> When I run and test "python setup.py py2exe" repeatedly for the hello.py
> sample, the resulting application always behaves as expected (i.e. every
> run is FUNCTIONALLY EQUIVALENT), but the files in the output (i.e.
> "library.zip") are BINARILY DIFFERENT each time. I suspect that one of the
> files inside the zip contains a time stamp or something that causes the
> zip to be different every time.

Doesn't do that on my system.  I saved library.zip that was generated
on 31 Aug 2006 and then created new one just now 11 Sep 2006.  Then ran
FC (file compare) on them and they are the same.  I suspect that you
are changing something that causes python to recreate at least one of
the .pyo files that are placed into library.zip.

-Larry Bates

> Doesn't do that on my system.  I saved library.zip that was generated
> on 31 Aug 2006 and then created new one just now 11 Sep 2006.  Then ran
> FC (file compare) on them and they are the same.  I suspect that you
> are changing something that causes python to recreate at least one of
> the .pyo files that are placed into library.zip.

That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You?

I looked deeper into the problem. Turns out the differences between my  
various library.zip files ARE timestamps, more precisely bytes 4 to 7 of  
the files bz2.pyc, unicodedata.pyc, and zlib.pyc inside the zips. (Same  
problem, but with .pyo files, when I ask py2exe to optimize the compiled  
files.) The sources for these files are regenerated on the fly (in the  
build/ subfolder) everytime I run py2exe on hello.py, as you can see at  
the start of the command output:

   running py2exe
   *** searching for required modules ***
   *** parsing results ***
   creating python loader for extension 'unicodedata'
   creating python loader for extension 'zlib'
   creating python loader for extension 'bz2'
   [...]

Since the source files are always new, python recompiles the .pyc files  
each time and this changes the embedded timestamp. Can anyone give me more  
information on these files, and how could I ensure that they are always  
compiled with the same timestamp? Thanks.

py2exe@... wrote:
>> Doesn't do that on my system.  I saved library.zip that was generated
>> on 31 Aug 2006 and then created new one just now 11 Sep 2006.  Then ran
>> FC (file compare) on them and they are the same.  I suspect that you
>> are changing something that causes python to recreate at least one of
>> the .pyo files that are placed into library.zip.
> 
> That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You?
> 
> I looked deeper into the problem. Turns out the differences between my  
> various library.zip files ARE timestamps, more precisely bytes 4 to 7 of  
> the files bz2.pyc, unicodedata.pyc, and zlib.pyc inside the zips. (Same  
> problem, but with .pyo files, when I ask py2exe to optimize the compiled  
> files.) The sources for these files are regenerated on the fly (in the  
> build/ subfolder) everytime I run py2exe on hello.py, as you can see at  
> the start of the command output:
> 
>    running py2exe
>    *** searching for required modules ***
>    *** parsing results ***
>    creating python loader for extension 'unicodedata'
>    creating python loader for extension 'zlib'
>    creating python loader for extension 'bz2'
>    [...]
> 
> Since the source files are always new, python recompiles the .pyc files  
> each time and this changes the embedded timestamp. Can anyone give me more  
> information on these files, and how could I ensure that they are always  
> compiled with the same timestamp? Thanks.
> 
>
> That's odd. I'm using Python 2.4.3 on Windows XP Home SP2. You?

I'm using 2.4.1 on Windows XP Pro SP2.

When I run I see:

Creating ZBKUP binary using py2exe
-------------------------------------------------------------------
running py2exe
*** searching for required modules ***
*** parsing results ***
*** finding dlls needed ***
*** create binaries ***
*** byte compile python files ***
writing byte-compilation script 'c:\temp\tmp_oohre.py'
c:\python25\python.exe -OO c:\temp\tmp_oohre.py
skipping byte-compilation of F:\Larry\Python\Library\Utils\__init__.py to Librar
y\Utils\__init__.pyo
skipping byte-compilation of F:\Larry\Python\Library\Utils\fmt_wcommas.py to Lib
rary\Utils\fmt_wcommas.pyo
.
. Lots more of these for other files.
.

I don't know why it would redo byte-compilation of files that aren't
changing.

You probably need to post your setup.py script also.

-Larry

> You probably need to post your setup.py script also.

It's the sample used in the py2exe tutorial  
(http://www.py2exe.org/index.cgi/Tutorial). I attached it for your  
convenience.

I just run "python setup.py py2exe", rename the resulting folders (dist/  
and build/), run again, and compare the two outputs.

Thanks for your help.

On 13/09/2006 3:09 AM, py2exe@... wrote:
>> You probably need to post your setup.py script also.
> 
> It's the sample used in the py2exe tutorial 
> (http://www.py2exe.org/index.cgi/Tutorial). I attached it for your 
> convenience.
> 
> I just run "python setup.py py2exe", rename the resulting folders (dist/ 
> and build/), run again, and compare the two outputs.
> 

Maybe I've missed something earlier in the thread, or just plain ol' 
lost the plot, but:

When it's run the second time, there are no dist and build folders, so 
the setup has to create everything afresh, yes/no? Maybe you should 
*copy* the contents of the dist and build folders before the second run, 
  instead of renaming.

HTH,
John

When it's run the second time, there are no dist and build folders, so
> the setup has to create everything afresh, yes/no? Maybe you should  
> *copy* the contents of the dist and build folders before the second run,  
>   instead of renaming.

John,

Even if I don't rename or delete the build/ and dist/ folders, py2exe  
regenerates the bz2.py, unicodedata.py and zlib.py files (in build/),  
which updates their timestamps, which changes the compiled versions  
(*.pyc) included in library.zip. Same result: library.zip is different  
every time I run py2exe, even if the sources haven't changed.

py2exe@... пишет:
>   When it's run the second time, there are no dist and build folders, so
>> the setup has to create everything afresh, yes/no? Maybe you should  
>> *copy* the contents of the dist and build folders before the second run,  
>>   instead of renaming.
> 
> John,
> 
> Even if I don't rename or delete the build/ and dist/ folders, py2exe  
> regenerates the bz2.py, unicodedata.py and zlib.py files (in build/),  
> which updates their timestamps, which changes the compiled versions  
> (*.pyc) included in library.zip. Same result: library.zip is different  
> every time I run py2exe, even if the sources haven't changed.

I confirm this effect.
My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5.

I think there is something straightforward in py2exe build process: it
does not check that some files may exist and have the same content but
rebuild it every time from scratch. I think it's possible to fix this
behaviour but it's not trivial fix.

I can propose you to check library.zip not at binary level as one file,
but checking content of each file inside zip. Per example by calculating
  md5 sum for all files inside zip. It's looks somewhat ugly, but it at
least simple solution that does not require to hack py2exe itself.

--
Alexander

Attached is a quick hack I wrote so that py2exe wouldn't recreate its  
"loader" files if their contents haven't changed since the last time. This  
ensures that py2exe always creates the same output for identical intput.

Alexander Belchenko wrote:
> py2exe@... пишет:
>>   When it's run the second time, there are no dist and build folders, so
>>> the setup has to create everything afresh, yes/no? Maybe you should  
>>> *copy* the contents of the dist and build folders before the second run,  
>>>   instead of renaming.
>> John,
>>
>> Even if I don't rename or delete the build/ and dist/ folders, py2exe  
>> regenerates the bz2.py, unicodedata.py and zlib.py files (in build/),  
>> which updates their timestamps, which changes the compiled versions  
>> (*.pyc) included in library.zip. Same result: library.zip is different  
>> every time I run py2exe, even if the sources haven't changed.
> 
> I confirm this effect.
> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5.
> 
> I think there is something straightforward in py2exe build process: it
> does not check that some files may exist and have the same content but
> rebuild it every time from scratch. I think it's possible to fix this
> behaviour but it's not trivial fix.
> 
> I can propose you to check library.zip not at binary level as one file,
> but checking content of each file inside zip. Per example by calculating
>   md5 sum for all files inside zip. It's looks somewhat ugly, but it at
> least simple solution that does not require to hack py2exe itself.
> 
> --
> Alexander
> 
> 
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Py2exe-users mailing list
> Py2exe-users@...
> https://lists.sourceforge.net/lists/listinfo/py2exe-users

Whatever is going on is subtle....My unicodedata.pyd does not change.
Hey wait, you said

"...bz2.py, unicodedata.py and zlib.py files..."

My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or
bz2.py what's up with that?  On my system these are all .pyd files and
they are not generated as they are binary extensions.

-Larry

On 14/09/2006 8:55 AM, Larry Bates wrote:
> Alexander Belchenko wrote:
>> py2exe@... =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
>>>   When it's run the second time, there are no dist and build folders,=
 so
>>>> the setup has to create everything afresh, yes/no? Maybe you should =
=20
>>>> *copy* the contents of the dist and build folders before the second =
run, =20
>>>>   instead of renaming.
>>> John,
>>>
>>> Even if I don't rename or delete the build/ and dist/ folders, py2exe=
 =20
>>> regenerates the bz2.py, unicodedata.py and zlib.py files (in build/),=
 =20
>>> which updates their timestamps, which changes the compiled versions =20
>>> (*.pyc) included in library.zip. Same result: library.zip is differen=
t =20
>>> every time I run py2exe, even if the sources haven't changed.
>> I confirm this effect.
>> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5.
>>
>> I think there is something straightforward in py2exe build process: it
>> does not check that some files may exist and have the same content but
>> rebuild it every time from scratch. I think it's possible to fix this
>> behaviour but it's not trivial fix.
>>
>> I can propose you to check library.zip not at binary level as one file=
,
>> but checking content of each file inside zip. Per example by calculati=
ng
>>   md5 sum for all files inside zip. It's looks somewhat ugly, but it a=
t
>> least simple solution that does not require to hack py2exe itself.
>>
>> --
>> Alexander
>>

> Whatever is going on is subtle....My unicodedata.pyd does not change.
> Hey wait, you said
>=20
> "...bz2.py, unicodedata.py and zlib.py files..."
>=20
> My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or
> bz2.py what's up with that?  On my system these are all .pyd files and
> they are not generated as they are binary extensions.
>=20
> -Larry

The library.zip includes (at the end) 3 files bz2.pyc etc. The dist=20
folder includes 3 files bz2.pyd etc. My guess is that py2exe=20
manufactures a "wrapper" foo.py on the fly for each foo.pyd.

I guess the OP will just have to stop pressing the build button when=20
nothing has changed :-)

John Machin wrote:
> On 14/09/2006 8:55 AM, Larry Bates wrote:
>> Alexander Belchenko wrote:
>>> py2exe@... =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
>>>>   When it's run the second time, there are no dist and build
>>>> folders, so
>>>>> the setup has to create everything afresh, yes/no? Maybe you
>>>>> should  *copy* the contents of the dist and build folders before
>>>>> the second run,    instead of renaming.
>>>> John,
>>>>
>>>> Even if I don't rename or delete the build/ and dist/ folders,
>>>> py2exe  regenerates the bz2.py, unicodedata.py and zlib.py files
>>>> (in build/),  which updates their timestamps, which changes the
>>>> compiled versions  (*.pyc) included in library.zip. Same result:
>>>> library.zip is different  every time I run py2exe, even if the
>>>> sources haven't changed.
>>> I confirm this effect.
>>> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5.
>>>
>>> I think there is something straightforward in py2exe build process: i=
t
>>> does not check that some files may exist and have the same content bu=
t
>>> rebuild it every time from scratch. I think it's possible to fix this
>>> behaviour but it's not trivial fix.
>>>
>>> I can propose you to check library.zip not at binary level as one fil=
e,
>>> but checking content of each file inside zip. Per example by
>>> calculating
>>>   md5 sum for all files inside zip. It's looks somewhat ugly, but it =
at
>>> least simple solution that does not require to hack py2exe itself.
>>>
>>> --=20
>>> Alexander
>>>
>
>> Whatever is going on is subtle....My unicodedata.pyd does not change.
>> Hey wait, you said
>>
>> "...bz2.py, unicodedata.py and zlib.py files..."
>>
>> My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or
>> bz2.py what's up with that?  On my system these are all .pyd files and
>> they are not generated as they are binary extensions.
>>
>> -Larry
>
> The library.zip includes (at the end) 3 files bz2.pyc etc. The dist
> folder includes 3 files bz2.pyd etc. My guess is that py2exe
> manufactures a "wrapper" foo.py on the fly for each foo.pyd.
>
> I guess the OP will just have to stop pressing the build button when
> nothing has changed :-)
>
>
>
>
I don't have any .pyc files in my library file.  I do however use
"bundle_files": 2 in my options in the
setup.py file.  This places the .pyd files inside the library file.=20
Maybe that's the "secret".

-Larry

On 14/09/2006 12:23 PM, Larry Bates wrote:
> John Machin wrote:
>> On 14/09/2006 8:55 AM, Larry Bates wrote:
>>> Alexander Belchenko wrote:
>>>> py2exe@... =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
>>>>>   When it's run the second time, there are no dist and build
>>>>> folders, so
>>>>>> the setup has to create everything afresh, yes/no? Maybe you
>>>>>> should  *copy* the contents of the dist and build folders before
>>>>>> the second run,    instead of renaming.
>>>>> John,
>>>>>
>>>>> Even if I don't rename or delete the build/ and dist/ folders,
>>>>> py2exe  regenerates the bz2.py, unicodedata.py and zlib.py files
>>>>> (in build/),  which updates their timestamps, which changes the
>>>>> compiled versions  (*.pyc) included in library.zip. Same result:
>>>>> library.zip is different  every time I run py2exe, even if the
>>>>> sources haven't changed.
>>>> I confirm this effect.
>>>> My system is Python 2.4.3 on Windows 2000; py2exe 0.6.5.
>>>>
>>>> I think there is something straightforward in py2exe build process: =
it
>>>> does not check that some files may exist and have the same content b=
ut
>>>> rebuild it every time from scratch. I think it's possible to fix thi=
s
>>>> behaviour but it's not trivial fix.
>>>>
>>>> I can propose you to check library.zip not at binary level as one fi=
le,
>>>> but checking content of each file inside zip. Per example by
>>>> calculating
>>>>   md5 sum for all files inside zip. It's looks somewhat ugly, but it=
 at
>>>> least simple solution that does not require to hack py2exe itself.
>>>>
>>>> --=20
>>>> Alexander
>>>>
>>> Whatever is going on is subtle....My unicodedata.pyd does not change.
>>> Hey wait, you said
>>>
>>> "...bz2.py, unicodedata.py and zlib.py files..."
>>>
>>> My Python 2.4 doesn't have a file named unicodedata.py, zlib.py, or
>>> bz2.py what's up with that?  On my system these are all .pyd files an=
d
>>> they are not generated as they are binary extensions.
>>>
>>> -Larry
>> The library.zip includes (at the end) 3 files bz2.pyc etc. The dist
>> folder includes 3 files bz2.pyd etc. My guess is that py2exe
>> manufactures a "wrapper" foo.py on the fly for each foo.pyd.
>>
>> I guess the OP will just have to stop pressing the build button when
>> nothing has changed :-)
>>
>>
>>
>>
> I don't have any .pyc files in my library file.  I do however use
> "bundle_files": 2 in my options in the
> setup.py file.  This places the .pyd files inside the library file.=20
> Maybe that's the "secret".
>=20

Could well be; I was emulating the OP: bog-standard no-options hello.py=20
and setup.py as per the tutorial.