cx-freeze-users Mailing List for cx_Freeze (Page 116)
Brought to you by:
atuining
You can subscribe to this list here.
2003 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
(1) |
Sep
(2) |
Oct
(11) |
Nov
|
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2004 |
Jan
(1) |
Feb
(2) |
Mar
(3) |
Apr
(9) |
May
(5) |
Jun
(18) |
Jul
(21) |
Aug
(55) |
Sep
(15) |
Oct
(8) |
Nov
(27) |
Dec
(19) |
2005 |
Jan
(47) |
Feb
(20) |
Mar
(48) |
Apr
(6) |
May
(6) |
Jun
(34) |
Jul
(28) |
Aug
(10) |
Sep
(20) |
Oct
|
Nov
(20) |
Dec
(41) |
2006 |
Jan
(2) |
Feb
(4) |
Mar
(26) |
Apr
|
May
(5) |
Jun
(13) |
Jul
(11) |
Aug
(24) |
Sep
(11) |
Oct
(2) |
Nov
(8) |
Dec
(8) |
2007 |
Jan
(13) |
Feb
(10) |
Mar
|
Apr
(16) |
May
(22) |
Jun
|
Jul
|
Aug
|
Sep
(10) |
Oct
(20) |
Nov
|
Dec
|
2008 |
Jan
(13) |
Feb
|
Mar
(27) |
Apr
(5) |
May
(15) |
Jun
(7) |
Jul
(36) |
Aug
(11) |
Sep
(12) |
Oct
(4) |
Nov
(3) |
Dec
(8) |
2009 |
Jan
(12) |
Feb
(5) |
Mar
(13) |
Apr
(16) |
May
(6) |
Jun
(17) |
Jul
(24) |
Aug
|
Sep
(30) |
Oct
(6) |
Nov
(32) |
Dec
(35) |
2010 |
Jan
(41) |
Feb
(54) |
Mar
(77) |
Apr
(42) |
May
(9) |
Jun
(6) |
Jul
(78) |
Aug
(54) |
Sep
(36) |
Oct
(8) |
Nov
(52) |
Dec
(44) |
2011 |
Jan
(13) |
Feb
(16) |
Mar
(48) |
Apr
(13) |
May
(13) |
Jun
(29) |
Jul
(26) |
Aug
(9) |
Sep
(21) |
Oct
(25) |
Nov
(9) |
Dec
(9) |
2012 |
Jan
(9) |
Feb
(47) |
Mar
(36) |
Apr
(59) |
May
(29) |
Jun
(27) |
Jul
(60) |
Aug
(20) |
Sep
(29) |
Oct
(25) |
Nov
(44) |
Dec
(6) |
2013 |
Jan
(19) |
Feb
(39) |
Mar
(23) |
Apr
(79) |
May
(41) |
Jun
(34) |
Jul
(80) |
Aug
(24) |
Sep
(54) |
Oct
(42) |
Nov
(34) |
Dec
(7) |
2014 |
Jan
(53) |
Feb
(17) |
Mar
(18) |
Apr
(22) |
May
(34) |
Jun
(6) |
Jul
(9) |
Aug
(20) |
Sep
(17) |
Oct
(22) |
Nov
(5) |
Dec
(13) |
2015 |
Jan
(9) |
Feb
(23) |
Mar
(11) |
Apr
(11) |
May
(3) |
Jun
(17) |
Jul
(3) |
Aug
(7) |
Sep
(3) |
Oct
(12) |
Nov
(15) |
Dec
(3) |
2016 |
Jan
|
Feb
(8) |
Mar
(14) |
Apr
(11) |
May
(3) |
Jun
(11) |
Jul
(1) |
Aug
(53) |
Sep
|
Oct
(3) |
Nov
(50) |
Dec
(16) |
2017 |
Jan
(14) |
Feb
(8) |
Mar
(2) |
Apr
(9) |
May
(15) |
Jun
(4) |
Jul
(5) |
Aug
(10) |
Sep
(1) |
Oct
(3) |
Nov
(10) |
Dec
(3) |
2018 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
(7) |
Dec
(1) |
2019 |
Jan
|
Feb
(2) |
Mar
|
Apr
(2) |
May
(1) |
Jun
|
Jul
|
Aug
(1) |
Sep
(14) |
Oct
(2) |
Nov
|
Dec
|
2020 |
Jan
(1) |
Feb
|
Mar
|
Apr
(3) |
May
|
Jun
(4) |
Jul
(2) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(3) |
2021 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Roger B. <ro...@ro...> - 2005-06-16 16:52:05
|
> So, does this mean you would prefer the default to be uncompressed or > that you would be using the --no-compress option? Or something else? It means that I will be using uncompressed zips no matter what default you pick. As for what the default should be, that should depend primarily on what you think concerns people the most - install size or download size. Personally I think there is less to go wrong if uncompressed (no zlib dependency) and people are freezing stuff to give to others so download size would be what I think matters to most. Roger |
From: Anthony T. <ant...@gm...> - 2005-06-16 15:24:27
|
So, does this mean you would prefer the default to be uncompressed or that you would be using the --no-compress option? Or something else? On 6/15/05, Roger Binns <ro...@ro...> wrote: > > I'll try to get some real numbers this evening to see what > > difference it makes. >=20 > Here they are, based on the bytecode shipped with BitPim: >=20 > 9863155 uncomp.zip Uncompressed in a zip > 2053295 comp.zip zip -9 > 2021580 comp.zip.bz2 bz2 of the zip -9 > 2009924 comp.zip.gz gzip -9 of the zip -9 > 1789575 uncomp.zip.gz gzip -9 of the uncompressed > 1353763 uncomp.zip.bz2 bz2 of the uncompressed >=20 > There is about 10% improvement in not compressing the zip before > compressing overall with gzip. It is closer to 40% if the overall > container is compressed with bzip2. I believe rpms use the latter > so it would make a non-trivial difference. >=20 > Roger >=20 >=20 > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=3D7477&alloc_id=3D16492&op=3Dclic= k > _______________________________________________ > cx-freeze-users mailing list > cx-...@li... > https://lists.sourceforge.net/lists/listinfo/cx-freeze-users > |
From: Roger B. <ro...@ro...> - 2005-06-16 04:23:21
|
> I'll try to get some real numbers this evening to see what > difference it makes. Here they are, based on the bytecode shipped with BitPim: 9863155 uncomp.zip Uncompressed in a zip 2053295 comp.zip zip -9 2021580 comp.zip.bz2 bz2 of the zip -9 2009924 comp.zip.gz gzip -9 of the zip -9 1789575 uncomp.zip.gz gzip -9 of the uncompressed 1353763 uncomp.zip.bz2 bz2 of the uncompressed There is about 10% improvement in not compressing the zip before compressing overall with gzip. It is closer to 40% if the overall container is compressed with bzip2. I believe rpms use the latter so it would make a non-trivial difference. Roger |
From: Roger B. <ro...@ro...> - 2005-06-15 20:17:47
|
> I don't see any reason to even offer uncompressed zipfiles. Space. zipfiles compress each member file individually. A zip file where each member is uncompressed but then the whole is compressed (such as if inside an rpm or similar distribution format) will be smaller. I tend to prefer smaller distributions even if what is installed is larger. I'll try to get some real numbers this evening to see what difference it makes. Roger |
From: Anthony T. <ant...@gm...> - 2005-06-15 20:12:05
|
On 6/15/05, Ralf Schmitt <ra...@br...> wrote: > Anthony Tuininga wrote: > > Strange. I just turned on zipfile compress for the members and it > > seems to work just fine for me. I've just checked in code that does > > just that and adds a "-c" or "--compress" option to allow you to turn > > it on. I'm assuming that most people would prefer the compression off > > by default? If not, speak up! >=20 > +1 for compressed zipfiles. Ok, that's 2 for defaulting to compressed zip files and none against. Anyone else? If the desire is for defaulting to compressing, I'll rename the option to --no-compress unless there are any objections. > > On 6/15/05, Anthony Tuininga <ant...@gm...> wrote: > > > >>I'll take a look into this when I get a chance. Thanks for pointing it = out. > >> > >>On 6/7/05, Ralf Schmitt <ra...@br...> wrote: > >> > >>>Ralf Schmitt wrote: > >>> > >>>>In Freezer.py the zipfile is created with > >>>>compression=3Dzipfile.ZIP_DEFLATED. But the zipfile module doesn't > >>>>compress members added with writestr if they don't have the right > >>>>'compress_type' (at least when using python 2.4). Guess this isn't > >>>>intended behaviour. > >>>> > >>> > >>>Of course, turning on zipfile compression, doesn't work without > >>>problems. One needs the zlib module, and even if it is installed, the > >>>frozen executable sometimes can't load it, and sometimes can load it, > >>>depending on the way the frozen executable is called. This is a proble= m > >>>with sys.path not being cleared before the initscript being called, >=20 > Try calling that script as dist/foobar, ./foobar, and foobar (with dist > being in path). One of them doesn't work...I can't remember which one ;) It was the one with the binary being in the path, as it turns out. Its rather strange that if a directory is included at all, then Python adds the directory in which the executable is found to the path and otherwise it does not. In any case, I've added code to explicitly do that in all situations and that seems to get around the problem quite nicely. > >>>maybe it should even be fixed, if one doesn't use zipfile compression. > >>>The attached patch solves that problem for me. > >>> >=20 >=20 > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=3D7477&alloc_id=3D16492&op=3Dclic= k > _______________________________________________ > cx-freeze-users mailing list > cx-...@li... > https://lists.sourceforge.net/lists/listinfo/cx-freeze-users > |
From: Ralf S. <ra...@br...> - 2005-06-15 19:40:01
|
Anthony Tuininga wrote: > Apologies for the delay in responding. I just got back from holidays. > The ability to put a number of scripts in one is quite doable without > any patching whatsoever. The patch you provided (as Bob noted) is > quite large and it is hard to determine which pieces are necessary and > which are still works in progress. If you want me to give you an idea > of how to do it without patching, let me know. If you have something I think the true and only way it should work, is calling FreezePython with multiple scripts. I can also think of other ways to do it, but none seems as easy as the above method (at least for those just using cx_Freeze). > very specific that ought to be changed to support what you are trying > to do, let me know that as well. I am quite happy to accept patches > and suggestions -- providing I am able to understand them. Thanks. > What I really missed, was an API to cx_Freeze and with that the ability to easily modify behaviour by subclassing. However the nice thing about cx_Freeze is that one can easily understand it's code, and so I have no problems tracking changes you make. - Ralf |
From: Ralf S. <ra...@br...> - 2005-06-15 18:43:38
|
Anthony Tuininga wrote: > Strange. I just turned on zipfile compress for the members and it > seems to work just fine for me. I've just checked in code that does > just that and adds a "-c" or "--compress" option to allow you to turn > it on. I'm assuming that most people would prefer the compression off > by default? If not, speak up! +1 for compressed zipfiles. > > On 6/15/05, Anthony Tuininga <ant...@gm...> wrote: > >>I'll take a look into this when I get a chance. Thanks for pointing it out. >> >>On 6/7/05, Ralf Schmitt <ra...@br...> wrote: >> >>>Ralf Schmitt wrote: >>> >>>>In Freezer.py the zipfile is created with >>>>compression=zipfile.ZIP_DEFLATED. But the zipfile module doesn't >>>>compress members added with writestr if they don't have the right >>>>'compress_type' (at least when using python 2.4). Guess this isn't >>>>intended behaviour. >>>> >>> >>>Of course, turning on zipfile compression, doesn't work without >>>problems. One needs the zlib module, and even if it is installed, the >>>frozen executable sometimes can't load it, and sometimes can load it, >>>depending on the way the frozen executable is called. This is a problem >>>with sys.path not being cleared before the initscript being called, Try calling that script as dist/foobar, ./foobar, and foobar (with dist being in path). One of them doesn't work...I can't remember which one ;) >>>maybe it should even be fixed, if one doesn't use zipfile compression. >>>The attached patch solves that problem for me. >>> |
From: Bob I. <bo...@re...> - 2005-06-15 18:10:56
|
I would imagine that compressed zipfiles are faster than uncompressed =20= ones, given the disparity between CPU speed and IO, and the =20 compressibility of Python bytecode. I don't see any reason to even =20 offer uncompressed zipfiles. The only downside is that it introduces a dependency on zlib.so -- =20 but all non-trivial applications are going to depend on several =20 extensions anyway. -bob On Jun 15, 2005, at 2:04 PM, Anthony Tuininga wrote: > Strange. I just turned on zipfile compress for the members and it > seems to work just fine for me. I've just checked in code that does > just that and adds a "-c" or "--compress" option to allow you to turn > it on. I'm assuming that most people would prefer the compression off > by default? If not, speak up! > > On 6/15/05, Anthony Tuininga <ant...@gm...> wrote: > >> I'll take a look into this when I get a chance. Thanks for =20 >> pointing it out. >> >> On 6/7/05, Ralf Schmitt <ra...@br...> wrote: >> >>> Ralf Schmitt wrote: >>> >>>> In Freezer.py the zipfile is created with >>>> compression=3Dzipfile.ZIP_DEFLATED. But the zipfile module doesn't >>>> compress members added with writestr if they don't have the right >>>> 'compress_type' (at least when using python 2.4). Guess this isn't >>>> intended behaviour. >>>> >>>> >>> >>> Of course, turning on zipfile compression, doesn't work without >>> problems. One needs the zlib module, and even if it is installed, =20= >>> the >>> frozen executable sometimes can't load it, and sometimes can load =20= >>> it, >>> depending on the way the frozen executable is called. This is a =20 >>> problem >>> with sys.path not being cleared before the initscript being called, >>> maybe it should even be fixed, if one doesn't use zipfile =20 >>> compression. >>> The attached patch solves that problem for me. >>> >>> - Ralf >>> >>> > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id=16492&op=3Dclick > _______________________________________________ > cx-freeze-users mailing list > cx-...@li... > https://lists.sourceforge.net/lists/listinfo/cx-freeze-users > |
From: Anthony T. <ant...@gm...> - 2005-06-15 18:04:45
|
Strange. I just turned on zipfile compress for the members and it seems to work just fine for me. I've just checked in code that does just that and adds a "-c" or "--compress" option to allow you to turn it on. I'm assuming that most people would prefer the compression off by default? If not, speak up! On 6/15/05, Anthony Tuininga <ant...@gm...> wrote: > I'll take a look into this when I get a chance. Thanks for pointing it ou= t. >=20 > On 6/7/05, Ralf Schmitt <ra...@br...> wrote: > > Ralf Schmitt wrote: > > > In Freezer.py the zipfile is created with > > > compression=3Dzipfile.ZIP_DEFLATED. But the zipfile module doesn't > > > compress members added with writestr if they don't have the right > > > 'compress_type' (at least when using python 2.4). Guess this isn't > > > intended behaviour. > > > > > > > Of course, turning on zipfile compression, doesn't work without > > problems. One needs the zlib module, and even if it is installed, the > > frozen executable sometimes can't load it, and sometimes can load it, > > depending on the way the frozen executable is called. This is a problem > > with sys.path not being cleared before the initscript being called, > > maybe it should even be fixed, if one doesn't use zipfile compression. > > The attached patch solves that problem for me. > > > > - Ralf > > |
From: Anthony T. <ant...@gm...> - 2005-06-15 17:45:22
|
I'll take a look into this when I get a chance. Thanks for pointing it out. On 6/7/05, Ralf Schmitt <ra...@br...> wrote: > Ralf Schmitt wrote: > > In Freezer.py the zipfile is created with > > compression=3Dzipfile.ZIP_DEFLATED. But the zipfile module doesn't > > compress members added with writestr if they don't have the right > > 'compress_type' (at least when using python 2.4). Guess this isn't > > intended behaviour. > > >=20 > Of course, turning on zipfile compression, doesn't work without > problems. One needs the zlib module, and even if it is installed, the > frozen executable sometimes can't load it, and sometimes can load it, > depending on the way the frozen executable is called. This is a problem > with sys.path not being cleared before the initscript being called, > maybe it should even be fixed, if one doesn't use zipfile compression. > The attached patch solves that problem for me. >=20 > - Ralf >=20 >=20 >=20 >=20 > Index: Console.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Console.c (revision 17719) > +++ Console.c (revision 17745) > @@ -53,11 +53,29 @@ > Py_Initialize(); > PySys_SetArgv(argc, argv); >=20 > + // PyRun_SimpleString("import sys; print 'sys.path after PySys_SetAr= gv:', sys.path"); > + > + // sys.path isn't as empty as one might hope. when using zipfile com= pression > + // the zlib module must be loaded. without cleaning sys.path here, i= t did happen > + // that trying to run a frozen executable sometimes did, and sometim= es did not find > + // zlib depending on the way you called the frozen executable (i.e. = 'dist/foobar' > + // vs 'foobar'). When cleaning sys.path with the following code, it = will always > + // fail :) > + > + if (PyRun_SimpleString("import sys; del sys.path[:]")) { > + goto error; > + } > + > // do the work > - if (ExecuteScript(fileName) < 0) > - return 1; > + if (ExecuteScript(fileName) < 0) { > + goto error; > + } >=20 > Py_Finalize(); > return 0; > + > + error: > + Py_Finalize(); > + return 1; > } >=20 > Index: Freezer.py > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Freezer.py (revision 17719) > +++ Freezer.py (revision 17745) > @@ -72,7 +72,7 @@ > return os.path.join(baseDir, subDir, name + ext) >=20 >=20 > -def Freeze(targetFileName, moduleFinder, zipIncludes): > +def Freeze(targetFileName, moduleFinder, zipIncludes, useCompression=3DT= rue): > """Freeze the modules found by the finder to the end of the file.""" > outFile =3D zipfile.PyZipFile(targetFileName, "a", zipfile.ZIP_DEFLA= TED) > mods =3D [] > @@ -95,7 +95,8 @@ > data =3D imp.get_magic() + struct.pack("<i", mtime) + \ > marshal.dumps(module.__code__) > zinfo =3D zipfile.ZipInfo(fileName + ".pyc", zipTime) > - zinfo.compress_type =3D zipfile.ZIP_DEFLATED > + if useCompression: > + zinfo.compress_type =3D zipfile.ZIP_DEFLATED > outFile.writestr(zinfo, data) > for spec in zipIncludes: > if '=3D' in spec: > Index: mycxfreeze.py > Index: Common.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Common.c (revision 17719) > +++ Common.c (revision 17745) > @@ -6,6 +6,16 @@ > #include <compile.h> > #include <eval.h> >=20 > +#include <limits.h> > + > +#ifdef HAVE_SYS_PARAM_H > +#include <sys/param.h> > +#endif > + > +#ifdef HAVE_STDLIB_H > +#include <stdlib.h> > +#endif > + > //----------------------------------------------------------------------= ------- > // ExecuteScript() > // Execute the script found within the file. > @@ -15,6 +25,10 @@ > { > PyObject *module, *importer, *code, *dict, *pathList, *temp; > int result; > + const char *resolved_path =3D fileName; > +#ifdef HAVE_REALPATH > + static char buffer[PATH_MAX+1]; > +#endif >=20 > #ifdef __WIN32__ > char dllName[PATH_MAX + 1]; > @@ -41,6 +55,13 @@ > return FatalError("cannot set dll name in module sys"); > #endif >=20 > +#ifdef HAVE_REALPATH > + if (realpath(fileName, buffer)) { > + resolved_path =3D buffer; > + } > +#endif > + > + > // add the file to sys.path > pathList =3D PySys_GetObject("path"); > if (!pathList) > @@ -53,6 +74,24 @@ > if (result < 0) > return FatalError("cannot insert file name in sys.path"); >=20 > + temp =3D PyString_FromString(resolved_path); > + if (!temp) > + return FatalError("cannot create Python string"); > + PySys_SetObject("realexecutable", temp); > + Py_DECREF(temp); > + > + > + if (PyRun_SimpleString( > + "import sys\n" > + "sep =3D sys.platform=3D=3D'win32' and '\\\\' or '/'\n" > + "sys.path.append(sys.realexecutable[:sys.realexecutable.r= find(sep)])\n" > + // "del sys.realexecutable\n" > + //"print sys.path, sys.executable, sys.realexecutable\n" > + )) { > + // error > + return -1; > + } > + > // load and execute initscript > module =3D PyImport_ImportModule("zipimport"); > if (!module) > Index: initscripts/MyConsole.py > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- initscripts/MyConsole.py (revision 17719) > +++ initscripts/MyConsole.py (revision 17745) > @@ -6,19 +6,11 @@ > # the Win32 extensions behave as expected. > #-----------------------------------------------------------------------= ------- >=20 > -import os > import sys > -import warnings > +import os > import zipimport >=20 > -fileName =3D sys.path[0] > -while os.path.islink(fileName): > - fileName =3D os.path.normpath(os.path.join(os.path.dirname(fileName)= , > - os.readlink(fileName))) > -dirName =3D os.path.dirname(fileName) > - > sys.frozen =3D True > -sys.path =3D [fileName, dirName] >=20 > try: > import encodings > @@ -30,8 +22,7 @@ > exe =3D exe[:-4] >=20 > m =3D __import__("__main__") > -importer =3D zipimport.zipimporter(fileName) > +importer =3D zipimport.zipimporter(sys.executable) > code =3D importer.get_code("__main__%s__" % exe) > m.__dict__['__file__'] =3D exe > exec code in m.__dict__ > - >=20 >=20 > |
From: Anthony T. <ant...@gm...> - 2005-06-15 17:03:19
|
Apologies for the delay in responding. I just got back from holidays. The ability to put a number of scripts in one is quite doable without any patching whatsoever. The patch you provided (as Bob noted) is quite large and it is hard to determine which pieces are necessary and which are still works in progress. If you want me to give you an idea of how to do it without patching, let me know. If you have something very specific that ought to be changed to support what you are trying to do, let me know that as well. I am quite happy to accept patches and suggestions -- providing I am able to understand them. Thanks. On 6/1/05, Ralf Schmitt <ra...@br...> wrote: > Hi all, >=20 > I have modified cx_Freeze to generate one executable from multiple > python scripts. It collects all dependencies of all scripts into one > executable with an attached zip file, saves each scripts bytecode with a > special name and dispatches on the name the executable is called with > (i.e. one creates symlinks or hardlinks to that executable for each > python script). > Is it possible to include such functionality into cx_Freeze? > What I am also missing is a cx_Freeze API, i.e. FreezePython's > functionality as a Module (my patch could have been shorter, if > cx_Freeze did provide such an API). >=20 > - Ralf >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > Index: myfreeze > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- myfreeze (revision 0) > +++ myfreeze (revision 17687) > @@ -0,0 +1,10 @@ > +#! /usr/bin/env python > + > +import sys > + > +if __name__=3D=3D'__main__': > + if len(sys.argv)=3D=3D1: > + print "usage: myfreeze script1 [script2 ...]" > + else: > + from mycxfreeze import freeze > + freeze(sys.argv[1:], includes=3D['encodings.*']) >=20 > Property changes on: myfreeze > ___________________________________________________________________ > Name: svn:executable > + * >=20 > Index: mycxfreeze.py > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- mycxfreeze.py (revision 0) > +++ mycxfreeze.py (revision 17687) > @@ -0,0 +1,135 @@ > +#! /usr/bin/env python > + > +import sys > +import os > +import re > +import shutil > +import sets > + > +import Freezer > +import ModuleFinder > + > +def getDependenciesLDD(path): > + s=3Dos.popen4("ldd '%s'" % path)[1].read() > + res =3D re.compile(r"^ *.* =3D> (.*) \(.*", re.MULTILINE).findall(s) > + return res > + > +def getDependenciesMach(path): > + s=3Dos.popen4("otool -L '%s'" % path)[1].read() > + res =3D re.compile(r"^\s*(.*) \(.*", re.MULTILINE).findall(s) > + print "DEPS:", path, res > + > + return res > + > + > +def getDependencies(paths): > + if sys.platform=3D=3D'win32': > + return [] > + > + > + if sys.platform=3D=3D'darwin': > + getdeps =3D getDependenciesMach > + else: > + getdeps =3D getDependenciesLDD > + > + res =3D sets.Set() > + for p in paths: > + res.update(getdeps(p)) > + return list(res) > + > + > +def freeze(scripts, targetdir=3D'dist', excludes =3D [], includes=3D[], = showReport=3DFalse): > + defaultName =3D "MyConsole.py" > + initscript =3D Freezer.FullFileName("initscripts", None, defaultName= ) > + > + defaultName =3D "Console" > + if sys.platform =3D=3D "win32": > + defaultName +=3D ".exe" > + baseName =3D Freezer.FullFileName("bases", None, defaultName) > + > + > + library =3D os.path.join(targetdir, 'library.zip') > + > + > + finder =3D ModuleFinder.ModuleFinder(excludes, {}) > + for x in includes: > + if x.endswith('.*'): > + finder.import_hook(x[:-2], fromlist=3D'*') > + else: > + finder.import_hook(x) > + > + for x in scripts: > + dirpath =3D os.path.dirname(os.path.abspath(x)) > + sys.path.insert(0, dirpath) > + print "analyzing", x > + > + finder.run_script(x) > + > + bn =3D os.path.basename(x) > + if bn.endswith('.py'): > + bn =3D bn[:-3] > + finder.modules['__main__%s__' % bn] =3D finder.modules['__main__= '] > + del finder.modules['__main__'] > + > + sys.path.remove(dirpath) > + > + > + finder.load_file(initscript, "cx_Freeze__init__") > + Freezer.CreateExtensionLoaders(finder) > + > + > + if showReport: > + finder.report() > + > + oldstdout =3D sys.stdout > + sys.stdout =3D open('%s-report.txt' % os.path.normpath(targetdir), '= wb') > + try: > + finder.report() > + > + # start the freezing process > + if not os.path.exists(targetdir): > + os.makedirs(targetdir) > + > + if os.path.exists(library): > + os.chmod(library, 0777) > + os.remove(library) > + > + > + shutil.copy(baseName, library) > + Freezer.Freeze(library, finder, []) > + > + sharedLibName =3D getattr(sys, "dllname", None) > + if sharedLibName is not None: > + finder.AddDependentFile(sharedLibName) > + > + bindeps =3D getDependencies(finder.dependentFiles.values()+[base= Name]) > + for x in bindeps: > + t =3D os.path.join(targetdir, os.path.basename(x)) > + if os.path.exists(t): > + os.chmod(t, 0777) > + os.remove(t) > + > + shutil.copy2(x, t) > + > + finder.CopyDependentFiles(targetdir, None) > + > + if sys.platform=3D=3D'win32': > + import bbutils.miscos.link # set os.link > + > + for x in scripts: > + if x.endswith(".py"): > + x =3D x[:-3] > + dst =3D os.path.join(targetdir, os.path.basename(x)) > + if sys.platform=3D=3D'win32': > + dst +=3D '.exe' > + if os.path.exists(dst): > + os.unlink(dst) > + os.link(library, dst) > + finally: > + sys.stdout =3D oldstdout > + > +if __name__=3D=3D'__main__': > + if len(sys.argv)=3D=3D1: > + print "usage: mycxfreeze script1 [script2 ...]" > + else: > + freeze(sys.argv[1:]) >=20 > Property changes on: mycxfreeze.py > ___________________________________________________________________ > Name: svn:executable > + * >=20 > Index: Freezer.py > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Freezer.py (revision 17637) > +++ Freezer.py (revision 17687) > @@ -67,16 +67,24 @@ > baseDir =3D os.path.dirname(sys.path[0]) > else: > baseDir =3D os.path.dirname(os.path.abspath(sys.argv[0])) > + > + baseDir =3D os.path.dirname(__file__) > return os.path.join(baseDir, subDir, name + ext) >=20 >=20 > def Freeze(targetFileName, moduleFinder, zipIncludes): > """Freeze the modules found by the finder to the end of the file.""" > outFile =3D zipfile.PyZipFile(targetFileName, "a", zipfile.ZIP_DEFLA= TED) > + mods =3D [] > for name, module in moduleFinder.modules.iteritems(): > if module.__code__ is None: > continue > fileName =3D "/".join(name.split(".")) > + mods.append((fileName, name, module)) > + > + mods.sort() > + > + for fileName, name, module in mods: > if module.__path__: > fileName +=3D "/__init__" > if os.path.exists(module.__file__): > Index: initscripts/MyConsole.py > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- initscripts/MyConsole.py (revision 0) > +++ initscripts/MyConsole.py (revision 17687) > @@ -0,0 +1,37 @@ > +#-----------------------------------------------------------------------= ------- > +# Console.py > +# Initialization script for cx_Freeze which manipulates the path so th= at the > +# directory in which the executable is found is searched for extensions = but > +# no other directory is searched. It also sets the attribute sys.frozen = so that > +# the Win32 extensions behave as expected. > +#-----------------------------------------------------------------------= ------- > + > +import os > +import sys > +import warnings > +import zipimport > + > +fileName =3D sys.path[0] > +while os.path.islink(fileName): > + fileName =3D os.path.normpath(os.path.join(os.path.dirname(fileName)= , > + os.readlink(fileName))) > +dirName =3D os.path.dirname(fileName) > + > +sys.frozen =3D True > +sys.path =3D [fileName, dirName] > + > +try: > + import encodings > +except ImportError: > + pass > + > +exe =3D os.path.basename(sys.executable) > +if exe.endswith(".exe"): > + exe =3D exe[:-4] > + > +m =3D __import__("__main__") > +importer =3D zipimport.zipimporter(fileName) > +code =3D importer.get_code("__main__%s__" % exe) > +m.__dict__['__file__'] =3D exe > +exec code in m.__dict__ > + >=20 >=20 > |
From: Ralf S. <ra...@br...> - 2005-06-07 08:45:55
|
Ralf Schmitt wrote: > In Freezer.py the zipfile is created with > compression=zipfile.ZIP_DEFLATED. But the zipfile module doesn't > compress members added with writestr if they don't have the right > 'compress_type' (at least when using python 2.4). Guess this isn't > intended behaviour. > Of course, turning on zipfile compression, doesn't work without problems. One needs the zlib module, and even if it is installed, the frozen executable sometimes can't load it, and sometimes can load it, depending on the way the frozen executable is called. This is a problem with sys.path not being cleared before the initscript being called, maybe it should even be fixed, if one doesn't use zipfile compression. The attached patch solves that problem for me. - Ralf |
From: Ralf S. <ra...@br...> - 2005-06-03 10:54:15
|
In Freezer.py the zipfile is created with compression=zipfile.ZIP_DEFLATED. But the zipfile module doesn't compress members added with writestr if they don't have the right 'compress_type' (at least when using python 2.4). Guess this isn't intended behaviour. Index: Freezer.py =================================================================== --- Freezer.py (revision 17714) +++ Freezer.py (working copy) @@ -95,6 +95,7 @@ data = imp.get_magic() + struct.pack("<i", mtime) + \ marshal.dumps(module.__code__) zinfo = zipfile.ZipInfo(fileName + ".pyc", zipTime) + zinfo.compress_type = zipfile.ZIP_DEFLATED outFile.writestr(zinfo, data) for spec in zipIncludes: if '=' in spec: |
From: Bob I. <bo...@re...> - 2005-06-02 17:01:35
|
On Jun 2, 2005, at 2:09 AM, Ralf Schmitt wrote: > Bob Ippolito wrote: > >>> >>> >>> I can't follow your arguments: >>> mcmillan installer has automatic dependency tracking. It uses >>> sensible defaults (i.e. excluding /lib, /usr/lib and a bunch of >>> dlls on windows) and it did work for us without any problems in >>> 99% of all cases. I'd rather manually care for that last 1 >>> percent than doing it every time. >>> >> If you're using something like Debian, everything you install >> with apt will be in /usr/lib... so how do you know what to ship? > > Look no further than one paragraph above. Skip everything from / > lib, and /usr/lib and include all libraries found elsewhere. My point was that if you're using apt to install everything, then skipping everything from /lib and /usr/lib will skip everything. There will not be any libraries found elsewhere. -bob |
From: Ralf S. <ra...@br...> - 2005-06-02 09:09:54
|
Bob Ippolito wrote: >> >> >> I can't follow your arguments: >> mcmillan installer has automatic dependency tracking. It uses >> sensible defaults (i.e. excluding /lib, /usr/lib and a bunch of dlls >> on windows) and it did work for us without any problems in 99% of all >> cases. I'd rather manually care for that last 1 percent than doing it >> every time. > > > If you're using something like Debian, everything you install with apt > will be in /usr/lib... so how do you know what to ship? > Look no further than one paragraph above. Skip everything from /lib, and /usr/lib and include all libraries found elsewhere. |
From: Bob I. <bo...@re...> - 2005-06-02 01:11:41
|
On Jun 1, 2005, at 5:07 PM, Ralf Schmitt wrote: > Bob Ippolito wrote: > > >> Well, you said "here's a patch to do multiple scripts", but it >> includes unfinished code for binary dependency tracking... >> Generally it's best to send patches that do only what you say >> they do, especially if the other stuff isn't working code :) > > Sorry for sending unfinished, buggy code. It's actually work in > progress, and I didn't take the time to clean it up for public > consumption, because I even don't know if anybody is interested in > the funtionality it provides, and if cx_Freeze's author would be > willing to accept patches. > > >> Unlike Mac OS X, which installs the same "unix system" regardless >> of installation options, to locations distinguishable from user- >> installed stuff, it's impossible to tell automatically what is >> included with a "default system" by ldd.. so the user is really >> going to have to manually choose what they think should be >> included at some point. That's probably why nobody has bothered >> to try and automate it, because it needs arbitrary decisions to >> be made. >> > > I can't follow your arguments: > mcmillan installer has automatic dependency tracking. It uses > sensible defaults (i.e. excluding /lib, /usr/lib and a bunch of > dlls on windows) and it did work for us without any problems in 99% > of all cases. I'd rather manually care for that last 1 percent than > doing it every time. If you're using something like Debian, everything you install with apt will be in /usr/lib... so how do you know what to ship? -bob |
From: Ralf S. <ra...@br...> - 2005-06-02 00:07:29
|
Bob Ippolito wrote: > > Well, you said "here's a patch to do multiple scripts", but it includes > unfinished code for binary dependency tracking... Generally it's best > to send patches that do only what you say they do, especially if the > other stuff isn't working code :) > Sorry for sending unfinished, buggy code. It's actually work in progress, and I didn't take the time to clean it up for public consumption, because I even don't know if anybody is interested in the funtionality it provides, and if cx_Freeze's author would be willing to accept patches. > Unlike Mac OS X, which installs the same "unix system" regardless of > installation options, to locations distinguishable from user- installed > stuff, it's impossible to tell automatically what is included with a > "default system" by ldd.. so the user is really going to have to > manually choose what they think should be included at some point. > That's probably why nobody has bothered to try and automate it, because > it needs arbitrary decisions to be made. > I can't follow your arguments: mcmillan installer has automatic dependency tracking. It uses sensible defaults (i.e. excluding /lib, /usr/lib and a bunch of dlls on windows) and it did work for us without any problems in 99% of all cases. I'd rather manually care for that last 1 percent than doing it every time. - Ralf |
From: Bob I. <bo...@re...> - 2005-06-01 16:49:06
|
On Jun 1, 2005, at 3:15 AM, Ralf Schmitt wrote: > Bob Ippolito wrote: > >> On Jun 1, 2005, at 1:13 AM, Ralf Schmitt wrote: >> >>> I have modified cx_Freeze to generate one executable from >>> multiple python scripts. It collects all dependencies of all >>> scripts into one executable with an attached zip file, saves >>> each scripts bytecode with a special name and dispatches on the >>> name the executable is called with (i.e. one creates symlinks or >>> hardlinks to that executable for each python script). >>> Is it possible to include such functionality into cx_Freeze? >>> What I am also missing is a cx_Freeze API, i.e. FreezePython's >>> functionality as a Module (my patch could have been shorter, if >>> cx_Freeze did provide such an API). >>> >> getDependenciesMach doesn't seem like it performs any useful >> function, since it doesn't do load command rewriting, it's not >> recursive, and it doesn't understand frameworks. Using >> getDependenciesMach as-is, the best you can do is copy the direct >> dependencies next to the executable -- but they're not going to >> get loaded, because "@executable_path/" is not part of the >> default search path, etc. It's kind of a red herring to put that >> kind of code in a patch because it's not correct and doesn't do >> anything useful yet :) >> > > This patch is not finished yet. > getDependenciesLDD currently also copies /lib/ld-linux.so and all > dependencies from /usr/lib (they should most probably be skipped). > On Windows no dependencies are computed. Nevertheless I think it > would be useful to include some code for binary dependency tracking > in cx_Freeze, at least when it's as easy as calling ldd. I think > most people somehow have to solve that problem... Well, you said "here's a patch to do multiple scripts", but it includes unfinished code for binary dependency tracking... Generally it's best to send patches that do only what you say they do, especially if the other stuff isn't working code :) Unlike Mac OS X, which installs the same "unix system" regardless of installation options, to locations distinguishable from user- installed stuff, it's impossible to tell automatically what is included with a "default system" by ldd.. so the user is really going to have to manually choose what they think should be included at some point. That's probably why nobody has bothered to try and automate it, because it needs arbitrary decisions to be made. -bob |
From: Roger B. <ro...@ro...> - 2005-06-01 16:03:30
|
> I mostly agree with that. Python itself might be a special case, > however, just because of the size it adds to the ZIP file. But I see > your point. Technically Python isn't in the zip file, only the byte code is. The files as a whole are wrapped up in an RPM for distribution. RPMs end up compressed as a whole anyway so the size of individual files inside isn't that relevant. > That's good to know. Perhaps what I suggested above is only of interest > to casual hackers who want to make a quick change without installing all > of required modules. But I see your point about .pyo vs .pyc, which is > good to keep in mind. I wouldn't accept patches and in most cases feedback from someone screwing around with archives in the way you are. You never quite know if things were done right. If you are running BitPim from source then many of the modules are optional. > That way you > could have a RPM for advanced users that would not include the large > statically linked executable (which would depend on a minimum version of > python), and one that does for everyone else. Nope! Advanced users can get the source and slice and dice things in any way they want. I give the fewest number of options possible and make sure that they "just work". I would always be happy with a larger redistributable file if the package is more likely to work well. > The prelink utility renders executables produced by cx_Freeze (which is > used by bitpim) unusable by stripping off the ZIP portion of the > executable. To prevent this from happening edit the configuration file > (typically /etc/prelink.conf) for prelink and exclude any files produced > by cx_Freeze: I intend to fix the problem by using the cx-Freeze option to have the zip of the bytecode be a seperate file rather than appended to the main executable. > And I agree with you about the proper behavior for setuid executables (dropping > privileges, etc.). But that behavior can be implemented in python, so > I guess the issue is just whether or not there are other issues that would > get in the way. I wouldn't implement the dropping privileges launcher in Python because way too much is executed which means more potential holes. You already mentioned environment variables. There are also lots of shared libraries used, character set/locale information and who knows what else. The point of the launchers is to be as trivial as possible - ideally you want the code to be a screenful or two. Python definitely doesn't fit those requirements. Roger |
From: Ralf S. <ra...@br...> - 2005-06-01 10:15:21
|
Bob Ippolito wrote: > > On Jun 1, 2005, at 1:13 AM, Ralf Schmitt wrote: > >> I have modified cx_Freeze to generate one executable from multiple >> python scripts. It collects all dependencies of all scripts into one >> executable with an attached zip file, saves each scripts bytecode >> with a special name and dispatches on the name the executable is >> called with (i.e. one creates symlinks or hardlinks to that >> executable for each python script). >> Is it possible to include such functionality into cx_Freeze? >> What I am also missing is a cx_Freeze API, i.e. FreezePython's >> functionality as a Module (my patch could have been shorter, if >> cx_Freeze did provide such an API). > > > getDependenciesMach doesn't seem like it performs any useful function, > since it doesn't do load command rewriting, it's not recursive, and it > doesn't understand frameworks. Using getDependenciesMach as-is, the > best you can do is copy the direct dependencies next to the executable > -- but they're not going to get loaded, because "@executable_path/" is > not part of the default search path, etc. It's kind of a red herring > to put that kind of code in a patch because it's not correct and > doesn't do anything useful yet :) > This patch is not finished yet. getDependenciesLDD currently also copies /lib/ld-linux.so and all dependencies from /usr/lib (they should most probably be skipped). On Windows no dependencies are computed. Nevertheless I think it would be useful to include some code for binary dependency tracking in cx_Freeze, at least when it's as easy as calling ldd. I think most people somehow have to solve that problem... > py2app does all of this stuff correctly, via its macholib (which is a > "separate" package, but is only distributed with py2app). It could > also be used by cx_freeze, I guess. > nice to know, maybe I'll try (mac os x currently isn't a target platform for our software, but it just happens that I own one of those macs and really like them). > -bob > |
From: Bob I. <bo...@re...> - 2005-06-01 09:17:39
|
On Jun 1, 2005, at 1:13 AM, Ralf Schmitt wrote: > I have modified cx_Freeze to generate one executable from multiple > python scripts. It collects all dependencies of all scripts into > one executable with an attached zip file, saves each scripts > bytecode with a special name and dispatches on the name the > executable is called with (i.e. one creates symlinks or hardlinks > to that executable for each python script). > Is it possible to include such functionality into cx_Freeze? > What I am also missing is a cx_Freeze API, i.e. FreezePython's > functionality as a Module (my patch could have been shorter, if > cx_Freeze did provide such an API). getDependenciesMach doesn't seem like it performs any useful function, since it doesn't do load command rewriting, it's not recursive, and it doesn't understand frameworks. Using getDependenciesMach as-is, the best you can do is copy the direct dependencies next to the executable -- but they're not going to get loaded, because "@executable_path/" is not part of the default search path, etc. It's kind of a red herring to put that kind of code in a patch because it's not correct and doesn't do anything useful yet :) py2app does all of this stuff correctly, via its macholib (which is a "separate" package, but is only distributed with py2app). It could also be used by cx_freeze, I guess. -bob |
From: Ralf S. <ra...@br...> - 2005-06-01 08:13:53
|
Hi all, I have modified cx_Freeze to generate one executable from multiple python scripts. It collects all dependencies of all scripts into one executable with an attached zip file, saves each scripts bytecode with a special name and dispatches on the name the executable is called with (i.e. one creates symlinks or hardlinks to that executable for each python script). Is it possible to include such functionality into cx_Freeze? What I am also missing is a cx_Freeze API, i.e. FreezePython's functionality as a Module (my patch could have been shorter, if cx_Freeze did provide such an API). - Ralf |
From: Steven E. <sel...@au...> - 2005-05-29 15:35:22
|
On Sun, 2005-05-22 at 23:27 -0700, Roger Binns wrote: > > For > > example, I assume that part of the reason bitpim uses cx_Freeze is that > > it depends on nearly ten different Python utilities, and they didn't > > want to include all of them. > > The actual reason is that because users don't and shouldn't have to care > what language a program is written in, what other components it > uses, and similar implementation details. As a user you install > one package and it *just works*. BTW if cx-Freeze did have optional > static vs dynamic libpython then I would pick the static one anyway. > (I have done a lot of commercial Unix software development and long > ago learned the lesson to make sure there are the least number of > things possible to go wrong.) I mostly agree with that. Python itself might be a special case, however, just because of the size it adds to the ZIP file. But I see your point. > > I found that I can make modifications to > > the bitpim zip (/usr/lib/bitpim-0.7.32/bp) by only installing the bitpim > > source, compiling it with compileall.py: > > python /usr/lib/python2.3/compileall.py /usr/local/src/bitpim- > > cvs/bitpim > > and then updating the pyc files I care about with: > > That is definitely not the right way to go about it. Some of the code > is not intended to be distributed and the code that is in the "zip" > is all .pyo (ie with the optimiser on which leaves out some code with > if __debug__). As a small example, the Audiovox CDM8900 is not available > in official builds but is available otherwise. That's good to know. Perhaps what I suggested above is only of interest to casual hackers who want to make a quick change without installing all of required modules. But I see your point about .pyo vs .pyc, which is good to keep in mind. > You can just run BitPim from source and everything will be fine. If > you want a redistributable package then run makedist.py which will > ensure you have all the components and run cx-Freeze and produce > the rpm for you. It doesn't taken very long. (If you are going > to make your own BitPim builds then please change the vendor in > version.py to be you.) > > At some point I will be using the option to put the bytecode into > a seperate file rather than appended to the ELF executable. This > will then more closely mirror how the Windows and Mac builds are > done. Sounds good. Maybe you could come up with some way of making the executable optional so that the python code could be executed directly without running the executable. Something like: export PYTHONPATH=$PYTHONPATH:/usr/lib/bitpim-0.7.32/bp python -c 'import cx_Freeze__init__' which does not currently work, but it might be close. That way you could have a RPM for advanced users that would not include the large statically linked executable (which would depend on a minimum version of python), and one that does for everyone else. If the executable was not present the /usr/bin/bitpim script (or whatever application is relevant) would start bitpim with something like what I have above. > It also stops the problem (another user recently got bitten) > of the operating system stripping ELF binaries in cron jobs (like > Fedora helpfully does) which strips the zip off the end. It sure does. To be exact, the problem is caused by the "prelink" utility launched by /etc/cron.daily/prelink on at least Fedora Core 3. It might by nice to add something like the following to the documentation for bitpim and or cx_Freeze: ---- The prelink utility renders executables produced by cx_Freeze (which is used by bitpim) unusable by stripping off the ZIP portion of the executable. To prevent this from happening edit the configuration file (typically /etc/prelink.conf) for prelink and exclude any files produced by cx_Freeze: > diff -ruw /etc/prelink.conf.orig /etc/prelink.conf --- /etc/prelink.conf.orig 2005-05-29 09:53:00.561694272 -0500 +++ /etc/prelink.conf 2005-05-29 09:53:31.340015256 -0500 @@ -18,6 +18,7 @@ -b /lib/modules -b /usr/lib/locale -b /usr/X11R6/lib{,64}/X11/xfig +-b /usr/lib/bitpim*/bp -l /bin -l /usr/bin -l /sbin ---- > > > Also as a matter of documentation you could indicate whether using > > cx_Freeze to create suid scripts is a good idea, and how to do it. > > Hopefully you don't want to do that with BitPim! > > In general the best way of making suid programs is having some launcher > process that only grabs the reserved items (eg privileged ports, file > handles), immediately drops permissions and then execs the more > conventional program run as a limited user which uses the inherited handles > for the privileged items. I wasn't planning on doing that with bitpim, just curious in general. I've found that there is at least the issue that the user may set PYTHONPATH to point to modified system pyc files (such as os.pyc), but at least the sys module is builtin, so you can startup, and then set sys.path to something safe. I don't know if there are any other major issues that render using cx_Freeze to produce suid executables insecure, but it might be nice to document them in any case. And I agree with you about the proper behavior for setuid executables (dropping privileges, etc.). But that behavior can be implemented in python, so I guess the issue is just whether or not there are other issues that would get in the way. -- ----------------------------------------------------------------------- | Steven Elliott | sel...@au... | ----------------------------------------------------------------------- |
From: Roger B. <ro...@ro...> - 2005-05-23 06:27:22
|
> For > example, I assume that part of the reason bitpim uses cx_Freeze is that > it depends on nearly ten different Python utilities, and they didn't > want to include all of them. The actual reason is that because users don't and shouldn't have to care what language a program is written in, what other components it uses, and similar implementation details. As a user you install one package and it *just works*. BTW if cx-Freeze did have optional static vs dynamic libpython then I would pick the static one anyway. (I have done a lot of commercial Unix software development and long ago learned the lesson to make sure there are the least number of things possible to go wrong.) > I found that I can make modifications to > the bitpim zip (/usr/lib/bitpim-0.7.32/bp) by only installing the bitpim > source, compiling it with compileall.py: > python /usr/lib/python2.3/compileall.py /usr/local/src/bitpim- > cvs/bitpim > and then updating the pyc files I care about with: That is definitely not the right way to go about it. Some of the code is not intended to be distributed and the code that is in the "zip" is all .pyo (ie with the optimiser on which leaves out some code with if __debug__). As a small example, the Audiovox CDM8900 is not available in official builds but is available otherwise. You can just run BitPim from source and everything will be fine. If you want a redistributable package then run makedist.py which will ensure you have all the components and run cx-Freeze and produce the rpm for you. It doesn't taken very long. (If you are going to make your own BitPim builds then please change the vendor in version.py to be you.) At some point I will be using the option to put the bytecode into a seperate file rather than appended to the ELF executable. This will then more closely mirror how the Windows and Mac builds are done. It also stops the problem (another user recently got bitten) of the operating system stripping ELF binaries in cron jobs (like Fedora helpfully does) which strips the zip off the end. > Also as a matter of documentation you could indicate whether using > cx_Freeze to create suid scripts is a good idea, and how to do it. Hopefully you don't want to do that with BitPim! In general the best way of making suid programs is having some launcher process that only grabs the reserved items (eg privileged ports, file handles), immediately drops permissions and then execs the more conventional program run as a limited user which uses the inherited handles for the privileged items. Roger |
From: Steven E. <sel...@au...> - 2005-05-23 06:07:11
|
I've been playing with cx_Freeze recently, and I think it is a great tool. I have a few ideas that I want to share. Hopefully they are not too obvious, redundant or impractical. Have you thought of adding an option to indicate whether the resulting executable is linked against the static or shared libpython? Linking against the shared libpython ought to just be a matter of modifying MakeFrozenBases.py so that it does not link against the static libpython in the "LIBPL" directory assuming that there is a shared libpython elsewhere on the system (/usr/lib/libpython2.3.so on my Fedora Core 3 system). Using the shared libpython obviously results in a much smaller executable, and the shared libpython could optionally be copied along with the other dependencies. And if more than one frozen executable is shipped nearly a 1 Mb would be saved per additional executable. When cx_Freeze is built in could generate bases "ConsoleShared", "ConsoleKeepPathShared", "ConsoleStatic" and "ConsoleKeepPathStatic" based on what is available and then generate frozen executables using either the shared or static base depending on the option specified. As a matter of documentation you might want to mention that the "zip" utility can be used to manipulate the resulting frozen executable. For example, I assume that part of the reason bitpim uses cx_Freeze is that it depends on nearly ten different Python utilities, and they didn't want to include all of them. I found that I can make modifications to the bitpim zip (/usr/lib/bitpim-0.7.32/bp) by only installing the bitpim source, compiling it with compileall.py: python /usr/lib/python2.3/compileall.py /usr/local/src/bitpim- cvs/bitpim and then updating the pyc files I care about with: cd /usr/lib/bitpim-0.7.32 ln -s bp bp.zip cd /usr/local/src/bitpim-cvs/bitpim zip -u ~-/bp.zip gui.pyc Also as a matter of documentation you could indicate whether using cx_Freeze to create suid scripts is a good idea, and how to do it. In other words, I've verified that setting the suid bits on the frozen executable works: chmod a+s frozen_exe (which needless to say does not work on the original python script), but I'm not sure how secure it is, or if there may be other problems with it. In other words, people might like having cx_Freeze as a tool to quickly and easily create suid tools if it is safe to do so. And if doing so is a good idea it provides an additional reason to link against the shared libpython since I imagine that such tools would typically be run on the system on which they were created. -- Steven Elliott |