From: Chuck E. <ec...@mi...> - 2001-02-23 19:33:35
|
In summary, the problem is that when launching a program from within a Python package, confusion can result about paths of modules resulting in duplicate modules residing in memory (and therefore duplication classes (and therefore problems with isinstance())). The solution was to provide a Launch.py program that essentially does an "os.chdir(os.pardir))" before importing the package, its "starter" module and invoking main(). Also, a "if '' not in sys.path: sys.path.insert(0, '')" was needed to fix problems when starting with "Launch.py" rather than "python Launch.py". Overall this solution is pretty small and keeps things straight. Thanks to Geoff Talvola for the fix and everyone else for their input. -Chuck "Chuck Esterbrook" <ec...@mi...> wrote in message news:<mai...@py...>... >You can find a tarball with this description and accompanying source code at: > ftp://webware.sourceforge.net/pub/webware/ModulesProb01.tar.gz >I have a problem where Python creates duplicate modules in memory rather >than reuse the same one. For example, a module in Pkg/Mod.py ends up in >sys.modules under keys "Pkg.Mod" and "Mod" **pointing to two distinct >modules**. >This creates further problems: Suppose a module Foo.py contains a class >named Foo. If the module is loaded twice as two separate instances (of >ModuleType), then there are 2 separate Foo classes. This causes confusion >including the failure of an assertion such as: > assert issubclass(foo, Foo) >The Foo in that code may be pointing to the first class, while the >instance foo may have been created from the second class. The assertion >then fails! >The problem stems from the fact that Python tracks modules by a relative, >rather than absolute, path. A simple os.chdir() or a subtley in packages >can cause this problem. >This problem is easiest to see in an os.chdir() situation: >C:\>mkdir foo >C:\>cd foo >C:\foo>mkdir bar >C:\foo>cd bar >C:\foo\bar>echo class baz: pass > baz.py >C:\foo\bar>echo ### > __init__.py >C:\foo\bar>cd .. >C:\foo>python >ActivePython 2.0, build 202 (ActiveState Tool Corp.) >based on Python 2.0 (#8, Oct 19 2000, 11:30:05) [MSC 32 bit (Intel)] on win32 > >>> from bar.baz import baz as baz1 > >>> import os > >>> os.chdir('bar') > >>> from baz import baz as baz2 > >>> baz1 ><class bar.baz.baz at 007908BC> > >>> baz2 ><class baz.baz at 0079075C> > >>> baz1 == baz2 >0 > >>> import sys > >>> [mod for mod in sys.modules.items() if mod[0].count('baz')] >[('baz', <module 'baz' from 'baz.pyc'>), > ('bar.baz', <module 'bar.baz' from 'bar\baz.pyc'>)] > >>> mod1 = sys.modules['bar.baz'] > >>> mod2 = sys.modules['baz'] > >>> id(mod1) >136147728 > >>> id(mod2) >136147744 > >This problem can also be seen without any use of os.chdir(). See >ManufactureWare/ which contains the "assert issubclass(foo, Foo)" problem >described above. >This all applies to Python 2.0 on Windows & UNIX. I haven't tried 2.1 yet. >I believe the solution is for Python to track modules by their absolute >path. I don't know of any other resolution to the situation other than >modifying Python in this manner. I also don't know of any disadvantage for >Python to track modules by absolute path. >- Does anyone know of any workarounds? >- Does anyone know why it would be bad for Python to track modules by >absolute path? >- Is there any chance Python will fix this in the future? >- If so, by 2.1? > >-Chuck >ftp://webware.sourceforge.net/pub/webware/ModulesProb01.tar.gz |
From: Terrel S. <tsh...@tr...> - 2001-02-23 20:29:42
|
Chuck Esterbrook wrote: > In summary, the problem is that when launching a program from within a > Python package, confusion can result about paths of modules resulting in > duplicate modules residing in memory (and therefore duplication classes > (and therefore problems with isinstance())). > > The solution was to provide a Launch.py program that essentially does an > "os.chdir(os.pardir))" before importing the package, its > "starter" module and invoking main(). I don't understand why this is necessary. It seems much cleaner to me to just be careful about sys.path and use the correct import statements. > Also, a "if '' not in sys.path: sys.path.insert(0, '')" was needed to fix > problems when starting with "Launch.py" rather than "python Launch.py". > Please explain this problem. Can you show us Lauch.py and examples of its use? I think having "" in sys.path is just asking for trouble, but oh well -- leave it in if you like. The ManufactureWare/FactoryKit/main.py works fine if you correct the broken import statement. __main__ is not part of the package and cannot use the relative import shortcut: "from FactoryKit import Factory" is the correct import. Removing "" from sys.path just makes the breakage more visible. |
From: Geoff T. <gta...@na...> - 2001-02-23 20:54:40
|
Terrel Shumway wrote: > Chuck Esterbrook wrote: > > > In summary, the problem is that when launching a program from within a > > Python package, confusion can result about paths of modules resulting in > > duplicate modules residing in memory (and therefore duplication classes > > (and therefore problems with isinstance())). > > > > The solution was to provide a Launch.py program that essentially does an > > "os.chdir(os.pardir))" before importing the package, its > > "starter" module and invoking main(). > > I don't understand why this is necessary. It seems much cleaner to me to just be > careful about sys.path and use the correct import statements. > > > Also, a "if '' not in sys.path: sys.path.insert(0, '')" was needed to fix > > problems when starting with "Launch.py" rather than "python Launch.py". > > > > Please explain this problem. Can you show us Lauch.py and examples of its use? > > I think having "" in sys.path is just asking for trouble, but oh well -- leave it > in if you like. The ManufactureWare/FactoryKit/main.py works fine if you correct > the broken import statement. __main__ is not part of the package and cannot use > the relative import shortcut: "from FactoryKit import Factory" is the correct > import. Removing "" from sys.path just makes the breakage more visible. But if I try to change "import Factory" to "from FactoryKit import Factory" I get this error: ImportError: No module named FactoryKit Presumably this is because I'm running a program that is *inside* of the FactoryKit directory itself, so it can't see the FactoryKit package. By changing down one directory, you can "see" the FactoryKit package and import it, and all is well. (Are you saying that you did _not_ get that error in main.py? Maybe I misunderstood the change you made to main.py that made it work properly.) As for adding "" to sys.path, on Windows, I've found that if you run a program with "python script.py" then "" is provided as the first element in sys.path. This means you can change directories and then import stuff in your new current directory. But if you run a program with "script.py", then the absolute path of the script is the first element in sys.path, and "" is nowhere in sys.path. This means when you change the current directory, you can't import anything in your new current directory. So the os.chdir(os.pardir) trick doesn't work. (Also, when you run a program as a service on NT, "" is not part of the path.) So in order for the os.chdir(os.pardir) to work, you need to add "" to sys.path if it's not already there. -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Chuck E. <ec...@mi...> - 2001-02-23 21:00:24
|
At 03:34 PM 2/23/2001 -0500, Terrel Shumway wrote: >Chuck Esterbrook wrote: > > > In summary, the problem is that when launching a program from within a > > Python package, confusion can result about paths of modules resulting in > > duplicate modules residing in memory (and therefore duplication classes > > (and therefore problems with isinstance())). > > > > The solution was to provide a Launch.py program that essentially does an > > "os.chdir(os.pardir))" before importing the package, its > > "starter" module and invoking main(). > >I don't understand why this is necessary. It seems much cleaner to me to >just be careful about sys.path and use the correct import statements. Um, I never had any incorrect or abusive use of import statements. When a file sat in the same directory, I said: from Foo import Foo When it sat in an external package, I said: from Pkg.Foo import Foo I learned these conventions from Python. I didn't invent them. Perhaps I had an "incorrect" situation if it was never intended that programs start out of their own packages. > > Also, a "if '' not in sys.path: sys.path.insert(0, '')" was needed to fix > > problems when starting with "Launch.py" rather than "python Launch.py". > > > >Please explain this problem. Can you show us Lauch.py and examples of its use? You can get it from CVS: https://sourceforge.net/cvs/?group_id=4866 See WebKit/Launch.py. Again, the problem is when running a program out of a package. And if you don't mind duplicate modules, you don't need the fix. But duplicates eventually lead to subtle, arcane bugs. We need the fix in WebKit. See more below. >I think having "" in sys.path is just asking for trouble, but oh well -- >leave it in if you like. The ManufactureWare/FactoryKit/main.py works >fine if you correct the broken import statement. __main__ is not part of >the package and cannot use the relative import shortcut: "from FactoryKit >import Factory" is the correct import. Removing "" from sys.path just >makes the breakage more visible. Regarding the import, the fix does what you describe: Makes the package explicit to the program. Regarding '' in sys.path, if you say: > python someprog.py you get '' as the first element in the path. Again that's Python, not me. However, if you say: > someprog.py then you get an absolute path in place of ''. Geoff ran into this first and it caused problems, hence the fix. Since the fix puts you back to the sys.path you would have gotten from Python anyway (by launching differently), it doesn't feel so evil to me. Regarding my choice of: os.chdir(os.pardir) I could done something like this instead: sys.path.insert(1, os.path.abspath(os.pardir)) The first seemed a tad simpler. Also, regarding the use of Launch.py, I might have chosen to bake this directly into the app server, BUT the problem is that we have at least 2 of them and will be adding another in the future. Also, Launch.py is a very small, very clean file, making it easier to deal with independent of all that goes on inside the app server source files. I have been flexing the plug-ins, examples and test cases for WebKit and things are working good. At the very least, they're working much better than before, even if Launch.py doesn't suit your taste. The standard way to launch the app server is still: > cd Webware/WebKit > ./AppServer -Chuck -- Get web dev at http://webware.sourceforge.net/ |
From: Chuck E. <ec...@mi...> - 2001-02-24 00:27:40
|
At 07:27 PM 2/23/2001 -0500, Jay Love wrote: >I would prefer the latter approach. Seems cleaner than os.chdir. >Any objections? Yeah, I think the former is cleaner. It matches the original idea of running the program above the framework rather than inside it. Plus after all this hassle, things finally work. :-) -Chuck |
From: Jay L. <js...@js...> - 2001-02-24 00:16:03
|
Chuck Esterbrook wrote: > > Regarding my choice of: > os.chdir(os.pardir) > > I could done something like this instead: > sys.path.insert(1, os.path.abspath(os.pardir)) > > The first seemed a tad simpler. I would prefer the latter approach. Seems cleaner than os.chdir. Any objections? Jay > |
From: Jay L. <js...@js...> - 2001-02-24 01:07:18
|
What if someone wants to run Webwrae from another directory? I haven't even tried ModPython/ModSnake yet, but ModSnake hates os.chdir(). What's the benefit of os.chdir()? Jay Chuck Esterbrook wrote: > At 07:27 PM 2/23/2001 -0500, Jay Love wrote: > >> I would prefer the latter approach. Seems cleaner than os.chdir. >> Any objections? > > > Yeah, I think the former is cleaner. It matches the original idea of > running the program above the framework rather than inside it. > > Plus after all this hassle, things finally work. :-) > > -Chuck > > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/lists/listinfo/webware-discuss |
From: Chuck E. <ec...@mi...> - 2001-02-24 01:17:12
|
At 08:18 PM 2/23/2001 -0500, Jay Love wrote: >What if someone wants to run Webwrae from another directory? >I haven't even tried ModPython/ModSnake yet, but ModSnake hates os.chdir(). If you're launching from the command line, there is no desire or need to do anything other than cd into Webware/WebKit and run AppServer. If you're running the app server from inside something else, then you need to do 2 things: [1] from WebKit.ThreadedAppServer import run run(useMonitor=whatever) [2] Make sure that WebKit is in your path. Whether you put that burden on the user (via PYTHONPATH or installing the components with distutils) or augment sys.path is up to you. In this scenario of running WebKit from something else, there is no Launch.py (another advantage of keeping Launch.py out of the app servers), so there is no issue with os.chdir(). -Chuck |
From: Chuck E. <ec...@mi...> - 2001-02-24 02:53:43
|
At 08:18 PM 2/23/2001 -0500, Jay Love wrote: >What's the benefit of os.chdir()? One other point I forgot, and a definite benefit of os.chdir(): It has flushed out several situations where we still relied on the current directory. That's a dependency that we earlier decided to get rid of in order to make the app server more flexible and more likely to work in interesting situations (like embedding in Apache). I have fixed at least 3-4 spots that I missed in my earlier efforts. -Chuck |