Thread: [SQLObject] classregistry and multiple imports
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: Philippe N. <ph...@re...> - 2005-01-08 14:38:46
|
Hi I've got a little issue with sqlo svn trunk. There's a module AlineaArticle storing one SQLObject declaration and i have to import it from multiple places. When i do so, sqlo raises a ValueError: ValueError: class AlineaArticle is already in the registry (other class is <class 'Alinea.Articles.AlineaArticle.AlineaArticle'>, from the module Alinea.Articles.AlineaArticle in /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.py; attempted new class is <class 'Articles.AlineaArticle.AlineaArticle'>, from the module Articles.AlineaArticle in /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.pyc) Seems like the .py and the .pyc try to register the SQLObject in their corner. That's weird, has anyboday ever had the same problem ? Philippe |
From: Carlos R. <car...@gm...> - 2005-01-08 15:37:21
|
On Sat, 8 Jan 2005 15:38:42 +0100 (CET), Philippe Normand <ph...@re...> wrote: > > Hi > > I've got a little issue with sqlo svn trunk. There's a module > AlineaArticle storing one SQLObject declaration and i have to import it > from multiple places. When i do so, sqlo raises a ValueError: > > ValueError: class AlineaArticle is already in the registry (other class > is <class 'Alinea.Articles.AlineaArticle.AlineaArticle'>, from the > module Alinea.Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.py; attempted new > class is <class 'Articles.AlineaArticle.AlineaArticle'>, from the > module Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.pyc) > > Seems like the .py and the .pyc try to register the SQLObject in their > corner. That's weird, has anyboday ever had the same problem ? I had the same issue. I don't think that this is a problem with SQLObject itself; it depends a lot on the way you write your code. Loading the same module twice with a simple "import" should not be a problem, but anything that runs the code more than once will make it fail. I believe that this can be triggered in some situations: - by a reload() call, which makes the module code to be re-executed. - by importing the same module from different locations, in such a way that the internal module cache used by Python isn't able to tell that the module was actually imported before. In this case the module code is effectively run twice, which may also trigger the problem. I solved the issue in my own code by putting all my SQLObject declarations inside a class that is guaranteed to be instantiated only once, as a singleton. But this solution will not work for everyone; it was actually more of a design choice in my own case, dicted not only by SQLObject, but also because of other architectural concerns. You could try something along these lines, too, if it suites your needs. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Philippe N. <ph...@re...> - 2005-01-08 21:33:39
|
Le 8/1/2005, "Carlos Ribeiro" <car...@gm...> a =E9crit: > >I had the same issue. I don't think that this is a problem with >SQLObject itself; it depends a lot on the way you write your code. >Loading the same module twice with a simple "import" should not be a >problem, but anything that runs the code more than once will make it >fail. I believe that this can be triggered in some situations: > >- by a reload() call, which makes the module code to be re-executed. > >- by importing the same module from different locations, in such a way >that the internal module cache used by Python isn't able to tell that >the module was actually imported before. In this case the module code >is effectively run twice, which may also trigger the problem. > But it's exactly what i am doing. The problem is that the classregistry's addClass hook is too restrictive. If the class is already in the registry it should not raise an error (in my opinion). Instead it should simply exit from the hook. Can other sqlo dev explain why addClass raises a ValueError such a way ? Philippe |
From: Martin B. <mar...@gm...> - 2005-01-08 21:54:46
|
after 4 years of python hacking, i'm still shocked at how fast it is to work with it. in a few minutes i had my existing connection going, with a few lines of code (there is a little hack for resetting the registry): from sqlobject.postgres.pgconnection import PostgresConnection #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D # LOCAL DECLARATIONS #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D # Reset the class registry to allow reloading. classregistry.MasterRegistry =3D classregistry._MasterRegistry() classregistry.registry =3D classregistry.MasterRegistry.registry #--------------------------------------------------------------------------= ----- # class ExistingPgConnection(PostgresConnection): """ Provide a connection object for SQLObject that reuses the existing connection. """ def __init__(self, connection): """ Initialize with an existing, opened connection from psycopg. """ PostgresConnection.__init__(self, autoCommit=3DFalse) self.existing_connection =3D connection def makeConnection(self): # Note: support for autocommit member has been removed in psycopg2. return self.existing_connection then in my code i do: from sqlobject import * __connection__ =3D ExistingPgConnection(database.m_connection) class Person(SQLObject): ... voila! 1. now, to be fair, it only "seems" to work, but i would really need to dig into that class registry issue to find out if the hack is enough of a reset. i need to reset the registry because my framework has an auto reload feature which make it really fast to develop and i'm not giving that up... 2. also, i'm not sure that returning the same single connection from makeConnection() is kosher with the connection pool code. i'm not very familiar with sql databases... why do we need connection pools? any issue that might help make this clean would be appreciated (anything off thetop of your head)... thanks cheers, On Sat, 8 Jan 2005 22:33:42 +0100 (CET), Philippe Normand <ph...@re...> wrote: >=20 > Le 8/1/2005, "Carlos Ribeiro" <car...@gm...> a =E9crit: >=20 > > > >I had the same issue. I don't think that this is a problem with > >SQLObject itself; it depends a lot on the way you write your code. > >Loading the same module twice with a simple "import" should not be a > >problem, but anything that runs the code more than once will make it > >fail. I believe that this can be triggered in some situations: > > > >- by a reload() call, which makes the module code to be re-executed. > > > >- by importing the same module from different locations, in such a way > >that the internal module cache used by Python isn't able to tell that > >the module was actually imported before. In this case the module code > >is effectively run twice, which may also trigger the problem. > > >=20 > But it's exactly what i am doing. The problem is that the > classregistry's addClass hook is too restrictive. If the class is > already in the registry it should not raise an error (in my opinion). > Instead it should simply exit from the hook. >=20 > Can other sqlo dev explain why addClass raises a ValueError such a way ? >=20 > Philippe >=20 > ------------------------------------------------------- > The SF.Net email is sponsored by: Beat the post-holiday blues > Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. > It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > |
From: Carlos R. <car...@gm...> - 2005-01-08 22:57:19
|
On Sat, 8 Jan 2005 22:33:42 +0100 (CET), Philippe Normand <ph...@re...> wrote: >=20 > Le 8/1/2005, "Carlos Ribeiro" <car...@gm...> a =E9crit: >=20 > > > >I had the same issue. I don't think that this is a problem with > >SQLObject itself; it depends a lot on the way you write your code. > >Loading the same module twice with a simple "import" should not be a > >problem, but anything that runs the code more than once will make it > >fail. I believe that this can be triggered in some situations: > > > >- by a reload() call, which makes the module code to be re-executed. > > > >- by importing the same module from different locations, in such a way > >that the internal module cache used by Python isn't able to tell that > >the module was actually imported before. In this case the module code > >is effectively run twice, which may also trigger the problem. > > >=20 > But it's exactly what i am doing. The problem is that the > classregistry's addClass hook is too restrictive. If the class is > already in the registry it should not raise an error (in my opinion). > Instead it should simply exit from the hook. If it simply exits from the hook, you end up with a dangerous situation. There will be two classes with the same name. For all purposes, they are not equivalent. If you add something at runtime to one of them, the other one will not be updated accordingly. Any code that depends on common class attributes to work will break. There is a possible solution for this problem but it's not recommended anyway (see below). =20 > Can other sqlo dev explain why addClass raises a ValueError such a way ? To avoid having two *different* declarations of the same class to be inserted into the registry? Note that the registry is necessary first place to avoid problems with forward references to still "undeclared" tables - it may happen with any of the columns that create relationships (ForeignKey, for instance). So it allows classes to be referenced by name; it in turn allows circular references to be created. One possible solution that could be attempted is to use the metaclass to make this check. It could create the class, and check if it is *equivalent* to the previously declared one. If it is found to be equivalent (it contains the same columns with the same attributes for each one of them), the metaclass could return the *old* reference instead of the newly created one. It's not easy, I must say. Too much work for little gain, so it's better to keep it as it is. --=20 Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Philippe N. <ph...@re...> - 2005-01-09 10:52:40
|
>If it simply exits from the hook, you end up with a dangerous >situation. There will be two classes with the same name. For all >purposes, they are not equivalent. If you add something at runtime to >one of them, the other one will not be updated accordingly. Any code >that depends on common class attributes to work will break. There is a >possible solution for this problem but it's not recommended anyway >(see below). >=20 Oops my mistake ;) addClass is not a hook at all. It's called when you want to register a class. At the end it effectively calls some pre-defined hooks. So if we get back to "import from multiple places" issue, i propose following: if class in registry: return addIt to the registry call some callbacks instead of: if class in registry: raise ValueError() Isn't it more friendly ? Philippe |
From: Carlos R. <car...@gm...> - 2005-01-09 11:14:09
|
On Sun, 9 Jan 2005 11:52:26 +0100 (CET), Philippe Normand <ph...@re...> wrote: > > >If it simply exits from the hook, you end up with a dangerous > >situation. There will be two classes with the same name. For all > >purposes, they are not equivalent. If you add something at runtime to > >one of them, the other one will not be updated accordingly. Any code > >that depends on common class attributes to work will break. There is a > >possible solution for this problem but it's not recommended anyway > >(see below). > > > > Oops my mistake ;) addClass is not a hook at all. It's called when you > want to register a class. At the end it effectively calls some > pre-defined hooks. So if we get back to "import from multiple places" > issue, i propose following: > > if class in registry: > return > > addIt to the registry > call some callbacks > > instead of: > > if class in registry: > raise ValueError() > > Isn't it more friendly ? > Philippe The catch is the "if class in registry" statement. How does it know that the class in the registry? By the name alone? And what would happen if by mistake your program has two *different* declarations with the same name but in different modules? I still prefer the ValueError... whenever it got me, it was because I did run the module twice inadvertently (in my case, due to unit test code). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Philippe N. <ph...@re...> - 2005-01-09 14:19:28
|
>The catch is the "if class in registry" statement. How does it know >that the class in the registry? By the name alone? And what would >happen if by mistake your program has two *different* declarations >with the same name but in different modules? I still prefer the >ValueError... whenever it got me, it was because I did run the module >twice inadvertently (in my case, due to unit test code). > What about "if cls in self.allClasses()" ? Thus we don't rely on class name vs module name, but directly on class objects. I'm not sure it would work on all cases though but i believe it's a good compromise. Philippe |
From: Philippe N. <ph...@re...> - 2005-01-09 15:56:54
Attachments:
addClass.diff
|
Le 9/1/2005, "Philippe Normand" <ph...@re...> a =E9crit: > >>The catch is the "if class in registry" statement. How does it know >>that the class in the registry? By the name alone? And what would >>happen if by mistake your program has two *different* declarations >>with the same name but in different modules? I still prefer the >>ValueError... whenever it got me, it was because I did run the module >>twice inadvertently (in my case, due to unit test code). >> > >What about "if cls in self.allClasses()" ? Thus we don't rely on class >name vs module name, but directly on class objects. I'm not sure it >would work on all cases though but i believe it's a good compromise. > That's ugly. I finally found one good fix, which doesn't break ClassRegistryTest. I can commit, but prefer submit the patch here before. Philippe |
From: Carlos R. <car...@gm...> - 2005-01-09 16:11:36
|
On Sun, 9 Jan 2005 15:19:26 +0100 (CET), Philippe Normand <ph...@re...> wrote: > > >The catch is the "if class in registry" statement. How does it know > >that the class in the registry? By the name alone? And what would > >happen if by mistake your program has two *different* declarations > >with the same name but in different modules? I still prefer the > >ValueError... whenever it got me, it was because I did run the module > >twice inadvertently (in my case, due to unit test code). > > > > What about "if cls in self.allClasses()" ? Thus we don't rely on class > name vs module name, but directly on class objects. I'm not sure it > would work on all cases though but i believe it's a good compromise. Sorry. Either I misunderstand you idea, or you do misunderstand how do class statements work in Python. At some risk I'll assume the second, and try to clarify some things. A class is created when the statement is executed. For example, it is possible (but not really that useful) to do something like this: x = [] for i in range(10): class C: def someMethod(self): pass x.append(C) Run this code in interactive mode, and check the results: >>> print x[0].__name__ C >>> print x[1].__name__ C >>> x[0] is x[1] False >>> x[0] == x[1] False You'll see that both classes give back the same result, and have the same name stored in but are *different* - although you can argue that the classes are in fact equivalent, as far as their internal structure is concerned. It's very difficult to test for it though. In short, that's the reason why it isn't possible to test for inclusion as you proposed. The newly declared class is *different* from the original one, and will not be found in the registry - even if it is the result from the same declaration... run twice! The search by name, however, will be able to find another previously defined class with the same name, but there's no guarantee that both are equivalent. That's why an error message is the best that can be done. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Ksenia M. <kse...@gm...> - 2005-01-08 17:39:19
|
> ValueError: class AlineaArticle is already in the registry (other class > is <class 'Alinea.Articles.AlineaArticle.AlineaArticle'>, from the > module Alinea.Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.py; attempted new > class is <class 'Articles.AlineaArticle.AlineaArticle'>, from the > module Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.pyc) I had a similar problem, it happened when I was importing module from a python file in a symbolic link directory and did not provide absolute import path: from mymodule import MyClass The solution was to use absolute import path: from mypackage.mymodule import MyClass HTH -- Ksenia |
From: Ian B. <ia...@co...> - 2005-01-10 21:54:56
|
Philippe Normand wrote: > Hi > > I've got a little issue with sqlo svn trunk. There's a module > AlineaArticle storing one SQLObject declaration and i have to import it > from multiple places. When i do so, sqlo raises a ValueError: > > ValueError: class AlineaArticle is already in the registry (other class > is <class 'Alinea.Articles.AlineaArticle.AlineaArticle'>, from the > module Alinea.Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.py; attempted new > class is <class 'Articles.AlineaArticle.AlineaArticle'>, from the > module Articles.AlineaArticle in > /home/phil/devel/alinea/Alinea/Articles/AlineaArticle.pyc) > > Seems like the .py and the .pyc try to register the SQLObject in their > corner. That's weird, has anyboday ever had the same problem ? It sounds like you have an abiguous sys.path. This can lead to horrible, horrible bugs! Bugs where everything looks right, but the data is just wrong. SQLObject in this case is saving you from yourself. This is because you have both /home/phil/devel/alinea and /home/phil/devel/alinea/Alinea in your sys.path, and you sometimes import from one path and sometimes from the other. You get two distinct modules with identical code. It's very bad. -- Ian Bicking / ia...@co... / http://blog.ianbicking.org |
From: Philippe N. <ph...@re...> - 2005-01-11 00:21:37
|
Le 10/1/2005, "Ian Bicking" <ia...@co...> a =E9crit: > >It sounds like you have an abiguous sys.path. This can lead to >horrible, horrible bugs! Bugs where everything looks right, but the >data is just wrong. SQLObject in this case is saving you from yourself. > >This is because you have both /home/phil/devel/alinea and >/home/phil/devel/alinea/Alinea in your sys.path, and you sometimes >import from one path and sometimes from the other. You get two distinct >modules with identical code. It's very bad. > Indeed as usual you were right :) I did some summer cleanup in all my import's and sys.path, now it works like a charm. Thx Philippe |