From: Armin R. <ar...@ul...> - 2002-04-13 13:33:48
|
Hello Matt, Thanks for your interest in Psyco! Yes, they are naturally tons of things in various directions that could be done for Psyco. Matt Kimball wrote: > Are you planning to speed up object member access in any way? Yes, I have at least planned to give it serious thoughts. Note that using Python 2.2 new-style classes and __slots__, we force Python to use an efficient offset-based memory layout that Psyco can readily use efficiently. This said we must now think about letting Psyco automatically create memory layouts for instances. This could let it represent, say, a slot known to be an integer as just a 32-bit integer instead of a pointer to a PyIntObject. In other words, Psyco should be able to manage instances whose layout is just the list of the run-time values of the PsyoObject structure describing the known things about the object. I believe it would be possible to completely avoid the conversion of the instance back to the classical Python layout when it exits Psyco's scope, by building new subtypes of InstanceType (for classical classes) or of the class (for new-style classes). The new subtype would have special attribute getters and setters so that regular Python code can access the attributes in the non-classical layout. This might break some code, however, as instances would have a type that is a subtype of the expected type, but the case might be rare and it is probably worth trying (should be user-controllable on a class-by-class basis). Another not-quite-unrelated thing to consider: mutable containers (lists, dicts) that have constraints on their elements, e.g. to represent a list of integer objects as a mere array of 32-bit ints. (Please reply in psy...@li....) A bient=F4t, Armin. |
From: Kevin J. <ja...@pe...> - 2002-04-13 14:15:40
|
On Sat, 13 Apr 2002, Armin Rigo wrote: > Yes, I have at least planned to give it serious thoughts. Note that > using Python 2.2 new-style classes and __slots__, we force Python to use > an efficient offset-based memory layout that Psyco can readily use > efficiently. This said we must now think about letting Psyco > automatically create memory layouts for instances. This could let it > represent, say, a slot known to be an integer as just a 32-bit integer > instead of a pointer to a PyIntObject. In other words, Psyco should be > able to manage instances whose layout is just the list of the run-time > values of the PsyoObject structure describing the known things about the > object. I've been advocating minor, but important changes to the __slots__ mechanism that would make this much easier. Unfortunately, Guido has not seen the value in my suggestions. Anyway, here are my specific proposals: 1) class.__slots__ is mutable, and should not be. This makes it difficult and expensive to determine which slots belong to an instance. 2) class.__slots__ contains only the slots declared in class, but not any in class.__bases__. A flattened list that includes all slots in an instance would make reflection extremely easy. > I believe it would be possible to completely avoid the conversion of the > instance back to the classical Python layout when it exits Psyco's > scope, by building new subtypes of InstanceType (for classical classes) > or of the class (for new-style classes). The new subtype would have > special attribute getters and setters so that regular Python code can > access the attributes in the non-classical layout. This might break some > code, however, as instances would have a type that is a subtype of the > expected type, but the case might be rare and it is probably worth > trying (should be user-controllable on a class-by-class basis). Specifically, this will break code that implements __getattr__ and __setattr__ assuming that the attribute lives in instance.__dict__. This is true for both new-style and classic-classes. This also breaks code that use class-level variables as default values to avoid initialization overhead. e.g.: class Foo: bar = 1 foo = Foo() print foo.bar > 1 foo.bar = 2 print Foo.bar > 1 If the class were converted into a new-style class with 'bar' as a slot, the descriptor for 'bar' would conflict with the default value. One could introduce an additional meta-class that could patch around this, but it would be difficult to get right in all cases. > Another not-quite-unrelated thing to consider: mutable containers > (lists, dicts) that have constraints on their elements, e.g. to > represent a list of integer objects as a mere array of 32-bit ints. > I have designed several meta-classes which enforce arbitrary constraints on attributes. These could easily be used as the basis for providing type information to Psyco, and can efficiently be translated into machine level constraints. Here is how it works (lengthy source for ConstrainedObject meta-class not included, e-mail me if you want a copy): class A(object): __metaclass__ = ConstrainedObject __slots__ = {'a':int,'b':float} class B(A): __attrs__ = {'c':str} foo = B() foo.a = foo.b = foo.c = 1 assert foo.a == 1 assert foo.b == 1.0 assert foo.c == "1" assert type(foo.a) == int assert type(foo.b) == float assert type(foo.c) == str foo.a = '5' foo.b = '0x5A' foo.c = 5.4 assert foo.a == 5 assert foo.b == 1.40625 # <--- don't panic, its a bug in an older glibc, # not Python assert foo.c == "5.4" assert type(foo.a) == int assert type(foo.b) == float assert type(foo.c) == str Food for thought, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: ja...@th... Fax: (216) 986-0714 WWW: http://www.theopalgroup.com |
From: Michael H. <mw...@py...> - 2002-04-15 12:03:12
|
Kevin Jacobs <ja...@pe...> writes: > 1) class.__slots__ is mutable, and should not be. This makes it difficult > and expensive to determine which slots belong to an instance. Isn't it enough (especially in a psyco context) to just say that if you mutate __slots__ you are liable to lose in arbitrarily nasty ways? Cheers, M. -- MAN: How can I tell that the past isn't a fiction designed to account for the discrepancy between my immediate physical sensations and my state of mind? -- The Hitch-Hikers Guide to the Galaxy, Episode 12 |
From: Kevin J. <ja...@pe...> - 2002-04-15 14:52:10
|
On 15 Apr 2002, Michael Hudson wrote: > Kevin Jacobs <ja...@pe...> writes: > > > 1) class.__slots__ is mutable, and should not be. This makes it difficult > > and expensive to determine which slots belong to an instance. > > Isn't it enough (especially in a psyco context) to just say that if > you mutate __slots__ you are liable to lose in arbitrarily nasty ways? That is Guido's argument. I'd prefer that __slots__ be a canonical list of the allocated space in the object. It makes a slew of other things easier when that is held invariant. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: ja...@th... Fax: (216) 986-0714 WWW: http://www.theopalgroup.com |
From: Michael H. <mw...@py...> - 2002-04-15 15:27:25
|
Kevin Jacobs <ja...@pe...> writes: > On 15 Apr 2002, Michael Hudson wrote: > > Kevin Jacobs <ja...@pe...> writes: > > > > > 1) class.__slots__ is mutable, and should not be. This makes it difficult > > > and expensive to determine which slots belong to an instance. > > > > Isn't it enough (especially in a psyco context) to just say that if > > you mutate __slots__ you are liable to lose in arbitrarily nasty ways? > > That is Guido's argument. I'd prefer that __slots__ be a canonical list of > the allocated space in the object. It makes a slew of other things easier > when that is held invariant. How does it being invariant make things easier than assuming it's invariant? I think I more or less agree with you (that __slots__ should be immutable) but I don't understand this remark. Cheers, M. -- >> REVIEW OF THE YEAR, 2000 << It was shit. Give us another one. -- NTK Know, 2000-12-29, http://www.ntk.net/ |
From: Kevin J. <ja...@pe...> - 2002-04-15 15:36:27
|
On 15 Apr 2002, Michael Hudson wrote: > Kevin Jacobs <ja...@pe...> writes: > > That is Guido's argument. I'd prefer that __slots__ be a canonical list of > > the allocated space in the object. It makes a slew of other things easier > > when that is held invariant. > > How does it being invariant make things easier than assuming it's invariant? > > I think I more or less agree with you (that __slots__ should be > immutable) but I don't understand this remark. I would rather forbid something, rather than say that the results are undefined. Undefined scares me. A nice, deterministic, immediate, easy to fix exception makes me very happy. It means I can sleep a little better at night. By your logic, why bother enforcing class.__mro__ is read-only, or __bases__ or anything that Guido felt needed to be protected in new-style classes? I manage projects that involve applications that frequently exceed 200k lines of Python code. We perform regular code review, use baseball bats on our developers to make sure they follow our style guide, but invariably some bad code slips through. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: ja...@th... Fax: (216) 986-0714 WWW: http://www.theopalgroup.com |
From: Michael H. <mw...@py...> - 2002-04-15 15:43:20
|
Kevin Jacobs <ja...@pe...> writes: > On 15 Apr 2002, Michael Hudson wrote: > > [__slots__] > > How does it being invariant make things easier than assuming it's > > invariant? > > > > I think I more or less agree with you (that __slots__ should be > > immutable) but I don't understand this remark. > > I would rather forbid something, rather than say that the results are > undefined. Undefined scares me. But this is psyco-devel... in this world rebinding builtins leads to undefined behaviour. I'll stop now. Cheers, M. -- NUTRIMAT: That drink was individually tailored to meet your personal requirements for nutrition and pleasure. ARTHUR: Ah. So I'm a masochist on a diet am I? -- The Hitch-Hikers Guide to the Galaxy, Episode 9 |
From: Armin R. <ar...@ul...> - 2002-04-15 18:22:21
|
Hello Kevin, ----- Original Message ----- > 1) class.__slots__ is mutable, and should not be. This makes it difficult > and expensive to determine which slots belong to an instance. Yes, this makes the semantics confusing. What occurs if __slots__ is mutated? The layout of the instances is computed when the class is create= d, so that I guess that the only thing that matters is the value __slots__ h= ad at this time. This means that __slots__ should not be modified by program= , otherwise it will simply get out of sync with what Python will continue t= o see as allowed attributes. > 2) class.__slots__ contains only the slots declared in class, but no= t any > in class.__bases__. A flattened list that includes all slots in a= n > instance would make reflection extremely easy. Here I don't agree with you. The purpose of __slots__ is to list the attributes that this class wants to add; it doesn't seem to make sense to list the parent attributes as well -- and what if you list less attribute= s, or even if a parent class allows arbitrary attributes? Besides, walking t= he parent classes looking for __slots__ is easy. > (...about special Psyco metaclasses...) > > Specifically, this will break code that implements __getattr__ and > __setattr__ assuming that the attribute lives in instance.__dict__. Th= is is > true for both new-style and classic-classes. Well, this might be emulated (by falling back to the non-optimized case), when someone actually tries to read the __dict__ attribute. > This also breaks code that use class-level variables as default values = to avoid > initialization overhead. This too could be emulated. I am now thinking about asking the user to explicitely mark its classes a= s Psyco-metaclassed. This is a way to make him aware that the semantics of = its classes are different: they are read-only after creation, i.e. you initialize them with the Python syntax but cannot change any attribute or method in it thereafter. Such classes are still the common case in Python= , and would let Psyco do a good and clean job. Currently, changing class methods can crash Psyco; with a custom metaclass we could raise AttributeError or "ReadOnlyError" instead. It would also make it possible= to later implement class attribute changes, e.g. by falling back to somethin= g slower. The trick of class-level default values would mean that by default (as fo= r methods) we assume that an instance attribute generally does not shadow a= ny class attribute; but the case could be detected at attribute assignment time. > > Another not-quite-unrelated thing to consider: mutable containers > > (lists, dicts) that have constraints on their elements, e.g. to > > represent a list of integer objects as a mere array of 32-bit ints. > > I have designed several meta-classes which enforce arbitrary constraint= s on > attributes. I was actually thinking about automatically deriving the constraints, not introducing types to the language level (with or without supporting synta= x). This is more in sync of the current approach. I even think it should not = be too complex any more. A bient=F4t, Armin. |
From: Kevin J. <ja...@pe...> - 2002-04-15 19:01:00
|
On Mon, 15 Apr 2002, Armin Rigo wrote: > ----- Original Message ----- > > 1) class.__slots__ is mutable, and should not be. This makes it > difficult > > and expensive to determine which slots belong to an instance. > > Yes, this makes the semantics confusing. What occurs if __slots__ is > mutated? The layout of the instances is computed when the class is created, > so that I guess that the only thing that matters is the value __slots__ had > at this time. This means that __slots__ should not be modified by program, > otherwise it will simply get out of sync with what Python will continue to > see as allowed attributes. Well, the problem is not what Python sees; Python keeps its own immutable list of slots hidden inside new-style objects. Application and framework programmers have to beware and not trust the current value of __slots__ on any instance. > > 2) class.__slots__ contains only the slots declared in class, but not > any > > in class.__bases__. A flattened list that includes all slots in an > > instance would make reflection extremely easy. > > Here I don't agree with you. The purpose of __slots__ is to list the > attributes that this class wants to add; it doesn't seem to make sense to > list the parent attributes as well -- and what if you list less attributes, > or even if a parent class allows arbitrary attributes? Besides, walking the > parent classes looking for __slots__ is easy. I was not advocating having the user specify all anscestor slots. Rather, it would be nice if the class object produced this flattened list during class construction (I use a meta-class to do this myself). e.g.: class Foo(object): __slots__ = ('foo',) class Bar(Foo): __slots__ = ('bar',) print Bar.__slots__ > ('foo','bar') While walking anscestor classes is not very expensive, it does introduce what I see as unnecessary overhead when pickling/mashaling new-style objects. > I am now thinking about asking the user to explicitely mark its classes as > Psyco-metaclassed. Exactly! My ConstrainedObject meta-class is one way for a application developer to explicitly delcare that they are willing to live with additional rules and constraints. > > > Another not-quite-unrelated thing to consider: mutable containers > > > (lists, dicts) that have constraints on their elements, e.g. to > > > represent a list of integer objects as a mere array of 32-bit ints. > > > > I have designed several meta-classes which enforce arbitrary constraints > on > > attributes. > > I was actually thinking about automatically deriving the constraints, not > introducing types to the language level (with or without supporting syntax). > This is more in sync of the current approach. I even think it should not be > too complex any more. Optimal code generation with type-constraints and type-constraint deduction seem like orthogonal issues. Why not use type declarations, like my ConstrainedObject meta-class, in situations where types are well-defined and make sure that the code generation is robust. Then worry about getting type constraint detection working. Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: ja...@th... Fax: (216) 986-0714 WWW: http://www.theopalgroup.com |
From: Armin R. <ar...@ul...> - 2002-04-17 13:49:07
|
Hello Kevin, On Mon, 15 Apr 2002, Kevin Jacobs wrote: > it would be nice if the class object produced this flattened list during > class construction (I use a meta-class to do this myself). Ok, there I understand your point, although it might be confusing to modify the content of __slots__ during construction. I'd prefer myself to have a function that lists slots for me, but I can understand both points of view. For the performance problems I'd suggest that a built-in function should list the slots as Python sees them (which can be done quickly). > Optimal code generation with type-constraints and type-constraint deduction > seem like orthogonal issues. Why not use type declarations, like my > ConstrainedObject meta-class, in situations where types are well-defined > and make sure that the code generation is robust. Then worry about getting > type constraint detection working. No. Psyco is all about dynamic constraint inference, and I would like to see how far we can go in this direction with no constraint declarations, in a style compatible with the current Python approach. Constraint declarations are a very interesting topic, and Python would in my opinion greatly benefit from nice typing systems, but this is an orthogonal issue for Psyco. I would say that if one wants to go in this direction, it should not be discussed from the Psyco point of view; we first need a good, well-tested, widely-accepted core Python extension, and then worry about using this information as hints to Psyco. A bientot, Armin. |
From: Kevin J. <ja...@pe...> - 2002-04-17 14:04:52
|
On Wed, 17 Apr 2002, Armin Rigo wrote: > Hello Kevin, > > On Mon, 15 Apr 2002, Kevin Jacobs wrote: > > it would be nice if the class object produced this flattened list during > > class construction (I use a meta-class to do this myself). > > Ok, there I understand your point, although it might be confusing to > modify the content of __slots__ during construction. I'd prefer myself to > have a function that lists slots for me, but I can understand both points > of view. For the performance problems I'd suggest that a built-in function > should list the slots as Python sees them (which can be done quickly). The problem is that Python does Not do this quickly! At least not when enumerating slots. Slot resolution entails multiple dictionary lookups for each class in the MRO until a descriptor is found. Enumerating all slots requires iterating through all members of all base classes and testing each member to see if it is a descriptor for the right class type. One can also delete descriptors, so slots can be totally hidden from Python. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: ja...@th... Fax: (216) 986-0714 WWW: http://www.theopalgroup.com |