pyobjc-dev

[Pyobjc-dev] ObjC GC with PyObjC?

[Barry W. <bar...@gm...>]

Can anyone comment on the current state of PyObjC in relation to the
Objective-C garbage collector? As @bbum said on Twitter: "... I just
don't write desktop code w/o GC anymore. Waste of time not to use GC."
In a PyObjC app that loads Objective-C code via plugin, can we use GC
in the plugin code?

Thanks,
Barry

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 3 Jun, 2009, at 0:19, Barry Wark wrote:

> Can anyone comment on the current state of PyObjC in relation to the
> Objective-C garbage collector? As @bbum said on Twitter: "... I just
> don't write desktop code w/o GC anymore. Waste of time not to use GC."
> In a PyObjC app that loads Objective-C code via plugin, can we use GC
> in the plugin code?

No. PyObjC application won't use garbage collection. Furthermore, even  
though I have added code to PyObjC that should make PyObjC safe when  
used in a GC environment (such as using CFRetain/CFRelease instead of - 
retain/-release) I haven't tested yet if that actually works as  
designed.   And to be honest, I haven't even checked yet how I ensure  
that GC is turned on automaticly.

Actually getting PyObjC to be to a point where it is safe to use in GC  
environment would be several days work, which would be needed to  
create an environment where the application runs with the ObjC GC  
turned on and then run tests to ensure that PyObjC keeps working.

If I had the time, and funding, I'd much rather work on a port of  
Python that uses the ObjC garbage collector instead of Python's native  
memory management. That would allow for a much improved integration  
between Python and Objective-C, and could remove a number of issues  
where the bridge is not as transparent as I'd like:  KVO could be made  
to work with native Python objects, no need to worry about ObjC-style  
weak references where you have to keep the proxy object alive manually  
(e.g. the outlineview dataprovider protocol).

Sadly enough this would probably require several weeks of full-time  
work, and that's not something I'm able to pull of without funding.

Ronald

>
> Thanks,
> Barry
>
> ------------------------------------------------------------------------------
> OpenSolaris 2009.06 is a cutting edge operating system for enterprises
> looking to deploy the next generation of Solaris that includes the  
> latest
> innovations from Sun and the OpenSource community. Download a copy and
> enjoy capabilities such as Networking, Storage and Virtualization.
> Go to: http://p.sf.net/sfu/opensolaris-get
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Greg E. <gre...@ca...>]

Ronald Oussoren wrote:

> If I had the time, and funding, I'd much rather work on a port of  
> Python that uses the ObjC garbage collector instead of Python's native  
> memory management.

While that might help PyObjC, I'd be very worried about
what effect it might have on the performance of the rest
of Python.

Also I'm not sure I like the idea of having to use a
special version of Python in order to be able to use
ObjC.

In general, I think that "create a special version of
Python to make X work" is an anti-pattern. Think about
what would happen if everyone did that, and then you
wanted to use X and Y at the same time, where X and Y
each required their own special Python version. It would
be DLL hell on steroids.

-- 
Greg

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 4 Jun, 2009, at 0:57, Greg Ewing wrote:

> Ronald Oussoren wrote:
>
>> If I had the time, and funding, I'd much rather work on a port of
>> Python that uses the ObjC garbage collector instead of Python's  
>> native
>> memory management.
>
> While that might help PyObjC, I'd be very worried about
> what effect it might have on the performance of the rest
> of Python.
>
> Also I'm not sure I like the idea of having to use a
> special version of Python in order to be able to use
> ObjC.
>
> In general, I think that "create a special version of
> Python to make X work" is an anti-pattern. Think about
> what would happen if everyone did that, and then you
> wanted to use X and Y at the same time, where X and Y
> each required their own special Python version. It would
> be DLL hell on steroids.

I don't agree. There could be advantages for regular python as well,  
IMHO removing CPython's refcounting is a necessary first step for  
removing the GIL.  Removing the GIL would definitely be an advantage  
for general python code, especially because using multiple independent  
threads of execution is the only way to seriously improve performance  
on modern hardware because sadly enough the "wait a year and then my  
single-threaded code is twice as fast"-era is over. Multiprocessing is  
a cool hack, but isn't a real solution; I have a number of programs  
that use multiple threads and need shared memory to get fast enough  
communications (basicly because large objects get transferred between  
threads).

Ronald
>
> -- 
> Greg
>
> ------------------------------------------------------------------------------
> OpenSolaris 2009.06 is a cutting edge operating system for enterprises
> looking to deploy the next generation of Solaris that includes the  
> latest
> innovations from Sun and the OpenSource community. Download a copy and
> enjoy capabilities such as Networking, Storage and Virtualization.
> Go to: http://p.sf.net/sfu/opensolaris-get
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Jonathan S. <sa...@gm...>]

Ronald Oussoren wrote:
>
> I don't agree. There could be advantages for regular python as well, 
> IMHO removing CPython's refcounting is a necessary first step for 
> removing the GIL.  Removing the GIL would definitely be an advantage 
> for general python code, especially because using multiple independent 
> threads of execution is the only way to seriously improve performance 
> on modern hardware because sadly enough the "wait a year and then my 
> single-threaded code is twice as fast"-era is over. Multiprocessing is 
> a cool hack, but isn't a real solution; I have a number of programs 
> that use multiple threads and need shared memory to get fast enough 
> communications (basicly because large objects get transferred between 
> threads).
>
> Ronald
As a curiosity for those who know the internals of Python better than I 
do, is there any reason to believe that most existing python code would 
even "care" which flavor of GC it's running under?


Jonathan 'usually lurking' Saggau

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Barry W. <bar...@gm...>]

On Thu, Jun 4, 2009 at 8:48 AM, Jonathan Saggau<sa...@gm...> wrote:
> Ronald Oussoren wrote:
>>
>> I don't agree. There could be advantages for regular python as well,
>> IMHO removing CPython's refcounting is a necessary first step for
>> removing the GIL.  Removing the GIL would definitely be an advantage
>> for general python code, especially because using multiple independent
>> threads of execution is the only way to seriously improve performance
>> on modern hardware because sadly enough the "wait a year and then my
>> single-threaded code is twice as fast"-era is over. Multiprocessing is
>> a cool hack, but isn't a real solution; I have a number of programs
>> that use multiple threads and need shared memory to get fast enough
>> communications (basicly because large objects get transferred between
>> threads).
>>
>> Ronald
> As a curiosity for those who know the internals of Python better than I
> do, is there any reason to believe that most existing python code would
> even "care" which flavor of GC it's running under?

As an aside, the Google guys think that they can pull this off without
affecting even C-language extensions. I just discovered the Unladed
Swallow project (http://code.google.com/p/unladen-swallow/) appears to
be their attempt to reimplement the Python VM on top of LLVM. Their
goals are performance, but they also mention pluggable GC and some
other very cool tricks. I think it would definitely be worth keeping
an eye on this project. If it progresses as they claim, it may be a
much better starting point for any efforts to bring Python and the
ObjC runtime closer.

-B

>
>
> Jonathan 'usually lurking' Saggau
>
> ------------------------------------------------------------------------------
> OpenSolaris 2009.06 is a cutting edge operating system for enterprises
> looking to deploy the next generation of Solaris that includes the latest
> innovations from Sun and the OpenSource community. Download a copy and
> enjoy capabilities such as Networking, Storage and Virtualization.
> Go to: http://p.sf.net/sfu/opensolaris-get
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 4 Jun, 2009, at 17:48, Jonathan Saggau wrote:

> Ronald Oussoren wrote:
>>
>> I don't agree. There could be advantages for regular python as  
>> well, IMHO removing CPython's refcounting is a necessary first step  
>> for removing the GIL.  Removing the GIL would definitely be an  
>> advantage for general python code, especially because using  
>> multiple independent threads of execution is the only way to  
>> seriously improve performance on modern hardware because sadly  
>> enough the "wait a year and then my single-threaded code is twice  
>> as fast"-era is over. Multiprocessing is a cool hack, but isn't a  
>> real solution; I have a number of programs that use multiple  
>> threads and need shared memory to get fast enough communications  
>> (basicly because large objects get transferred between threads).
>>
>> Ronald
> As a curiosity for those who know the internals of Python better  
> than I do, is there any reason to believe that most existing python  
> code would even "care" which flavor of GC it's running under?

Sadly enough, yes.  Code like "data = open(filename, 'r').read()" is  
pretty common in small python scripts, and mostly works fine in a  
refcounting python, but doesn't work as well with other GCs because  
the result of open might not be collected until a long time after the  
file is used. That's no problem in general, but is a problem when  
dealing with scarce resources such as file handles.

Such code would already run into problems on Jython, and possibly on  
IronPython as well. When you use a modern version of python the  
correct way to write this isn't much more code:

:>   with open(filename 'r') as fp:
:>       data = fp.read()

In earlier versions of python this would have to be written as:

:>   fp = open(filename, 'r')
:>   try:
:>      data = fp.read()
:>   finally:
:>      fp.close()

Which explains why a lot of people would rely on an implementation  
detail of the garbage collector ;-).

The really interesting questions are if there is code like this in  
major libraries (such as sqlalchemy or twisted), and if there is code  
that relies on really obscure edge-cases that might be hard to  
reproduce with another collector.  In CPython you can revive an object  
in the __del__ of a class. That is:


:>   junk = []
:>   class MyClass:
:>        def __del__(self):
:>            junk.append(self)
:>
:>   a = MyClass()
:>  del a

In this snippet an instance of MyClass is created, and immediately  
after that the only reference to it is removed and CPython tries to  
delete the object. The __del__ method then adds the object to a global  
list, and therefore revives the object.

Code like this cannot work with Apple's GC, that collector requires  
that objects go away when the GC finalizes them.  I sincerely hope  
that no-one actually uses this loophole in production code, it is  
really nasty behaviour.


Ronald

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 4 Jun, 2009, at 0:57, Greg Ewing wrote:

> Ronald Oussoren wrote:
>
>> If I had the time, and funding, I'd much rather work on a port of
>> Python that uses the ObjC garbage collector instead of Python's  
>> native
>> memory management.
>
> While that might help PyObjC, I'd be very worried about
> what effect it might have on the performance of the rest
> of Python.
>
> Also I'm not sure I like the idea of having to use a
> special version of Python in order to be able to use
> ObjC.
>
> In general, I think that "create a special version of
> Python to make X work" is an anti-pattern. Think about
> what would happen if everyone did that, and then you
> wanted to use X and Y at the same time, where X and Y
> each required their own special Python version. It would
> be DLL hell on steroids.

BTW. We already have this in a way, there's Jython if you want to use  
Java, IronPython if you want to use .NET, .... and let's not forget  
Stackless Python.

There is a risk for fragmentation, but I don't think this is a very  
large risk because maintaining a patched version of Python is a  
relatively large amount  of work and it is therefore for most usecases  
way more convenient to not do that.

If I'd ever get around to creating a port of Python that uses the  
Python GC I'd try to keep it as close as possible to the real one, any  
deviation from CPython is a possibility for maintenance problems later  
on and would therefore need to have a big advantage to offset the  
maintenance cost.   Another thing that's worrisome is 3th-party  
extensions, one of the reasons to like Python is the large set of  
packages at PyPI, loosing access to those would seriously reduce the  
uptake of a patched version of python.

Ronald
>
> -- 
> Greg
>
> ------------------------------------------------------------------------------
> OpenSolaris 2009.06 is a cutting edge operating system for enterprises
> looking to deploy the next generation of Solaris that includes the  
> latest
> innovations from Sun and the OpenSource community. Download a copy and
> enjoy capabilities such as Networking, Storage and Virtualization.
> Go to: http://p.sf.net/sfu/opensolaris-get
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 3 Jun, 2009, at 23:54, Ronald Oussoren wrote:
>>
>
> I don't agree. There could be advantages for regular python as well,  
> IMHO removing CPython's refcounting is a necessary first step for  
> removing the GIL.  Removing the GIL would definitely be an advantage  
> for general python code, especially because using multiple  
> independent threads of execution is the only way to seriously  
> improve performance on modern hardware because sadly enough the  
> "wait a year and then my single-threaded code is twice as fast"-era  
> is over. Multiprocessing is a cool hack, but isn't a real solution;  
> I have a number of programs that use multiple threads and need  
> shared memory to get fast enough communications (basicly because  
> large objects get transferred between threads).

I have another reason for wanting this: better decoupling model code  
from the GUI.

I'm sure I'm not the only one that either has to jump through hoops to  
expose Python objects to the GUI (e.g. using the output of a library  
like SQLAlchemy in a GUI that uses Key-Value Bindings), or uses Cocoa  
datastructures such as NSArray in code that basicly has nothing to do  
with the GUI.

A version of Python that has more pervasive support for ObjC classes  
could make it easier to keep GUI code and non-GUI code.

Ronald

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Ronald O. <ron...@ma...>]

On 10 Jun, 2009, at 7:55, Barry Wark wrote:

> On Thu, Jun 4, 2009 at 8:48 AM, Jonathan Saggau<sa...@gm...>  
> wrote:
>> Ronald Oussoren wrote:
>>>
>>> I don't agree. There could be advantages for regular python as well,
>>> IMHO removing CPython's refcounting is a necessary first step for
>>> removing the GIL.  Removing the GIL would definitely be an advantage
>>> for general python code, especially because using multiple  
>>> independent
>>> threads of execution is the only way to seriously improve  
>>> performance
>>> on modern hardware because sadly enough the "wait a year and then my
>>> single-threaded code is twice as fast"-era is over.  
>>> Multiprocessing is
>>> a cool hack, but isn't a real solution; I have a number of programs
>>> that use multiple threads and need shared memory to get fast enough
>>> communications (basicly because large objects get transferred  
>>> between
>>> threads).
>>>
>>> Ronald
>> As a curiosity for those who know the internals of Python better  
>> than I
>> do, is there any reason to believe that most existing python code  
>> would
>> even "care" which flavor of GC it's running under?
>
> As an aside, the Google guys think that they can pull this off without
> affecting even C-language extensions. I just discovered the Unladed
> Swallow project (http://code.google.com/p/unladen-swallow/) appears to
> be their attempt to reimplement the Python VM on top of LLVM. Their
> goals are performance, but they also mention pluggable GC and some
> other very cool tricks. I think it would definitely be worth keeping
> an eye on this project. If it progresses as they claim, it may be a
> much better starting point for any efforts to bring Python and the
> ObjC runtime closer.

I know of Unladen Swallow and tracking their progress.  I've spoken  
with someone on their team at Pycon and they are very confident that  
they can pull of their performance goals, basicly by going through the  
available literature and implementing those ideas. That's easier said  
than done, but Google doesn't have a shortage of PhD's and offices of  
the Unladen Swallow team are close to that of some of the authors of  
research papers that they intend to use.

I'm not so sure about not affecting C extensions, especially not in  
the context of the ObjC garbage collector due to some special  
limitations.

BTW. The bit that excites me even more than the performance work is  
their secondairy goal of getting rid of the GIL. I have a number of  
projects where I have written a custom C extension just to make use of  
additional cores and where I would have been able to use an existing  
extension or even pure python code if Python hadn't had the GIL. (And  
no, multiprocessing wouldn't work in those projects, the communication  
overhead between processes would have been too large).

Ronald
> -B
>
>>
>>
>> Jonathan 'usually lurking' Saggau
>>
>> ------------------------------------------------------------------------------
>> OpenSolaris 2009.06 is a cutting edge operating system for  
>> enterprises
>> looking to deploy the next generation of Solaris that includes the  
>> latest
>> innovations from Sun and the OpenSource community. Download a copy  
>> and
>> enjoy capabilities such as Networking, Storage and Virtualization.
>> Go to: http://p.sf.net/sfu/opensolaris-get
>> _______________________________________________
>> Pyobjc-dev mailing list
>> Pyo...@li...
>> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>>

Re: [Pyobjc-dev] ObjC GC with PyObjC?

[Barry W. <bar...@gm...>]

On Wed, Jun 10, 2009 at 9:05 AM, Ronald Oussoren<ron...@ma...> wrote:
>
> On 10 Jun, 2009, at 7:55, Barry Wark wrote:
>
>> On Thu, Jun 4, 2009 at 8:48 AM, Jonathan Saggau<sa...@gm...> wrote:
>>>
>>> Ronald Oussoren wrote:
>>>>
>>>> I don't agree. There could be advantages for regular python as well,
>>>> IMHO removing CPython's refcounting is a necessary first step for
>>>> removing the GIL.  Removing the GIL would definitely be an advantage
>>>> for general python code, especially because using multiple independent
>>>> threads of execution is the only way to seriously improve performance
>>>> on modern hardware because sadly enough the "wait a year and then my
>>>> single-threaded code is twice as fast"-era is over. Multiprocessing is
>>>> a cool hack, but isn't a real solution; I have a number of programs
>>>> that use multiple threads and need shared memory to get fast enough
>>>> communications (basicly because large objects get transferred between
>>>> threads).
>>>>
>>>> Ronald
>>>
>>> As a curiosity for those who know the internals of Python better than I
>>> do, is there any reason to believe that most existing python code would
>>> even "care" which flavor of GC it's running under?
>>
>> As an aside, the Google guys think that they can pull this off without
>> affecting even C-language extensions. I just discovered the Unladed
>> Swallow project (http://code.google.com/p/unladen-swallow/) appears to
>> be their attempt to reimplement the Python VM on top of LLVM. Their
>> goals are performance, but they also mention pluggable GC and some
>> other very cool tricks. I think it would definitely be worth keeping
>> an eye on this project. If it progresses as they claim, it may be a
>> much better starting point for any efforts to bring Python and the
>> ObjC runtime closer.
>
> I know of Unladen Swallow and tracking their progress.  I've spoken with
> someone on their team at Pycon and they are very confident that they can
> pull of their performance goals, basicly by going through the available
> literature and implementing those ideas. That's easier said than done, but
> Google doesn't have a shortage of PhD's and offices of the Unladen Swallow
> team are close to that of some of the authors of research papers that they
> intend to use.
>
> I'm not so sure about not affecting C extensions, especially not in the
> context of the ObjC garbage collector due to some special limitations.

My (limited!) understanding of the Python memory semantics wrt C
extensions made me think this was going to be tough too, but you never
know. As you said, they have some very smart folks there.

>
> BTW. The bit that excites me even more than the performance work is their
> secondairy goal of getting rid of the GIL. I have a number of projects where
> I have written a custom C extension just to make use of additional cores and
> where I would have been able to use an existing extension or even pure
> python code if Python hadn't had the GIL. (And no, multiprocessing wouldn't
> work in those projects, the communication overhead between processes would
> have been too large).

Yes, getting rid of the GIL would be huge.

>
> Ronald
>>
>> -B
>>
>>>
>>>
>>> Jonathan 'usually lurking' Saggau
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> OpenSolaris 2009.06 is a cutting edge operating system for enterprises
>>> looking to deploy the next generation of Solaris that includes the latest
>>> innovations from Sun and the OpenSource community. Download a copy and
>>> enjoy capabilities such as Networking, Storage and Virtualization.
>>> Go to: http://p.sf.net/sfu/opensolaris-get
>>> _______________________________________________
>>> Pyobjc-dev mailing list
>>> Pyo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>>>
>
>