pyobjc-dev

[Pyobjc-dev] PyObjC and macOS 10.12 (Sierra)

[Ronald O. <ron...@ma...>]

Hi,

I’m slowly working my way through the SDK headers for macOS 10.12 during idle time at EuroPython2016, from the frameworks with small diffs to those with larger diffs.  

There are at the moment about 26 frameworks I haven’t looked at at all, and another 10 where I know I have to do some more work. The rest should have up to date metadata on my machine, but haven’t been tested on macOS 10.12 yet (and that won’t change until I get around to actually installing a VM running 10.12).

All of this was done using the headers in the Xcode 8 beta 2, I haven’t looked at the incremental changes to the SDK in beta 3 yet (and likely won’t until I have made a pass through all frameworks).  That said, the diffs from beta 2 to later beta’s and the final release should be fairly small (he wrote hopefully).

What needs to be done to get proper support for 10.12:

* Finish updating the metadata.  I suspect that this is a couple of days work, I have worked may way through the diffs of a fairly large number of frameworks, but haven’t looked at the more complicated ones yet (such as Foundation and AppKit). Also, a fairly large subset of the frameworks I have looked at didn’t have any significant updates compared to 10.11 (but some of them still had large textual diffs due to restructuring of header files).

* Install a 10.12 VM and run tests there. This will likely result in more work, there’s already a pull request about NSSecureCoder warnings on 10.12 that needs looking into and there’s bound to be more issues.

* Do the build and test dance on older OSX releases. 
   The annoying bit for that: there’s a know issue with building on OSX 10.7, see issue #100. I haven’t been able to look into that yet
   because I don’t have an VM with 10.7 on my laptop and haven’t been able to install 10.7 there yet. I probably have such a VM on
   an external disk somewhere, but I’m not sure about that.

When all that is done I can do a release that includes 10.12 support, with some luck around the time of the 10.12 release itself. I’ll definitely push the metadata updates to bitbucket once I’ve finished looking through the 10.12 SDK diffs.

In the longer run I want to provide wheels for PyObjC, but that requires more work. In particular, I only want to do that after automating more of the release work because I’ve noticed that I’m already postponing doing new releases due to the amount of work and don’t want to make things worse in that regard.

BTW. I’ll likely won’t work on metadata updates during the EP2016 sprints, my current plan is to keep the tradition of working on PEP 447 alive and try to actually get it accepted this year. 

Ronald

PS. “A couple of days work” probably means that it will take me several calendar weeks to actually finish the work because I have limited time to work on this and don’t want to spent all that time on this rather tedious work.

[#100]: https://bitbucket.org/ronaldoussoren/pyobjc/issues/100/cannot-find-interface-declaration-for

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Jul 22, 2016, at 6:46 AM, Ronald Oussoren <ron...@ma...> wrote:
> 
> I’m slowly working my way through the SDK headers for macOS 10.12 during idle time at EuroPython2016, from the frameworks with small diffs to those with larger diffs.  

I don't have any technical content to add here, but I do just want to say thank you Ronald for all the work that you're doing to keep pace with new APIs and OSes and make it possible for us all to hack our Macs with Python.

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 11, 2016, at 3:12 AM, Andrew Jaffe <a.h...@gm...> wrote:
> 
> If I do the usual "pip --upgrade" for these, it fails, seemingly because of permissions (Apologies, but I don't have access to the messages anymore): it is clearly trying to delete these versions which seem to live in /System/Library/Frameworks/Python.framework/Versions/2.7/. This fails, of course, due to permissions (and system integrity protection).

This is expected, and desired, even.  Don't install packages into your system Python.  Make a virtualenv and install pyobjc there, or do a `pip install --user`.  That should work fine.

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

On 11/09/2016 20:57, Glyph Lefkowitz wrote:
>
>> On Sep 11, 2016, at 3:12 AM, Andrew Jaffe <a.h...@gm...
>> <mailto:a.h...@gm...>> wrote:
>>
>> If I do the usual "pip --upgrade" for these, it fails, seemingly
>> because of permissions (Apologies, but I don't have access to the
>> messages anymore): it is clearly trying to delete these versions which
>> seem to live in
>> /System/Library/Frameworks/Python.framework/Versions/2.7/. This fails,
>> of course, due to permissions (and system integrity protection).
>
> This is expected, and desired, even.  Don't install packages into your
> system Python.  Make a virtualenv and install pyobjc there, or do a `pip
> install --user`.  That should work fine.

Hi, sorry if I wasn't clear -- this is a python.org install, and the pip 
I'm running comes from that. It shouldn't try to change the 
system-installed packages; it never has in the past, and that's why this 
behavior seems strange.

Re: [Pyobjc-dev] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

[Apologies if more than one copy of this message is posted!]

On 11/09/2016 20:57, Glyph Lefkowitz wrote:
>
>> On Sep 11, 2016, at 3:12 AM, Andrew Jaffe <a.h...@gm...
>> <mailto:a.h...@gm...>> wrote:
>>
>> If I do the usual "pip --upgrade" for these, it fails, seemingly
>> because of permissions (Apologies, but I don't have access to the
>> messages anymore): it is clearly trying to delete these versions which
>> seem to live in
>> /System/Library/Frameworks/Python.framework/Versions/2.7/. This fails,
>> of course, due to permissions (and system integrity protection).
>
> This is expected, and desired, even.  Don't install packages into your
> system Python.  Make a virtualenv and install pyobjc there, or do a `pip
> install --user`.  That should work fine.

Sorry if I'm not being clear. I'm using python.org framework python, 
which supplies its own pip, and its own location for packages: 
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages

In the past, upgrades have never been a problem, so I don't know if 
anything has changed with the latest macOS, or the latest pyobjc, or the 
latest python build, or... And I don't know if the "--ignore-installed" 
solution is a workaround, or actually the right answer.

(I do know about virtualenv, but I don't think that's the route I want 
to go at this point, and I'm pretty sure I shouldn't have to.)

Yours,

Andrew

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Jack J. <jac...@cw...>]

I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.

Could it be that you’ve installed pyobjc a couple of OSX releases ago?

And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?

A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.

Jack
> On  13-Sep-2016, at 20:50 , Andrew Jaffe <a.h...@gm...> wrote:
> 
> Dear Chris and Glyph,
>> 
>> On Sunday, September 11, 2016, Andrew Jaffe <a.h...@gm...> wrote:
>> Dear Ronald,
>> 
>> Thanks, as usual, for all this.
>> 
>> I have upgraded to the GM version of 10.12 on the beta track. I use the python.org framework builds of python.
>> 
>> When I do "pip list --outdated", I get a long list:
> 
> ….
> 
>> If I do the usual "pip --upgrade" for these, it fails, seemingly because of permissions (Apologies, but I don't have access to the messages anymore): it is clearly trying to delete these versions which seem to live in /System/Library/Frameworks/Python.framework/Versions/2.7/. This fails, of course, due to permissions (and system integrity protection).
>> 
>> You can, in fact, do the upgrade with the "--ignore-installed" flag in pip (although there's still a problem with pyobjc-framework-Message).
>> 
>> So: are these errors expected? Is it something in my particular setup? Or the beta program? Is "--ignore-installed" the correct solution?
>> 
>> Thanks,
>> 
>> Andrew
>> 
>>> On 13 Sep 2016, at 16:13, Christopher Barker <pyt...@gm...> wrote:
>>> 
>>> If you are dealing with stuff in /System, then you are dealing with the Ape installed Python, not the Python.org build.
>>> 
>>> It's easy for "pip" and python to get out of sync. 
>>> 
>>> Try "which pip" to see which pip you are running.
>>> 
>>> This is why I recommend encoding pip via:
>>> 
>>> python -m pip [args]
> 
> [I tried to comment via the gmane newsgroup version of the lists and my messages never appeared so I hope this works…]
> 
> Just to be clear, I am indeed using the framework pip, which is part of why this is so confusing. In the past, as far as I know, it’s never tried to have anything to do with /System. Again, I don’t know if it’s something about macOS 10.12 Sierra, or my particular setup that is the problem. Also, to be clear, there are no PYTHON* or other environment variables that would affect the python path. Here is a failing session…
> 
> (Aside: I understand that "pyobjc-framework-Message” is actually a bad example: it is the only package that even —-ignore-installed actually fails on, because it has the "--single-version-externally-managed” problem which is a different kettle of fish. But I think the above error message is the same as the one that I would get for any of the other packages in question. Note that pip2 install works fine for packages other than those in my original list.)
> 
> $ command which pip2
> /Library/Frameworks/Python.framework/Versions/2.7/bin/pip2
> 
> $ pip2 install --upgrade pyobjc-framework-Message
> Collecting pyobjc-framework-Message
>  Using cached pyobjc-framework-Message-3.1.1.tar.gz
> Requirement already up-to-date: pyobjc-core>=3.1.1 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from pyobjc-framework-Message)
> Requirement already up-to-date: pyobjc-framework-Cocoa>=3.1.1 in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from pyobjc-framework-Message)
> Installing collected packages: pyobjc-framework-Message
>  Found existing installation: pyobjc-framework-Message 2.5.1
>    Uninstalling pyobjc-framework-Message-2.5.1:
> Exception:
> Traceback (most recent call last):
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
>    status = self.run(options, args)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
>    prefix=options.prefix_path,
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/req/req_set.py", line 736, in install
>    requirement.uninstall(auto_confirm=True)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/req/req_install.py", line 742, in uninstall
>    paths_to_remove.remove(auto_confirm)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
>    renames(path, new_path)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames
>    shutil.move(old, new)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
>    copytree(src, real_dst, symlinks=True)
>  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
>    raise Error, errors
> Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.py', '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.py', "[Errno 1] Operation not permitted: '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.pyc', '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.pyc', "[Errno 1] Operation not permitted: '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.py', '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.py', "[Errno 1] Operation not permitted: '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.pyc', '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.pyc', "[Errno 1] Operation not permitted: '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message/_metadata.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message', '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message', "[Errno 1] Operation not permitted: '/var/folders/bk/zdhpxqj14y5fzcvlpybb4b640000gn/T/pip-Gd3OiE-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC/Message'”)]
> 
> 
> _______________________________________________
> Pythonmac-SIG maillist  -  Pyt...@py...
> https://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: https://mail.python.org/mailman/options/Pythonmac-SIG

--
Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 13, 2016, at 12:05 PM, Jack Jansen <Jac...@cw...> wrote:
> 
> I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.
> 
> Could it be that you’ve installed pyobjc a couple of OSX releases ago?

This is always worth checking ;).  Particularly if it was a few Setuptools releases ago.  Also worth checking: ~/Library/Python.

> And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?

SIP locks down /System, not /Library.

> A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.


This should be an _absolute_ last resort, though.  You should be able to clean out /Library just fine.  If you have a /Library/Frameworks/Python.framework, that's probably Python.org <http://python.org/> python, not system python.

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Jack J. <jac...@cw...>]

You’re absolutely right (both on SIP and on /Library/Frameworks/Python.framework probably being a python.org install), sorry for the confusion.

This seems to be due to the way Apple has done the “Extras” directory, and adding things there to sys.path.

See for example https://github.com/pypa/pip/issues/2468

If you can get rid of /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python in sys.path you should be all set.

> On  13-Sep-2016, at 22:59 , Glyph Lefkowitz <gl...@tw...> wrote:
> 
> 
>> On Sep 13, 2016, at 12:05 PM, Jack Jansen <Jac...@cw...> wrote:
>> 
>> I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.
>> 
>> Could it be that you’ve installed pyobjc a couple of OSX releases ago?
> 
> This is always worth checking ;).  Particularly if it was a few Setuptools releases ago.  Also worth checking: ~/Library/Python.
> 
>> And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?
> 
> SIP locks down /System, not /Library.
> 
>> A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.
> 
> This should be an _absolute_ last resort, though.  You should be able to clean out /Library just fine.  If you have a /Library/Frameworks/Python.framework, that's probably Python.org python, not system python.
> 
> -glyph

--
Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

Hi,


> On 13 Sep 2016, at 22:26, Jack Jansen <jac...@cw...> wrote:
> 
> You’re absolutely right (both on SIP and on /Library/Frameworks/Python.framework probably being a python.org install), sorry for the confusion.
> 
> This seems to be due to the way Apple has done the “Extras” directory, and adding things there to sys.path.
> 
> See for example https://github.com/pypa/pip/issues/2468
> 
> If you can get rid of /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python in sys.path you should be all set.

Thanks, guys.

Indeed, these /System/Library dirs are in sys.path:

    In [1]: import sys
    In [2]: print [p for p in sys.path if 'System' in p]
    ['/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC’]

But I’m still confused: why is this problem only showing up now? Is the same setup that everyone has? Or is it just me for some reason? How and where would sys.path be set to this, and how and where should I change it? (Without disabling SIP, please!)

Andrew







>> On  13-Sep-2016, at 22:59 , Glyph Lefkowitz <gl...@tw...> wrote:
>> 
>> 
>>> On Sep 13, 2016, at 12:05 PM, Jack Jansen <Jac...@cw...> wrote:
>>> 
>>> I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.
>>> 
>>> Could it be that you’ve installed pyobjc a couple of OSX releases ago?
>> 
>> This is always worth checking ;).  Particularly if it was a few Setuptools releases ago.  Also worth checking: ~/Library/Python.
>> 
>>> And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?
>> 
>> SIP locks down /System, not /Library.
>> 
>>> A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.
>> 
>> This should be an _absolute_ last resort, though.  You should be able to clean out /Library just fine.  If you have a /Library/Frameworks/Python.framework, that's probably Python.org python, not system python.
>> 
>> -glyph
> 
> --
> Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
> If I can't dance I don't want to be part of your revolution -- Emma Goldman
> 
> 
> 

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Ronald O. <ron...@ma...>]

> On 12 Oct 2016, at 23:06, Nicholas Riley <nj...@il...> wrote:
> 
> Have you tried DevToolsSecurity?  Sounds like it might be a
> security/codesigning issue.
> 
> https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man8/DevToolsSecurity.8.html
> 

I hadn’t, but changing this doesn’t help.  Debugging on the console isn’t too bad, although I am stuck at the moment while debugging one of the crashes I get running the testsuite on 10.12.

The two programs below both call DCSGetTermRangeInString, both work on 10.11, but the python version crashes on 10.12. Sadly enough it crashes in a secondary thread that’s doing some work for the DictionaryServices framework. I’m not yet sure why the python script causes a crash of the interpreter.

# Python
from DictionaryServices import DCSGetTermRangeInString

text = "the hello world program"
rng = DCSGetTermRangeInString(None, text, 5)
print(rng)
#end

// Objective-C
#import <CoreServices/CoreServices.h>
#import <Foundation/Foundation.h>

int main(void)
{
   CFRange rng;

   rng = DCSGetTermRangeInString(NULL, CFSTR("the hello world program"), 5);
   NSLog(@"%ld %ld", rng.location, rng.length);
   return 0;
}
// end

Ronald

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Ronald O. <ron...@ma...>]

> On 13 Oct 2016, at 12:47, Ronald Oussoren <ron...@ma...> wrote:
> 
> 
>> On 12 Oct 2016, at 23:06, Nicholas Riley <nj...@il... <mailto:nj...@il...>> wrote:
>> 
>> Have you tried DevToolsSecurity?  Sounds like it might be a
>> security/codesigning issue.
>> 
>> https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man8/DevToolsSecurity.8.html <https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man8/DevToolsSecurity.8.html>
>> 
> 
> I hadn’t, but changing this doesn’t help.  Debugging on the console isn’t too bad, although I am stuck at the moment while debugging one of the crashes I get running the testsuite on 10.12.
> 
> The two programs below both call DCSGetTermRangeInString, both work on 10.11, but the python version crashes on 10.12. Sadly enough it crashes in a secondary thread that’s doing some work for the DictionaryServices framework. I’m not yet sure why the python script causes a crash of the interpreter.
> 
> # Python
> from DictionaryServices import DCSGetTermRangeInString
> 
> text = "the hello world program"
> rng = DCSGetTermRangeInString(None, text, 5)
> print(rng)
> #end
> 
> // Objective-C
> #import <CoreServices/CoreServices.h>
> #import <Foundation/Foundation.h>
> 
> int main(void)
> {
>    CFRange rng;
> 
>    rng = DCSGetTermRangeInString(NULL, CFSTR("the hello world program"), 5);
>    NSLog(@"%ld %ld", rng.location, rng.length);
>    return 0;
> }
> // end


I can now reproduce the problem using a small extension that doesn’t even touch PyObjC. See <https://bitbucket.org/ronaldoussoren/pyobjc/issues/174/crash-using-dictionaryservices-on-1012>. There is some material difference between /usr/bin/python and a python.org binary installation of Python 3.5.  This demonstrates the same crash using small C extension that exports a function that calls DCSGetTermInRange.

The crash is in a background thread that appears to do work for DCSGetTermInRange.

Next steps:
* Try to reproduce using python.org installer for 2.7
* Try to reproduce using a local build of 3.5.2

Now where did I leave my Yak shaving kit?

Ronald

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Ronald O. <ron...@ma...>]

> On 24 Oct 2016, at 19:37, Ronald Oussoren <ron...@ma...> wrote:
> […]
> I can now reproduce the problem using a small extension that doesn’t even touch PyObjC. See <https://bitbucket.org/ronaldoussoren/pyobjc/issues/174/crash-using-dictionaryservices-on-1012 <https://bitbucket.org/ronaldoussoren/pyobjc/issues/174/crash-using-dictionaryservices-on-1012>>. There is some material difference between /usr/bin/python and a python.org <http://python.org/> binary installation of Python 3.5.  This demonstrates the same crash using small C extension that exports a function that calls DCSGetTermInRange.
> 
> The crash is in a background thread that appears to do work for DCSGetTermInRange.
> 
> Next steps:
> * Try to reproduce using python.org <http://python.org/> installer for 2.7
> * Try to reproduce using a local build of 3.5.2
> 
> Now where did I leave my Yak shaving kit?

The 2.7 framework build on python.org causes the same crash, a local unix build of 3.5.2 doesn’t. As this isn’t a PyObjC issue I won’t investigate this until after the next release.

The good news w.r.t. that release: I just finished my first pass through the entire testsuite on 10.12 and fixed all issues w.r.t. metadata. The next step toward a release is to perform testing on older OSX release to ensure I didn’t introduce issues there.

Ronald

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Jack J. <jac...@cw...>]

It’s hardcoded in the Python executable, I’m afraid:-(

Just tried “python -s -S -v”, and the Extras/lib/python is still in sys.path.

That wasn’t a very smart move by the Apple engineers, I guess….

What you could do (but this is getting rather hacky) is create a /Library/Python/2.7/site-packages/removeSystemExtras.pth where you import sys and manually remove the Extras entries. Be careful, putting code (as opposed to pathnames) into a .pth file requires the line to start with “import “.

Jack

> On  13-Sep-2016, at 23:37 , Andrew Jaffe <a.h...@gm...> wrote:
> 
> Hi,
> 
> 
>> On 13 Sep 2016, at 22:26, Jack Jansen <jac...@cw...> wrote:
>> 
>> You’re absolutely right (both on SIP and on /Library/Frameworks/Python.framework probably being a python.org install), sorry for the confusion.
>> 
>> This seems to be due to the way Apple has done the “Extras” directory, and adding things there to sys.path.
>> 
>> See for example https://github.com/pypa/pip/issues/2468
>> 
>> If you can get rid of /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python in sys.path you should be all set.
> 
> Thanks, guys.
> 
> Indeed, these /System/Library dirs are in sys.path:
> 
>    In [1]: import sys
>    In [2]: print [p for p in sys.path if 'System' in p]
>    ['/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC’]
> 
> But I’m still confused: why is this problem only showing up now? Is the same setup that everyone has? Or is it just me for some reason? How and where would sys.path be set to this, and how and where should I change it? (Without disabling SIP, please!)
> 
> Andrew
> 
> 
> 
> 
> 
> 
> 
>>> On  13-Sep-2016, at 22:59 , Glyph Lefkowitz <gl...@tw...> wrote:
>>> 
>>> 
>>>> On Sep 13, 2016, at 12:05 PM, Jack Jansen <Jac...@cw...> wrote:
>>>> 
>>>> I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.
>>>> 
>>>> Could it be that you’ve installed pyobjc a couple of OSX releases ago?
>>> 
>>> This is always worth checking ;).  Particularly if it was a few Setuptools releases ago.  Also worth checking: ~/Library/Python.
>>> 
>>>> And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?
>>> 
>>> SIP locks down /System, not /Library.
>>> 
>>>> A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.
>>> 
>>> This should be an _absolute_ last resort, though.  You should be able to clean out /Library just fine.  If you have a /Library/Frameworks/Python.framework, that's probably Python.org python, not system python.
>>> 
>>> -glyph
>> 
>> --
>> Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
>> If I can't dance I don't want to be part of your revolution -- Emma Goldman
>> 
>> 
>> 
> 

--
Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

OK, I’m still being dense:

> On 13 Sep 2016, at 22:56, Jack Jansen <jac...@cw...> wrote:
> 
> It’s hardcoded in the Python executable, I’m afraid:-(
> 
> Just tried “python -s -S -v”, and the Extras/lib/python is still in sys.path.
> 
> That wasn’t a very smart move by the Apple engineers, I guess….

But this is the framework (non-apple!) build!…

And, again: why isn’t everyone seeing this all the time? (And why didn’t I see it before?)

> What you could do (but this is getting rather hacky) is create a /Library/Python/2.7/site-packages/removeSystemExtras.pth where you import sys and manually remove the Extras entries. Be careful, putting code (as opposed to pathnames) into a .pth file requires the line to start with “import “.
> 
> Jack
> 
>> On  13-Sep-2016, at 23:37 , Andrew Jaffe <a.h...@gm...> wrote:
>> 
>> Hi,
>> 
>> 
>>> On 13 Sep 2016, at 22:26, Jack Jansen <jac...@cw...> wrote:
>>> 
>>> You’re absolutely right (both on SIP and on /Library/Frameworks/Python.framework probably being a python.org install), sorry for the confusion.
>>> 
>>> This seems to be due to the way Apple has done the “Extras” directory, and adding things there to sys.path.
>>> 
>>> See for example https://github.com/pypa/pip/issues/2468
>>> 
>>> If you can get rid of /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python in sys.path you should be all set.
>> 
>> Thanks, guys.
>> 
>> Indeed, these /System/Library dirs are in sys.path:
>> 
>>   In [1]: import sys
>>   In [2]: print [p for p in sys.path if 'System' in p]
>>   ['/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC’]
>> 
>> But I’m still confused: why is this problem only showing up now? Is the same setup that everyone has? Or is it just me for some reason? How and where would sys.path be set to this, and how and where should I change it? (Without disabling SIP, please!)
>> 
>> Andrew
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>>> On  13-Sep-2016, at 22:59 , Glyph Lefkowitz <gl...@tw...> wrote:
>>>> 
>>>> 
>>>>> On Sep 13, 2016, at 12:05 PM, Jack Jansen <Jac...@cw...> wrote:
>>>>> 
>>>>> I think /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages is a very old location for storing Python packages. Recently things have been installed in /Library/Python/2.7/site-packages.
>>>>> 
>>>>> Could it be that you’ve installed pyobjc a couple of OSX releases ago?
>>>> 
>>>> This is always worth checking ;).  Particularly if it was a few Setuptools releases ago.  Also worth checking: ~/Library/Python.
>>>> 
>>>>> And could it be that the OSX upgrade that introduced SIP somehow didn’t clean out user-installed things from /Library/Frameworks before turning off write permission?
>>>> 
>>>> SIP locks down /System, not /Library.
>>>> 
>>>>> A possible workaround is to turn off SIP (or boot from the recovery partition), record what is in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages and then clean it out. Then after a reboot re-install the packages you’re still using.
>>>> 
>>>> This should be an _absolute_ last resort, though.  You should be able to clean out /Library just fine.  If you have a /Library/Frameworks/Python.framework, that's probably Python.org python, not system python.
>>>> 
>>>> -glyph
>>> 
>>> --
>>> Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
>>> If I can't dance I don't want to be part of your revolution -- Emma Goldman
>>> 
>>> 
>>> 
>> 
> 
> --
> Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
> If I can't dance I don't want to be part of your revolution -- Emma Goldman
> 
> 
> 

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 13, 2016, at 3:05 PM, Andrew Jaffe <a.h...@gm...> wrote:
> 
> But this is the framework (non-apple!) build!…

"framework build" refers to the way that Python is built.  Apple's python, Python.org's python, and Homebrew's python are all framework builds.  So, to be clear: this is python.org python?

> And, again: why isn’t everyone seeing this all the time? (And why didn’t I see it before?)

There's probably a .pth file somewhere that is stuffing Extras on your sys.path.  easy_install (especially older versions) is notorious for creating such things in ways that are hard to inspect for.  My usual recommendation at this point is to blow away _everything_ in /Library/Python, ~/Library/Python, ~/.local/lib/python*, and /Library/Frameworks/Python.framework and then reinstall from scratch.  (Thanks to SIP you don't need to blow away anything in /System anymore; hooray!)

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

Hi,


> On 13 Sep 2016, at 23:28, Glyph Lefkowitz <gl...@tw...> wrote:
> 
> 
>> On Sep 13, 2016, at 3:05 PM, Andrew Jaffe <a.h...@gm...> wrote:
>> 
>> But this is the framework (non-apple!) build!…
> 
> "framework build" refers to the way that Python is built.  Apple's python, Python.org's python, and Homebrew's python are all framework builds.  So, to be clear: this is python.org python?
> 

Doh! Yes, sorry, I think I said this way back in the thread: python.org 2.7.12 (problem doesn’t arise with 3.5.2, also installed). But see this...

>> And, again: why isn’t everyone seeing this all the time? (And why didn’t I see it before?)
> 
> There's probably a .pth file somewhere that is stuffing Extras on your sys.path.  easy_install (especially older versions) is notorious for creating such things in ways that are hard to inspect for.  My usual recommendation at this point is to blow away _everything_ in /Library/Python, ~/Library/Python, ~/.local/lib/python*, and /Library/Frameworks/Python.framework and then reinstall from scratch.  (Thanks to SIP you don't need to blow away anything in /System anymore; hooray!)

Aha!

  $ ls -lt /Library/Python/2.7/site-packages/
  total 0
  -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
  -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
  $ more /Library/Python/2.7/site-packages/Extras.pth 
  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC

Now I wonder how those got there?!

A




> 
> -glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...> wrote:
> 
> Aha!
> 
>  $ ls -lt /Library/Python/2.7/site-packages/
>  total 0
>  -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>  -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>  $ more /Library/Python/2.7/site-packages/Extras.pth 
>  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
> 
> Now I wonder how those got there?!
> 

Hah!  Thanks for sharing.  Very satisfying to actually make a *correct* prediction about setuptools' behavior :)

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Ned D. <na...@py...>]

On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>> <mailto:a.h...@gm...>> wrote:
>>
>> Aha!
>>
>>  $ ls -lt /Library/Python/2.7/site-packages/
>>  total 0
>>  -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>  -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>  $ more /Library/Python/2.7/site-packages/Extras.pth 
>>  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>  /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>
>> Now I wonder how those got there?!
>>
> 
> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
> prediction about setuptools' behavior :)

This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
Extras.pth file in /Library/Python/2.7; it's something new.  And,
unfortunately, due to https://bugs.python.org/issue4865, the
site-packages directory for the system Python 2.7 is included in
sys.path along with the non-system framework Python site-packages.

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
> 
> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>>> <mailto:a.h...@gm...>> wrote:
>>> 
>>> Aha!
>>> 
>>> $ ls -lt /Library/Python/2.7/site-packages/
>>> total 0
>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>> $ more /Library/Python/2.7/site-packages/Extras.pth 
>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>> 
>>> Now I wonder how those got there?!
>>> 
>> 
>> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
>> prediction about setuptools' behavior :)
> 
> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
> Extras.pth file in /Library/Python/2.7; it's something new.  And,
> unfortunately, due to https://bugs.python.org/issue4865, the
> site-packages directory for the system Python 2.7 is included in
> sys.path along with the non-system framework Python site-packages.

Hrm.  I guess everyone I knew on the beta was using homebrew python :(.

I'm surprised that Apple is putting stuff in /Library.  I don't have a Sierra box handy - /Library isn't SIP-protected now, is it?

This seems wrong; someone should file a radar (and probably share on http://www.openradar.me for further discussion).

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

On 17/09/2016 18:59, Glyph Lefkowitz wrote:
>
>> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
>>
>> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>>>> <mailto:a.h...@gm...>> wrote:
>>>>
>>>> Aha!
>>>>
>>>> $ ls -lt /Library/Python/2.7/site-packages/
>>>> total 0
>>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>>> $ more /Library/Python/2.7/site-packages/Extras.pth
>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>>>
>>>> Now I wonder how those got there?!
>>>>
>>>
>>> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
>>> prediction about setuptools' behavior :)
>>
>> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
>> Extras.pth file in /Library/Python/2.7; it's something new.  And,
>> unfortunately, due to https://bugs.python.org/issue4865, the
>> site-packages directory for the system Python 2.7 is included in
>> sys.path along with the non-system framework Python site-packages.
>
> Hrm.  I guess everyone I knew on the beta was using homebrew python :(.
>
> I'm surprised that Apple is putting stuff in /Library.  I don't have a Sierra box handy - /Library isn't SIP-protected now, is it?

Nope, that's not a problem.
>
> This seems wrong; someone should file a radar (and probably share on http://www.openradar.me for further discussion).

In the meantime, what's the recommended workaround?

A

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

On 17/09/2016 18:59, Glyph Lefkowitz wrote:
>
>> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
>>
>> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>>>> <mailto:a.h...@gm...>> wrote:
>>>>
>>>> Aha!
>>>>
>>>> $ ls -lt /Library/Python/2.7/site-packages/
>>>> total 0
>>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>>> $ more /Library/Python/2.7/site-packages/Extras.pth
>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>>>
>>>> Now I wonder how those got there?!
>>>>
>>>
>>> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
>>> prediction about setuptools' behavior :)
>>
>> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
>> Extras.pth file in /Library/Python/2.7; it's something new.  And,
>> unfortunately, due to https://bugs.python.org/issue4865, the
>> site-packages directory for the system Python 2.7 is included in
>> sys.path along with the non-system framework Python site-packages.
>

 >

So, a little more data:

If you rename or remove /Library/Python/2.7/site-packages/Extras.pth 
then pip2 works. *However*, lots of other stuff breaks -- anything that 
uses Apple's python and relies on access to pyobjc and the frameworks 
(e.g., TextMate's latex package).

What I don't understand is: what changed from Yosemite? This file did 
not exist before Sierra, but there were no problems with (Apple) python 
accessing these packages.

(Or is there something unique in my setup that is causing this? I kind 
of doubt it, but it's possible...)

Does anyone have any insight?

Thanks,

Andrew

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 20, 2016, at 10:29 AM, Andrew Jaffe <a.h...@gm...> wrote:
> 
> On 17/09/2016 18:59, Glyph Lefkowitz wrote:
>> 
>>> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
>>> 
>>> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>>>>> <mailto:a.h...@gm...>> wrote:
>>>>> 
>>>>> Aha!
>>>>> 
>>>>> $ ls -lt /Library/Python/2.7/site-packages/
>>>>> total 0
>>>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>>>> $ more /Library/Python/2.7/site-packages/Extras.pth
>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>>>> 
>>>>> Now I wonder how those got there?!
>>>>> 
>>>> 
>>>> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
>>>> prediction about setuptools' behavior :)
>>> 
>>> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
>>> Extras.pth file in /Library/Python/2.7; it's something new.  And,
>>> unfortunately, due to https://bugs.python.org/issue4865, the
>>> site-packages directory for the system Python 2.7 is included in
>>> sys.path along with the non-system framework Python site-packages.
>> 
>> Hrm.  I guess everyone I knew on the beta was using homebrew python :(.
>> 
>> I'm surprised that Apple is putting stuff in /Library.  I don't have a Sierra box handy - /Library isn't SIP-protected now, is it?
> 
> Nope, that's not a problem.
>> 
>> This seems wrong; someone should file a radar (and probably share on http://www.openradar.me <http://www.openradar.me/> for further discussion).
> 
> In the meantime, what's the recommended workaround?

Looking back over the thread, my first reply was: "Make a virtualenv and install pyobjc there", but I didn't see a direct response to that.  That would still be my first choice for a workaround.  Is there some reason that doesn't work for you?

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Andrew J. <a.h...@gm...>]

On 20/09/2016 20:54, Glyph Lefkowitz wrote:
 >> On Sep 20, 2016, at 10:29 AM, Andrew Jaffe <a.h...@gm...> wrote:
 >>
 >> On 17/09/2016 18:59, Glyph Lefkowitz wrote:
 >>>
 >>>> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
 >>>>
 >>>> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
 >>>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...> 
wrote:
 >>>>>>
 >>>>>> Aha!
 >>>>>>
 >>>>>> $ ls -lt /Library/Python/2.7/site-packages/
 >>>>>> total 0
 >>>>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
 >>>>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
 >>>>>> $ more /Library/Python/2.7/site-packages/Extras.pth
 >>>>>> 
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
 >>>>>> 
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
 >>>>>>
 >>>>> Hah!  Thanks for sharing.  Very satisfying to actually make a 
*correct*
 >>>>> prediction about setuptools' behavior :)
 >>>>
 >>>> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
 >>>> Extras.pth file in /Library/Python/2.7; it's something new.  And,
 >>>> unfortunately, due to https://bugs.python.org/issue4865, the
 >>>> site-packages directory for the system Python 2.7 is included in
 >>>> sys.path along with the non-system framework Python site-packages.
 >>>
 >>> This seems wrong; someone should file a radar (and probably share
 >>> on http://www.openradar.me <http://www.openradar.me/> for further
 >>> discussion).
 >>
 >> In the meantime, what's the recommended workaround?
 >
 > Looking back over the thread, my first reply was: "Make a virtualenv and
 > install pyobjc there", but I didn't see a direct response to that.  That
 > would still be my first choice for a workaround.  Is there some reason
 > that doesn't work for you?

That would work, and in fact I don't really need PyObjC (sorry, Ronald!) 
but I've got my whole setup working with the "global" python.org 
framework build, so I am used to that... and the Sierra status quo does 
seem ugly (and quite possibly is a bug!).

Andrew

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Sep 21, 2016, at 01:52, Andrew Jaffe <a.h...@gm...> wrote:
> 
> That would work, and in fact I don't really need PyObjC (sorry, Ronald!) but I've got my whole setup working with the "global" python.org <http://python.org/> framework build, so I am used to that... and the Sierra status quo does seem ugly (and quite possibly is a bug!).

Well, Apple ships certain libraries, and they want them to be on your path.

The nice thing about virtualenv is that, along with pip wheel caching, re-creating them is really fast, so you don't have to just do one setup; you just make a requirements.txt for each separate environment you want, and regularly re-create it, so you know that if you give directions to anyone else, your setup will work for them as well.

One bad habit we get into is getting a full-on environment set up on our own workstation, and then not knowing exactly what we installed, and therefore what we need in order for someone else to build and work on our software.  Virtualenv helps reduce that problem a bit.

-glyph

Re: [Pyobjc-dev] [Pythonmac-SIG] PyObjC and macOS 10.12 (Sierra)

[Glyph L. <gl...@tw...>]

> On Oct 6, 2016, at 2:56 AM, Andrew Jaffe <a.h...@gm...> wrote:
> 
> On 17/09/2016 18:59, Glyph Lefkowitz wrote:
>> 
>>> On Sep 17, 2016, at 9:27 AM, Ned Deily <na...@py...> wrote:
>>> 
>>> On 2016-09-13 19:33, Glyph Lefkowitz wrote:
>>>>> On Sep 13, 2016, at 3:35 PM, Andrew Jaffe <a.h...@gm...
>>>>> <mailto:a.h...@gm...>> wrote:
>>>>> 
>>>>> Aha!
>>>>> 
>>>>> $ ls -lt /Library/Python/2.7/site-packages/
>>>>> total 0
>>>>> -rwxr-xr-x  1 root  wheel  157 31 Jul 02:36 Extras.pth*
>>>>> -rw-r--r--  1 root  wheel  119 31 Jul 02:36 README
>>>>> $ more /Library/Python/2.7/site-packages/Extras.pth
>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
>>>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC
>>>>> 
>>>>> Now I wonder how those got there?!
>>>>> 
>>>> 
>>>> Hah!  Thanks for sharing.  Very satisfying to actually make a *correct*
>>>> prediction about setuptools' behavior :)
>>> 
>>> This seems to be Apple's doing.  AFAICT, 10.12 is shipping with this
>>> Extras.pth file in /Library/Python/2.7; it's something new.  And,
>>> unfortunately, due to https://bugs.python.org/issue4865, the
>>> site-packages directory for the system Python 2.7 is included in
>>> sys.path along with the non-system framework Python site-packages.
>> 
> 
> >
> 
> So, a little more data:
> 
> If you rename or remove /Library/Python/2.7/site-packages/Extras.pth then pip2 works.

What do you mean by "works"?  Your original error is pip refusing to upgrade pyObjC because to upgrade pyObjC it has to delete the old version, and pyObjC is part of the operating system, and it can't delete the installed version.  This is correct; the error reporting could be nicer, but pip is not broken.  Don't mess with files in /System.

The suggestion to use virtualenv isn't really optional.  If you really, really want things to be importable by a bare 'python', not inside a venv, `pip install --user` is another option you might have.

> *However*, lots of other stuff breaks -- anything that uses Apple's python and relies on access to pyobjc and the frameworks (e.g., TextMate's latex package).

Yep, that's expected behavior.  This is exactly why Apple is making it increasingly difficult to screw up /System.  Apps can, should, and do rely upon the libraries installed on the system.

> What I don't understand is: what changed from Yosemite? This file did not exist before Sierra, but there were no problems with (Apple) python accessing these packages.

Do you mean from El Capitan?

This file previously existed in a different location, and (while the particulars stubbornly refuse to stick to memory for me, for some reason) this new location is the one generally preferred by the python packaging maintainers.

> (Or is there something unique in my setup that is causing this? I kind of doubt it, but it's possible...)
> 
> Does anyone have any insight?

If you really, really, really want to do this in such a way that /System stuff is only present for /System's python and not for python.org's, you can take advantage of the 'import' hack <https://docs.python.org/2/library/site.html>, and rewrite /Library/Python/2.7/site-packages/Extras.pth to _conditionally_ add those entries to sys.path, checking sys.executable or some other fingerprint.

But: don't.  Given that system python and python.org python share /Library and ~/Library, they're not really distinct environments, and things installed into one will show up in the other, so excluding the /System directories on one but not the other will just cause other, more confusing issues.

In conclusion: just use virtualenv.  This is not a problem that should be fixed.

-glyph