Thread: [Pyobjc-dev] ctypes MemoryError when attempting to launch notarized app
Brought to you by:
ronaldoussoren
Menu
▾
▴
pyobjc-dev
|
From: Glyph <gl...@tw...> - 2018-10-28 05:53:14
|
I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get:
Traceback (most recent call last):
File "my.app/Contents/Resources/__boot__.py", line 93, in <module>
_setup_ctypes()
File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes
from ctypes.macholib import dyld
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "ctypes/__init__.pyc", line 538, in <module>
File "ctypes/__init__.pyc", line 273, in _reset_cache
(If anyone wants to follow along in the traceback, this is using python.org <http://python.org/> 3.6.6.)
This happens before any of my code even runs, so I can't just try to avoid ctypes.
Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 <https://forum.kodi.tv/showthread.php?tid=329171>, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself.
Does anyone have experience with this, or ideas about what to do?
-glyph |
|
From: Ronald O. <ron...@ma...> - 2018-10-28 09:27:21
|
> On 28 Oct 2018, at 06:27, Glyph <gl...@tw...> wrote: > > I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get: > > Traceback (most recent call last): > File "my.app/Contents/Resources/__boot__.py", line 93, in <module> > _setup_ctypes() > File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes > from ctypes.macholib import dyld > File "<frozen importlib._bootstrap>", line 971, in _find_and_load > File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked > File "<frozen importlib._bootstrap>", line 656, in _load_unlocked > File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible > File "ctypes/__init__.pyc", line 538, in <module> > File "ctypes/__init__.pyc", line 273, in _reset_cache > > (If anyone wants to follow along in the traceback, this is using python.org 3.6.6.) On what version of macOS? I expect 10.14 because that’s the only release that actually knows about notarization, but enabling this feature might also affect how the app is signed. > > This happens before any of my code even runs, so I can't just try to avoid ctypes. You could patch the __boot__.py file before signing to see if that helps. Although this should cause problems further on, the call to _setup_ctypes should only be created when some code in your app bundle has a dependency on ctypes. > > Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. > > Does anyone have experience with this, or ideas about what to do? I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. With some luck there’s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4” for the application? Both with and without notarisation? Ronald > > -glyph > _______________________________________________ > Pyobjc-dev mailing list > Pyo...@li... > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev |
|
From: Glyph <gl...@tw...> - 2018-10-28 18:21:01
|
> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <ron...@ma...> wrote: > > > >> On 28 Oct 2018, at 06:27, Glyph <gl...@tw...> wrote: >> >> I adjusted my code-signing to use the new, stricter requirements imposed by app notarization. I managed to get it successfully notarized, but the app is now non-functional as a result: at startup, I get: >> >> Traceback (most recent call last): >> File "my.app/Contents/Resources/__boot__.py", line 93, in <module> >> _setup_ctypes() >> File "my.app/Contents/Resources/__boot__.py", line 86, in _setup_ctypes >> from ctypes.macholib import dyld >> File "<frozen importlib._bootstrap>", line 971, in _find_and_load >> File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked >> File "<frozen importlib._bootstrap>", line 656, in _load_unlocked >> File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible >> File "ctypes/__init__.pyc", line 538, in <module> >> File "ctypes/__init__.pyc", line 273, in _reset_cache >> >> (If anyone wants to follow along in the traceback, this is using python.org 3.6.6.) > > On what version of macOS? I expect 10.14 because that’s the only release that actually knows about notarization, but enabling this feature might also affect how the app is signed. ProductName: Mac OS X ProductVersion: 10.14 BuildVersion: 18A391 >> This happens before any of my code even runs, so I can't just try to avoid ctypes. > > You could patch the __boot__.py file before signing to see if that helps. Although this should cause problems further on, the call to _setup_ctypes should only be created when some code in your app bundle has a dependency on ctypes. I'll give that a shot. >> >> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 <https://forum.kodi.tv/showthread.php?tid=329171>, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >> >> Does anyone have experience with this, or ideas about what to do? > > I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. As I see it, there's 2 problems here: py2app's __boot__.py should fail more gracefully if initializing ctypes doesn't work, since not everybody needs ctypes. Shall I file this on the tracker? ctypes itself should address whatever eldritch hideousness is causing this; in addition to the windows security layer stuff I found, grsecurity TPE causes the same traceback: https://bugs.python.org/issue28429 > With some luck there’s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4” for the application? Both with and without notarisation? Sorry, my original message was not clear. App notarization itself is not the problem, it's the "stricter requirements" that I ambiguously referenced. The requirement in question is the '--options runtime' flag passed to 'codesign'. So you can just codesign an app (even with an ad-hoc identity, you technically could do this without even having a valid cert, although the way one generates one of those escapes me) with the 'runtime' option, you can reproduce this. So if I sign my app like this: #!/bin/bash find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' | while read libfile; do codesign --sign "${IDENTITY}" \ --deep "${libfile}" \ --force \ --options runtime; done; codesign --sign "${IDENTITY}" \ --deep "${NAME}.app" \ --force \ --options runtime; and then run it as "./${NAME}.app/Contents/MacOS/${NAME}". I immediately get the traceback given above. -glyph |
> On Oct 28, 2018, at 11:20 AM, Glyph <gl...@tw...> wrote: > > > >> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <ron...@ma... <mailto:ron...@ma...>> wrote: > >>> >>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 <https://forum.kodi.tv/showthread.php?tid=329171>, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>> >>> Does anyone have experience with this, or ideas about what to do? >> >> I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. > > The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. > > The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. Apparently my app doesn't actually need ctypes. The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) -glyph |
|
From: Ronald O. <ron...@ma...> - 2018-10-28 20:48:25
|
> On 28 Oct 2018, at 19:47, Glyph <gl...@tw...> wrote: > > > >> On Oct 28, 2018, at 11:20 AM, Glyph <gl...@tw... <mailto:gl...@tw...>> wrote: >> >> >> >>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <ron...@ma... <mailto:ron...@ma...>> wrote: >> >>>> >>>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 <https://forum.kodi.tv/showthread.php?tid=329171>, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>>> >>>> Does anyone have experience with this, or ideas about what to do? >>> >>> I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. >> >> The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. >> >> The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. > > On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. > > Apparently my app doesn't actually need ctypes. Good to hear that. > > The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. Hmm…. I wonder what’s the best way forward here. I could add on option to disable ctypes support, but that is a kludge. A weak importing hook (something like the never withdrawn PEP 369) could execute this code only when actually needed, but I have no idea how hard it would be to implement this. > > (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) Interesting… I haven’t had complaints about PyObjC yet, and that also uses libffi. I wonder what the “hardened runtime” option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn’t like (being writeable + executable memory). P.S. I just noticed that the traceback in your initial message doesn’t include the actual exception, just the traceback. Ronald |
|
From: Ronald O. <ron...@ma...> - 2018-10-28 20:55:44
|
> On 28 Oct 2018, at 19:20, Glyph <gl...@tw...> wrote: > > > >>> >>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171 <https://forum.kodi.tv/showthread.php?tid=329171>, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>> >>> Does anyone have experience with this, or ideas about what to do? >> >> I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. > > The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. Yup. That’s something that worries me as well, and not just for Python apps. Not being able to run your own code without paying Apple for the privilege is not something I look forward to. I’m still hoping that the option to run unsigned code doesn’t go away. > > The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. > > As I see it, there's 2 problems here: > > py2app's __boot__.py should fail more gracefully if initializing ctypes doesn't work, since not everybody needs ctypes. Shall I file this on the tracker? Yes please. I’d prefer a solution that doesn’t involve ignoring errors, but that’s probably the easiest fix for now. What’s the exception you’re getting? Tweaking the code in py2app/bootstrap/ctypes_setup.py to ignore that exception would be trivial. > ctypes itself should address whatever eldritch hideousness is causing this; in addition to the windows security layer stuff I found, grsecurity TPE causes the same traceback: https://bugs.python.org/issue28429 <https://bugs.python.org/issue28429> >> With some luck there’s some entitlement or code signing option that causes this problem. What is the output of "codesign --display --verbose=4” for the application? Both with and without notarisation? > > Sorry, my original message was not clear. App notarization itself is not the problem, it's the "stricter requirements" that I ambiguously referenced. The requirement in question is the '--options runtime' flag passed to 'codesign'. So you can just codesign an app (even with an ad-hoc identity, you technically could do this without even having a valid cert, although the way one generates one of those escapes me) with the 'runtime' option, you can reproduce this. > > So if I sign my app like this: > > #!/bin/bash > find "${NAME}.app" -iname '*.so' -or -iname '*.dylib' | > while read libfile; do > codesign --sign "${IDENTITY}" \ > --deep "${libfile}" \ > --force \ > --options runtime; > done; > > codesign --sign "${IDENTITY}" \ > --deep "${NAME}.app" \ > --force \ > --options runtime; > > and then run it as "./${NAME}.app/Contents/MacOS/${NAME}". I immediately get the traceback given above. Great. That should make it easier for me to reproduce the issue. Ronald |
> On Oct 28, 2018, at 1:48 PM, Ronald Oussoren <ron...@ma...> wrote: > > > >> On 28 Oct 2018, at 19:47, Glyph <gl...@tw...> wrote: >> >> >> >>> On Oct 28, 2018, at 11:20 AM, Glyph <gl...@tw...> wrote: >>> >>> >>> >>>> On Oct 28, 2018, at 2:27 AM, Ronald Oussoren <ron...@ma...> wrote: >>> >>>>> >>>>> Curiously, this is the same traceback that comes from https://forum.kodi.tv/showthread.php?tid=329171, which suggests it's something fundamental to strict shared-library sandboxing that ctypes trips over when trying to initialize itself. >>>>> >>>>> Does anyone have experience with this, or ideas about what to do? >>>> >>>> I’m afraid not. I currently get away with not signing apps at all, although properly supporting signing is on my way too long wish list for py2app. >>> >>> The ability to distribute unsigned apps is not-so-slowly going away; even the ability to distribute non-notarized apps has a very limited shelf-life at this point. So this ought to be an alarming development for everyone - having Python apps effectively banned from macOS distribution is a big potential problem :-\. >>> >>> The good news here is that aside from having to write a little for loop in shell (shown below) getting the app codesigned previously was easy, and my app *did* pass notarization, so nothing that py2app is doing is breaking things on apple's end. It's just a matter of a ctypes bug. >> >> On that note: more good news. While I haven't round-tripped through notarization again yet, this is a bit less dire than it first appeared. If I prevent the import of ctypes with an `import sys; sys.modules['ctypes'] = None`, and add a 'sed' script to my build process to prevent _setup_ctypes from running in __boot__, then the app launches again. >> >> Apparently my app doesn't actually need ctypes. > > Good to hear that. > >> >> The problem seems to be that Twisted includes a ctypes import; modulegraph sees this and thinks there is a hard dependency, and inserts the ctypes setup blob into __boot__. However, this is a conditional import, and it's for Windows support anyway. > > Hmm…. I wonder what’s the best way forward here. I could add on option to disable ctypes support, but that is a kludge. A weak importing hook (something like the never withdrawn PEP 369) could execute this code only when actually needed, but I have no idea how hard it would be to implement this. > > >> >> (There also seem to be problems with cffi-using libraries, but not other shared objects, so maybe this is a bug in libffi; however, these don't interfere with py2app itself starting up.) > > Interesting… I haven’t had complaints about PyObjC yet, and that also uses libffi. > > I wonder what the “hardened runtime” option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn’t like (being writeable + executable memory). Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory then? Any chance you know how to jam that into a `codesign` command line somehow? :-) > P.S. I just noticed that the traceback in your initial message doesn’t include the actual exception, just the traceback. Oh; it’s “MemoryError”, no exception message. > Ronald > |
|
From: Glyph <gl...@tw...> - 2018-10-28 23:57:05
|
> On Oct 28, 2018, at 2:57 PM, Glyph <gl...@tw...> wrote: > >> I wonder what the “hardened runtime” option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn’t like (being writeable + executable memory). > > Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory <https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory> then? Any chance you know how to jam that into a `codesign` command line somehow? :-) > Thank you so much for this tip, Ronald! This was much easier than I anticipated, and things are working now! The relevant entitlements file is literally just: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>com.apple.security.cs.allow-unsigned-executable-memory</key> <true/> </dict> </plist> I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to my codesign invocations, removed all my workarounds for ctypes et. al. (except for the hard-coded 'import _cffi_backend' still necessary to convince modulegraph to include enough code for SSL to work), and then tried launching my app. Success! Then I tried notarizing it: also success! Time permitting, I'll be updating my blog post at https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html <https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html> with this information, and possibly publishing the now unfortunately somewhat complex tooling I use to do signing now. So I don't know if I'm the first to do this, but looking at the archives for these lists I seem to be the first to report it: you can successfully codesign and notarize apps created with py2app and python 3.6! It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm guessing?) libffi needs to be using it for the memory it places synthetic closures into, so that this entitlement won't be necessary with some future version of Python. But it looks like Apple is not pushing particularly hard to deprecate this one right now, thank goodness :-). -glyph |
|
From: Ronald O. <ron...@ma...> - 2018-10-29 06:58:18
|
> On 29 Oct 2018, at 00:56, Glyph <gl...@tw...> wrote: > > > >> On Oct 28, 2018, at 2:57 PM, Glyph <gl...@tw... <mailto:gl...@tw...>> wrote: >> >>> I wonder what the “hardened runtime” option actually does and enforces. In 3.7 the line in ctypes/__init__.py that causes the exception is a call that creates a dummy C function, and likely triggers the first allocation for storing a libffi closure which could be something the hardened runtime doesn’t like (being writeable + executable memory). >> >> Interesting. Perhaps what I want is simply https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory <https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory> then? Any chance you know how to jam that into a `codesign` command line somehow? :-) >> > > Thank you so much for this tip, Ronald! This was much easier than I anticipated, and things are working now! Great. > > The relevant entitlements file is literally just: > > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd <http://www.apple.com/DTDs/PropertyList-1.0.dtd>"> > <plist version="1.0"> > <dict> > <key>com.apple.security.cs.allow-unsigned-executable-memory</key> > <true/> > </dict> > </plist> > > I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to my codesign invocations, removed all my workarounds for ctypes et. al. (except for the hard-coded 'import _cffi_backend' still necessary to convince modulegraph to include enough code for SSL to work), and then tried launching my app. Which package needs _cffi_backend? I can add a recipe for that to py2app to do this automagically. > Success! Then I tried notarizing it: also success! Time permitting, I'll be updating my blog post at https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html <https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html> with this information, and possibly publishing the now unfortunately somewhat complex tooling I use to do signing now. > > So I don't know if I'm the first to do this, but looking at the archives for these lists I seem to be the first to report it: you can successfully codesign and notarize apps created with py2app and python 3.6! > > It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm guessing?) libffi needs to be using it for the memory it places synthetic closures into, so that this entitlement won't be necessary with some future version of Python. But it looks like Apple is not pushing particularly hard to deprecate this one right now, thank goodness :-). MAP_JIT is a mmap flag that’s apparently introduced in 10.14. The slides at https://developer.apple.com/videos/play/wwdc2018/702/ <https://developer.apple.com/videos/play/wwdc2018/702/> mention this flag and the hardened runtime. I guess we should add this flag to the code in Modules/_ctypes/malloc_closure.c CPython) and in the similar code in PyObjC. The annoying bit is that the flag is new in 10.14, and CPython installers are created on 10.9 which means those won’t include the new flag for a long time. I’ll have to check if using MAP_JIT is ok when deploying on older macOS versions, or if the code should do a runtime version check. Ronald > > -glyph > _______________________________________________ > Pyobjc-dev mailing list > Pyo...@li... > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev |
|
From: Ronald O. <ron...@ma...> - 2018-10-29 08:14:16
|
> On 29 Oct 2018, at 07:58, Ronald Oussoren via Pyobjc-dev <pyo...@li...> wrote: > > MAP_JIT is a mmap flag that’s apparently introduced in 10.14. The slides at https://developer.apple.com/videos/play/wwdc2018/702/ <https://developer.apple.com/videos/play/wwdc2018/702/> mention this flag and the hardened runtime. > > I guess we should add this flag to the code in Modules/_ctypes/malloc_closure.c CPython) and in the similar code in PyObjC. The annoying bit is that the flag is new in 10.14, and CPython installers are created on 10.9 which means those won’t include the new flag for a long time. > > I’ll have to check if using MAP_JIT is ok when deploying on older macOS versions, or if the code should do a runtime version check. I filed an issue with PyObjC to ensure I don’t forget to look into this: https://bitbucket.org/ronaldoussoren/pyobjc/issues/253/use-map_fixed-on-macos-1014 <https://bitbucket.org/ronaldoussoren/pyobjc/issues/253/use-map_fixed-on-macos-1014>. I’ll look into ctypes when I have a good solution for PyObjC. Ronald |
|
From: Glyph <gl...@tw...> - 2018-10-29 17:22:18
|
On Sun, Oct 28, 2018, at 11:58 PM, Ronald Oussoren wrote: > > >> On 29 Oct 2018, at 00:56, Glyph <gl...@tw...> wrote: >> >> >> >>> On Oct 28, 2018, at 2:57 PM, Glyph <gl...@tw...> wrote:>>> >>>> I wonder what the “hardened runtime” option actually does and >>>> enforces. In 3.7 the line in ctypes/__init__.py that causes the >>>> exception is a call that creates a dummy C function, and likely >>>> triggers the first allocation for storing a libffi closure which >>>> could be something the hardened runtime doesn’t like (being >>>> writeable + executable memory).>>> >>> Interesting. Perhaps what I want is simply >>> https://developer.apple.com/documentation/security/com_apple_security_cs_allow-unsigned-executable-memory >>> then? Any chance you know how to jam that into a `codesign` command >>> line somehow? :-)>>> >> >> Thank you so much for this tip, Ronald! This was much easier than I >> anticipated, and things are working now!> > Great. > >> >> The relevant entitlements file is literally just: >> >>> <?xml version="1.0" encoding="UTF-8"?> >>> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" " >>> http://www.apple.com/DTDs/PropertyList-1.0.dtd">>>> <plist version="1.0"> >>> <dict> >>> <key>com.apple.security.cs.allow-unsigned-executable-memory</key> >>> <true/> >>> </dict> >>> </plist> >> >> I dropped that in a file, added `--entitlements=$THAT_FILE.plist` to >> my codesign invocations, removed all my workarounds for ctypes et. >> al. (except for the hard-coded 'import _cffi_backend' still necessary >> to convince modulegraph to include enough code for SSL to work), and >> then tried launching my app.> > Which package needs _cffi_backend? I can add a recipe for that to > py2app to do this automagically. This may sound obvious, but: cffi :-). In my case, pyOpenSSL -> cryptography -> cffi. > >> Success! Then I tried notarizing it: also success! Time permitting, >> I'll be updating my blog post at >> https://glyph.twistedmatrix.com/2018/01/shipping-pygame-mac-app.html >> with this information, and possibly publishing the now unfortunately >> somewhat complex tooling I use to do signing now.>> >> So I don't know if I'm the first to *do* this, but looking at the >> archives for these lists I seem to be the first to *report* it: you >> can successfully codesign and notarize apps created with py2app and >> python 3.6!>> >> It seems to me that whatever "MAP_JIT" is (an mmap flag, I'm >> guessing?) libffi needs to be using it for the memory it places >> synthetic closures into, so that this entitlement won't be necessary >> with some future version of Python. But it looks like Apple is not >> pushing particularly hard to deprecate this one right now, thank >> goodness :-).> > MAP_JIT is a mmap flag that’s apparently introduced in 10.14. The > slides at https://developer.apple.com/videos/play/wwdc2018/702/ > mention this flag and the hardened runtime.> > I guess we should add this flag to the code in > Modules/_ctypes/malloc_closure.c CPython) and in the similar code in > PyObjC. The annoying bit is that the flag is new in 10.14, and > CPython installers are created on 10.9 which means those won’t include > the new flag for a long time.> > I’ll have to check if using MAP_JIT is ok when deploying on older > macOS versions, or if the code should do a runtime version check. I can't shed any light on this, but I suspect the cffi folks will also have to figure this out, and may already have some sense of how this works. I filed an issue with them here: https://bitbucket.org/cffi/cffi/issues/391/cffi-doesnt-work-inside-a-macos-app-bundle |
|
From: Ronald O. <ron...@ma...> - 2018-10-30 08:24:30
|
> On 29 Oct 2018, at 18:22, Glyph <gl...@tw...> wrote: >>> >> >> Which package needs _cffi_backend? I can add a recipe for that to py2app to do this automagically. > > This may sound obvious, but: cffi :-). In my case, pyOpenSSL -> cryptography -> cffi. I’ll look into adding a recipe for this to py2app. Ronald |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X