Thread: [Pyobjc-dev] Bridge Support Performances
Brought to you by:
ronaldoussoren
Menu
▾
▴
pyobjc-dev
|
From: Virgil D. <hs...@ha...> - 2010-01-20 11:58:34
|
Hi there, I recently started to use PyObjC 2.2 (after having use 1.4 for a long time), mainly for 64-bit support, and the biggest problem I have with it is bridge support's memory usage. One of my applications which used to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb when built with pyobjc 2.2. I have read that other thread about it and Ronald mentioned that this was because of bridge support code which he didn't have enough time to work on. I plan to try to tackle the issue soon and I'd appreciate your feedback, Ronald, so I can know early if I'm just on a fool's errand. As a stopgap measure, I plan to simply clone the Cocoa and Foundation packages and remove everything I don't use from the bridgeSupport xml files. My guess is that it will greatly enhance loading speed and memory usage since I only use a tiny fraction of Foundation and AppKit. Then, I noticed that Foundation and AppKit bridgesupport files are huge, not really because there's a lot of symbols in them (there are, however), but because the XML file format has so much redudancy in it (For example, funtion arguments elements require way too much crap. the types could be one string for 32 bit and one string for 64 bit instead of being this long list of <arg> element). What I'm thinking about is to convert those bridge support files to a more concise format, maybe in YAML or something. This wouldn't help memory usage, but it might help loading speed. Last, the most important part of my super plan, would be smart importers. I don't know much about them, but I keep seeing that Brett Canon keeps writing article about custom imports and stuff. Maybe that by using this, we could make PyObjC only load the requested elements. If, for example, someone would type "from AppKit import symbol1, symbol2" instead of "from AppKit import *", then only symbol1 and symbol2 would be loaded from the bridge support. So, what do you think? Is that plan realistic, will there be a show-stopper down the road? -- Virgil Dupras Hardcoded Software http://www.hardcoded.net |
|
From: Ronald O. <ron...@ma...> - 2010-01-20 21:45:42
Attachments:
smime.p7s
|
On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: > Hi there, > > I recently started to use PyObjC 2.2 (after having use 1.4 for a long > time), mainly for 64-bit support, and the biggest problem I have with > it is bridge support's memory usage. One of my applications which used > to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb > when built with pyobjc 2.2. Ouch. > > I have read that other thread about it and Ronald mentioned that this > was because of bridge support code which he didn't have enough time to > work on. I plan to try to tackle the issue soon and I'd appreciate > your feedback, Ronald, so I can know early if I'm just on a fool's > errand. I haven't had time yet to measure where memory is going, although I have done some micro-optimizations (such as making sure that all strings are interned). I'm currently working on a py3k port and some other changes, although much slower than I'd like. I haven't managed to spent more than on hour or two a week on PyObjC this year :-(. > > As a stopgap measure, I plan to simply clone the Cocoa and Foundation > packages and remove everything I don't use from the bridgeSupport xml > files. My guess is that it will greatly enhance loading speed and > memory usage since I only use a tiny fraction of Foundation and > AppKit. That would certainly help in the short term, but isn't a workable solution in the longer turn. > > Then, I noticed that Foundation and AppKit bridgesupport files are > huge, not really because there's a lot of symbols in them (there are, > however), but because the XML file format has so much redudancy in it > (For example, funtion arguments elements require way too much crap. > the types could be one string for 32 bit and one string for 64 bit > instead of being this long list of <arg> element). What I'm thinking > about is to convert those bridge support files to a more concise > format, maybe in YAML or something. This wouldn't help memory usage, > but it might help loading speed. I don't think that replacing the XML by an YAML file would help, unless there is a memory leak in the XML parsing code (and that should then be solved by fixing that memory leak). I am thinking of "compiling" the bridgesupport files into some other format though, such as a python or C file. That's mostly to get rid of the dependency on libxml, but could also help to reduce overhead (why parse an XML file when you can also load an .pyc file). Compiling into C would make it possible to create pre-constructed objects, at the cost of much greater coupling between pyobjc-core and the framework wrappers. That would only be useful when there is clear evidence that such a route could save a lot of memory and/or time. > > Last, the most important part of my super plan, would be smart > importers. I don't know much about them, but I keep seeing that Brett > Canon keeps writing article about custom imports and stuff. Maybe that > by using this, we could make PyObjC only load the requested elements. > If, for example, someone would type "from AppKit import symbol1, > symbol2" instead of "from AppKit import *", then only symbol1 and > symbol2 would be loaded from the bridge support. > > So, what do you think? Is that plan realistic, will there be a > show-stopper down the road? One problem with that is that the framework wrappers themself use 'from Foo import *' to ensure that imports behave simularly to ObjC includes. Changing that for most wrappers wouldn't be much of a problem, but it would be for a more complex one like the Quartz wrappers. We'll only know for sure once someone measures what's going on, but my gut feeling is that it is possible to reduce the amount of memory used by PyObjC and that it is also possible to reduce the amount of time that it takes to import a framework wrapper. BTW. One possible source of memory use is the proxy for NSString objects: for compatiblity with the rest of python that proxy is a subclass of Python's unicode type and therefore contains a copy of the content of the string (with 16-bits per character in the string regardless of how the string is represented in ObjC). BTW2. Measuring where memory has gone will be harder than I'd like due to Python's memory management routines: Python has its own malloc-like allocator which means that the Instruments won't be very helpfull to do the measurements. With some luck a Python build with '--without-pymalloc' still works... Ronald > -- > Virgil Dupras > Hardcoded Software > http://www.hardcoded.net > > ------------------------------------------------------------------------------ > Throughout its 18-year history, RSA Conference consistently attracts the > world's best and brightest in the field, creating opportunities for Conference > attendees to learn about information security's most important issues through > interactions with peers, luminaries and emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > Pyobjc-dev mailing list > Pyo...@li... > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev |
|
From: Leonardo S. <san...@gm...> - 2010-01-20 21:51:03
|
On Jan 20, 2010, at 7:45 PM, Ronald Oussoren wrote: > On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: > >> Hi there, >> >> I recently started to use PyObjC 2.2 (after having use 1.4 for a long >> time), mainly for 64-bit support, and the biggest problem I have with >> it is bridge support's memory usage. One of my applications which used >> to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb >> when built with pyobjc 2.2. > > Ouch. Can it be that a good part of this doubling in size is just from pointers going from 32 to 64bits? -- Leonardo Santagada santagada at gmail.com |
|
From: Virgil D. <hs...@ha...> - 2010-01-20 21:52:10
|
On Wed, Jan 20, 2010 at 10:50 PM, Leonardo Santagada <san...@gm...> wrote: > > On Jan 20, 2010, at 7:45 PM, Ronald Oussoren wrote: > >> On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: >> >>> Hi there, >>> >>> I recently started to use PyObjC 2.2 (after having use 1.4 for a long >>> time), mainly for 64-bit support, and the biggest problem I have with >>> it is bridge support's memory usage. One of my applications which used >>> to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb >>> when built with pyobjc 2.2. >> >> Ouch. > > Can it be that a good part of this doubling in size is just from pointers going from 32 to 64bits? > > -- > Leonardo Santagada > santagada at gmail.com > > > > No, 48mb is in 32bit mode. When in 64-bit mode, it's 95mb -- Virgil Dupras Hardcoded Software http://www.hardcoded.net |
|
From: Ronald O. <ron...@ma...> - 2010-01-20 21:55:21
Attachments:
smime.p7s
|
On 20 Jan, 2010, at 22:52, Virgil Dupras wrote: > On Wed, Jan 20, 2010 at 10:50 PM, Leonardo Santagada > <san...@gm...> wrote: >> >> On Jan 20, 2010, at 7:45 PM, Ronald Oussoren wrote: >> >>> On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: >>> >>>> Hi there, >>>> >>>> I recently started to use PyObjC 2.2 (after having use 1.4 for a long >>>> time), mainly for 64-bit support, and the biggest problem I have with >>>> it is bridge support's memory usage. One of my applications which used >>>> to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb >>>> when built with pyobjc 2.2. >>> >>> Ouch. >> >> Can it be that a good part of this doubling in size is just from pointers going from 32 to 64bits? >> >> -- >> Leonardo Santagada >> santagada at gmail.com >> >> >> >> > > No, 48mb is in 32bit mode. When in 64-bit mode, it's 95mb This definitly needs time with a measurement tool then, I wouldn't have expected that the amount of memory used would actually double in 64-bit mode. Ronald |
|
From: Virgil D. <hs...@ha...> - 2010-01-26 14:08:37
|
On Wed, Jan 20, 2010 at 10:55 PM, Ronald Oussoren <ron...@ma...> wrote: > > On 20 Jan, 2010, at 22:52, Virgil Dupras wrote: > >> On Wed, Jan 20, 2010 at 10:50 PM, Leonardo Santagada >> <san...@gm...> wrote: >>> >>> On Jan 20, 2010, at 7:45 PM, Ronald Oussoren wrote: >>> >>>> On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: >>>> >>>>> Hi there, >>>>> >>>>> I recently started to use PyObjC 2.2 (after having use 1.4 for a long >>>>> time), mainly for 64-bit support, and the biggest problem I have with >>>>> it is bridge support's memory usage. One of my applications which used >>>>> to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb >>>>> when built with pyobjc 2.2. >>>> >>>> Ouch. >>> >>> Can it be that a good part of this doubling in size is just from pointers going from 32 to 64bits? >>> >>> -- >>> Leonardo Santagada >>> santagada at gmail.com >>> >>> >>> >>> >> >> No, 48mb is in 32bit mode. When in 64-bit mode, it's 95mb > > This definitly needs time with a measurement tool then, I wouldn't have expected that the amount of memory used would actually double in 64-bit mode. > > Ronald > > I just wanted to let you know, Ronald (and everyone else), that I began working on an experimental fork of PyObjC to reduce its initial memory usage. I have just committed a change that reduces initial memory usage for "import AppKit" in 64-bit from 58mb to 31 mb without making a single test unit fail. This fork is at http://hg.hardcoded.net/pyobjc . I have also documented by experiments in the project's wiki. I'd greatly appreciate your feedbacks on this. -- Virgil Dupras Hardcoded Software http://www.hardcoded.net |
|
From: Ronald O. <ron...@ma...> - 2010-01-26 14:56:24
Attachments:
smime.p7s
|
On 26 Jan, 2010, at 14:40, Virgil Dupras wrote: > On Wed, Jan 20, 2010 at 10:55 PM, Ronald Oussoren > <ron...@ma...> wrote: >> >> On 20 Jan, 2010, at 22:52, Virgil Dupras wrote: >> >>> On Wed, Jan 20, 2010 at 10:50 PM, Leonardo Santagada >>> <san...@gm...> wrote: >>>> >>>> On Jan 20, 2010, at 7:45 PM, Ronald Oussoren wrote: >>>> >>>>> On 20 Jan, 2010, at 12:30, Virgil Dupras wrote: >>>>> >>>>>> Hi there, >>>>>> >>>>>> I recently started to use PyObjC 2.2 (after having use 1.4 for a long >>>>>> time), mainly for 64-bit support, and the biggest problem I have with >>>>>> it is bridge support's memory usage. One of my applications which used >>>>>> to use 22mb of memory on launch when built with pyobjc 1.4 used 48mb >>>>>> when built with pyobjc 2.2. >>>>> >>>>> Ouch. >>>> >>>> Can it be that a good part of this doubling in size is just from pointers going from 32 to 64bits? >>>> >>>> -- >>>> Leonardo Santagada >>>> santagada at gmail.com >>>> >>>> >>>> >>>> >>> >>> No, 48mb is in 32bit mode. When in 64-bit mode, it's 95mb >> >> This definitly needs time with a measurement tool then, I wouldn't have expected that the amount of memory used would actually double in 64-bit mode. >> >> Ronald >> >> > > I just wanted to let you know, Ronald (and everyone else), that I > began working on an experimental fork of PyObjC to reduce its initial > memory usage. I have just committed a change that reduces initial > memory usage for "import AppKit" in 64-bit from 58mb to 31 mb without > making a single test unit fail. This fork is at > http://hg.hardcoded.net/pyobjc . I have also documented by experiments > in the project's wiki. I'd greatly appreciate your feedbacks on this. Interesting. At first glance your change indicates that the xml files contain loads of junk that should be removed. That is, the nodes that shouldn't be included according to "methodNodeHasSpecialArgs" probably shouldn't be in the xml file in the first place. That means the bridgesupport generator is even more broken than I already knew. That is not a complete surprise though, the pyobjc generator is a crude hack that should be replaced by a cleaner design. If I had the time I'd rearchitect the metadata generator into two parts: (1) a clang-based tool that just prints all definitions in header files into some convenient format (json or xml) and (2) a tool that can merge the output of the first tool for various architectures and an exceptions file. For added bonus points there should be a gui tool that makes editing the exceptions easier. BTW. I noticed you also intend to work on documentation: I'm (slowly) converting documentation into Sphynx format in the pyobjc-core trunk. I wouldn't mind adding internal C documentation to that as well. Ronald > -- > Virgil Dupras > Hardcoded Software > http://www.hardcoded.net > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Pyobjc-dev mailing list > Pyo...@li... > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X