Re: [pysnmp-users] Re: [Twisted-Python] Twisted and SNMP

[Mike C. F. <mcf...@ro...>]

A list for users of pure-Python SNMP framework <pysnmp-users.lists.sourceforge.net>
List-Post: <mailto:pys...@li...>
List-Help: <mailto:pys...@li...?subject=help>
List-Subscribe: <https://lists.sourceforge.net/lists/listinfo/pysnmp-users>,
	<mailto:pys...@li...?subject=subscribe>
List-Archive: <http://sourceforge.net/mailarchive/forum.php?forum=pysnmp-users>

Ilya Etingof wrote:

>>>but if anyone is interested in bringing a full high performance ASN
>>>coder-decoder to twisted, i'd be willing to help.
>>>      
>>>
>>Ilya (creator of PySNMP) just released a new version of PySNMP
>>yesterday.  It's noticeably faster feeling than the 3.3.x version.  I
>>don't have time to do timings with it, but I'd guess it's probably
>>taking 3/4 of the time of the previous version.  Still likely not 0.02
>>seconds on getbulk requests, but noticeably faster.
>>    
>>
...

>Perhaps the most noticable bottleneck there lays in the fact that building
>a tree of ASN1 objects involves massive classes instantiations. In the
>latest pysnmp code (3.4.x) I attempted to save on object creation by caching
>and reusing once created ASN1 objects.
>  
>
Hmm, caching is fine, but it can only go so far, and it's often 
detrimental to application structure and maintainability (particularly 
when a library starts requiring client apps to do caching of internal 
objects).  All in all I'd rather spend the time speeding up the core 
structures if/when I do wind up needing more speed.

Looking at the asn.base module, it looks as though there's a lot of 
internal structure being allocated for each ASN object, an instance 
dictionary, and two or three lists.  For most of those objects I'd guess 
we could use __slots__ to reduce memory thrashing somewhat (about 26 
bytes/object + 8 bytes/(attribute*record) if my calculations are 
right).  Slots are a bit of a pain, but this is exactly the kind of 
situation for which they are intended, huge numbers of tiny objects with 
well-defined structure.

For the fixed types may be possible to convert the base class to 
meta-classes in order to have them calculate and then share the _names 
lists (~ another 26 bytes/object + 4bytes/(name*object)) among all 
instances (preferably as a tuple in that case, of course)?  Would need 
to study how these classes are used throughout the system to really 
guess what can be optimised.

Need to do a hotspot profile of running code to know what to start 
working on.  The initialisation code looks pretty involved in a number 
of cases... anyway, no time tonight, have to work on application-level 
stuff.

Have fun all,
Mike

_______________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://members.rogers.com/mcfletch/