cl-net-snmp-general — cl-net-snmp general

Re: [lisp-snmp] latest SBCL not working with latest cl-net-snmp

[David L. <le...@ma...>]

I will try trunk.

My next question is how exactly to build it and use it without asdf- 
install :-)

I'm a bit new to all this common lisp packaging stuff and want to make  
sure I get my paths correct or whatever I need to do to get it  
loaded.  I suppose that's more of a question with an answer in the  
SBCL manual.

Thank you for your prompt reply!

Dave
On Jul 9, 2009, at 10:04 AM, Chun Tian (binghe) wrote:

> Hi, David
>
> Latest release of cl-net-snmp is a bit old, and your modifications  
> on update-mib.lisp are not supported in theory.
>
> I'm not sure what exactly you want to do, but I suggest you trying  
> my "trunk" code, in which SBCL are supported by update-mib.lisp  
> already.
>
> You'll need following packages directly from SVN:
>
> 1) SNMP trunk (pre 6.0), checkout from
>
> https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/snmp/trunk
>
> 2) ASN.1 trunk (pre 5.0), checkout from
>
> https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/asn.1/ 
> trunk
>
> 3) USOCKET "experimental-udp" branch (pre 0.5), from
>
> svn://common-lisp.net/project/usocket/svn/usocket/branches/ 
> experimental-udp
>
> I hope all your issues can be directly solved by these latest trunk  
> code.
>
> Regards,
>
> Chun Tian (binghe)
>
>> I, with the help of a friend, discovered that many functions are not
>> exported unless used with the lispworks Common Lisp environment.   
>> This
>> includes seemingly very useful functions like 'snmp:update-mib.
>>
>> I'd REALLY like to use this project to test an SNMP agent I've been
>> developing in another language, but I really would like to be able to
>> compile MIBs too.
>>
>> Here's what I've done so far.
>>
>> [on snmp version 5.19]
>>
>> 1. I've asdf-installed the whole snmp software stack
>> 2. I've removed the #+(lispworks) line from above
>> (:file "update-mib" :depends-on ("mib"))
>>
>> so that I can get that file compiled via sbcl
>>
>> 3. to make update-mib.lisp compile I've commented out:
>> ;;; Empty and old MIB module
>> ;(eval-when (:load-toplevel :execute)
>> ;  (setf *mib-name-map*
>> ;        '((rfc1155-smi . snmpv2-smi)
>> ;          (rfc1212 . nil)
>> ;          (rfc-1212 . nil)
>> ;          (rfc1213-mib . nil)
>> ;          (rfc-1215 . nil)))
>> ;  (make-mib-name-map))
>>
>> from update-mib.lisp in the snmp-5.19 directory of my site lisp  
>> stuff.
>>
>> 4. Now I can  (require :asdf) and (require :snmp) and everything
>> seemingly loads up ok.  But when I try to use snmp:update-mib I get
>> complaints about undefined functions for asn.1:compile-asn.1
>>
>> I'm at the point where I feel I need help, and would like to make  
>> sure
>> my changes, if they're any good, go upstream :-)
>>
>>
>> Any help would be greatly appreciated.  I do love the with-open-
>> session and snmp-walk implementation and results by the way.  This
>> looks like a great package!  I hope I can use it!
>>
>> Dave
>>
>> ------------------------------------------------------------------------------
>> Enter the BlackBerry Developer Challenge
>> This is your chance to win up to $100,000 in prizes! For a limited  
>> time,
>> vendors submitting new applications to BlackBerry App World(TM)  
>> will have
>> the opportunity to enter the BlackBerry Developer Challenge. See  
>> full prize
>> details at: http://p.sf.net/sfu/Challenge
>> _______________________________________________
>> cl-net-snmp-general mailing list
>> cl-...@li...
>> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general
>

Re: [lisp-snmp] latest SBCL not working with latest cl-net-snmp

[Chun T. (binghe) <bin...@gm...>]

Hi, David

Latest release of cl-net-snmp is a bit old, and your modifications on  
update-mib.lisp are not supported in theory.

I'm not sure what exactly you want to do, but I suggest you trying my  
"trunk" code, in which SBCL are supported by update-mib.lisp already.

You'll need following packages directly from SVN:

1) SNMP trunk (pre 6.0), checkout from

https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/snmp/trunk

2) ASN.1 trunk (pre 5.0), checkout from

https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/asn.1/trunk

3) USOCKET "experimental-udp" branch (pre 0.5), from

svn://common-lisp.net/project/usocket/svn/usocket/branches/ 
experimental-udp

I hope all your issues can be directly solved by these latest trunk  
code.

Regards,

Chun Tian (binghe)

> I, with the help of a friend, discovered that many functions are not
> exported unless used with the lispworks Common Lisp environment.  This
> includes seemingly very useful functions like 'snmp:update-mib.
>
> I'd REALLY like to use this project to test an SNMP agent I've been
> developing in another language, but I really would like to be able to
> compile MIBs too.
>
> Here's what I've done so far.
>
> [on snmp version 5.19]
>
> 1. I've asdf-installed the whole snmp software stack
> 2. I've removed the #+(lispworks) line from above
> (:file "update-mib" :depends-on ("mib"))
>
> so that I can get that file compiled via sbcl
>
> 3. to make update-mib.lisp compile I've commented out:
> ;;; Empty and old MIB module
> ;(eval-when (:load-toplevel :execute)
> ;  (setf *mib-name-map*
> ;        '((rfc1155-smi . snmpv2-smi)
> ;          (rfc1212 . nil)
> ;          (rfc-1212 . nil)
> ;          (rfc1213-mib . nil)
> ;          (rfc-1215 . nil)))
> ;  (make-mib-name-map))
>
> from update-mib.lisp in the snmp-5.19 directory of my site lisp stuff.
>
> 4. Now I can  (require :asdf) and (require :snmp) and everything
> seemingly loads up ok.  But when I try to use snmp:update-mib I get
> complaints about undefined functions for asn.1:compile-asn.1
>
> I'm at the point where I feel I need help, and would like to make sure
> my changes, if they're any good, go upstream :-)
>
>
> Any help would be greatly appreciated.  I do love the with-open-
> session and snmp-walk implementation and results by the way.  This
> looks like a great package!  I hope I can use it!
>
> Dave
>
> ------------------------------------------------------------------------------
> Enter the BlackBerry Developer Challenge
> This is your chance to win up to $100,000 in prizes! For a limited  
> time,
> vendors submitting new applications to BlackBerry App World(TM) will  
> have
> the opportunity to enter the BlackBerry Developer Challenge. See  
> full prize
> details at: http://p.sf.net/sfu/Challenge
> _______________________________________________
> cl-net-snmp-general mailing list
> cl-...@li...
> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general

[lisp-snmp] latest SBCL not working with latest cl-net-snmp

[David L. <le...@ma...>]

I, with the help of a friend, discovered that many functions are not  
exported unless used with the lispworks Common Lisp environment.  This  
includes seemingly very useful functions like 'snmp:update-mib.

I'd REALLY like to use this project to test an SNMP agent I've been  
developing in another language, but I really would like to be able to  
compile MIBs too.

Here's what I've done so far.

[on snmp version 5.19]

1. I've asdf-installed the whole snmp software stack
2. I've removed the #+(lispworks) line from above
(:file "update-mib" :depends-on ("mib"))

so that I can get that file compiled via sbcl

3. to make update-mib.lisp compile I've commented out:
;;; Empty and old MIB module
;(eval-when (:load-toplevel :execute)
;  (setf *mib-name-map*
;        '((rfc1155-smi . snmpv2-smi)
;          (rfc1212 . nil)
;          (rfc-1212 . nil)
;          (rfc1213-mib . nil)
;          (rfc-1215 . nil)))
;  (make-mib-name-map))

from update-mib.lisp in the snmp-5.19 directory of my site lisp stuff.

4. Now I can  (require :asdf) and (require :snmp) and everything  
seemingly loads up ok.  But when I try to use snmp:update-mib I get  
complaints about undefined functions for asn.1:compile-asn.1

I'm at the point where I feel I need help, and would like to make sure  
my changes, if they're any good, go upstream :-)


Any help would be greatly appreciated.  I do love the with-open- 
session and snmp-walk implementation and results by the way.  This  
looks like a great package!  I hope I can use it!

Dave

[lisp-snmp] ipAdEntAddr versus the simple-oid-cache

[John F. <jf...@ms...>]

Dear Chun Tian,

We have to walk IF-MIB::ipAdEntAddr from many routers. The results of
this are a list of IP addresses the router has. Unfortunately, each OID
in the result encodes the IP address. 

For example,

IP-MIB::ipAdEntAddr.127.0.0.1 = IpAddress: 127.0.0.1

We were fetching from many routers and storing these OIDs, so the
*simple-oid-cache* was getting full. The :test 'equal on the OIDs with
similar prefixes was very slow.

Without the cache our benchmark completes in about 5 seconds, with the
cache it does not complete after a few minutes.

We were mucking about with some code for replacing the simple-oid-cache
with one that is limited in size. However, in our case, there isn't any
benefit in having the cache at all, as the OIDs are always different, so
we stopped our experiments. In case you are interested, the experiments
are below (note that the array version incorrectly does not check if the
element in the array actually has a matching oid-number-list).

What was the reason for introducing the simple-oid-cache in the first
place? Maybe we can develop a compromise version that suits both use
cases?


PS. I am sorry, I made a mistake about the weakness for the hash table -- it
can be downgraded to :weakness :value, I think.


Index: oid-reader.lisp
===================================================================
--- oid-reader.lisp	(revision 769)
+++ oid-reader.lisp	(working copy)
@@ -81,10 +81,42 @@
 (defmethod oid ((oid-vector vector))
   (oid (coerce oid-vector 'list)))
 
+#+ (and asn.1-features:oid-cache nil)
+(progn
+  (defvar *simple-oid-cache* 
+    (trivial-garbage:make-weak-hash-table :test 'equal
+					  :weakness :value))
+  
+  (defun (setf simple-oid-cache) (oid oid-number-list)
+    (setf (gethash oid-number-list *simple-oid-cache*) oid))
+  
+  (defun simple-oid-cache (oid-number-list)
+    (gethash oid-number-list  *simple-oid-cache*))
+)
+
 #+asn.1-features:oid-cache
-(defvar *simple-oid-cache* (make-hash-table :test 'equal
-                                            #+lispworks :weak-kind #+lispworks :value))
+(progn
+  (defvar *simple-oid-cache* 
+    (make-array 10))
+ 
+  (defun simple-oid-cache-hash (list)
+    (declare (optimize speed))
+    (let ((sum 0))
+      (declare (type fixnum sum))
+      (loop for i of-type unsigned-byte in list
+	    do (setf sum (mod (* 17 (+ (* 13 sum) i)) (1+ most-positive-fixnum))))
+      (mod sum (length *simple-oid-cache*))))
 
+  (defmacro simple-oid-cache-aref (oid-number-list)
+    `(aref *simple-oid-cache* (simple-oid-cache-hash ,oid-number-list)))
+  
+  (defun (setf simple-oid-cache) (oid oid-number-list)
+    (setf (simple-oid-cache-aref oid-number-list) oid))
+  
+  (defun simple-oid-cache (oid-number-list)
+    (simple-oid-cache-aref oid-number-list)))
+
+
 (defun oid-helper (tokens base)
   (declare (type list tokens)
            (type object-id base))
@@ -99,7 +131,7 @@
                (oid-helper (cdr tokens) b)
              (let ((o (make-instance 'simple-oid :value tokens :parent base)))
                #+asn.1-features:oid-cache
-               (setf (gethash (oid-number-list o) *simple-oid-cache*) o)
+               (setf (simple-oid-cache (oid-number-list o)) o)
                o))))
         (symbol ; walk through children of base to find a child with the name
          (with-hash-table-iterator (get-child (oid-children base))
@@ -121,7 +153,7 @@
          ;; optimize: plain number list directly to simple-oid when possible         
          (if #+asn.1-features:oid-cache (every #'numberp (cdr oid-tokens))
              #-asn.1-features:oid-cache nil
-             (let ((o (gethash oid-tokens *simple-oid-cache*)))
+             (let ((o (simple-oid-cache oid-tokens)))
                (or o
                    (prog1 (oid-helper oid-tokens |zero|)
                      #+ignore

Re: [lisp-snmp] Faster ASN encode (reducing memory pressure)

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@me...> writes:

> Last night I did some basic test, and it's obviously that your patch
> works just fine and didn't change the interface between ASN.1 and SNMP
> package. It's a excellent work which towards a highly optimized ASN.1/
> SNMP package, and thank you again for look into my work deeply.
>
> I believe, sometimes to make Lisp code running fast, something have to
> be gave up, like code beauty and express, 

Hopefully not, if you look at it in the right way and put in a good set
of support macros and general infrastructure then it should be smaller
to read and faster to run as well.

> and only those greatest Lisp
> projects can keep both the efficiency and beauty at the same time (for
> example, CL-HTTP).

I don't think cl-http is very efficient ;-)

> And as you already told, "the best way forward would be to restructure
> the whole thing ...", I think you're totally right.
>
> So, I've made some decision:
>
> * Create a branch [1] for holding all your "fast encode" work,
> * Do a "restructure" work on trunk in next few weeks to try to achieve
> both speed and art (by learning your work mainly)

Great, well my small changes were just done very quickly and I'm sure
that you can think of many ways to improve them and integrate a more
sensible infrastructure around encode and decode.

I figure the best thing to do for encode is to set up a nice bunch of
macros so that it is easy and pretty to buffer up temporary responses
before calculating the length. The old code did lots of horrible things
like building up a list of bytes then coercing it to a vector, which
were neither efficient nor pretty.

For decode, I'm not sure how to do it. Why shouldn't the basic buffer
structure be a simple-array of unsigned-byte, maybe with an extra start
and end pointer (or an array displaced from such a simple-array so that
sub parts can be passed around easily)?  Using declared simple-arrays is
fast and allowing arrays displaced from simple arrays is only a little
slower provided you write explicit aref/etc. replacements that take this
into account.

I vaguely remember seeing places where the decode routines converted
haphazardly between lists, vectors, streams and so on, and then
mysteriously build up lists of bytes which then have to be nreverse. All
of this is overly complicated and can not only be made more efficient
but simpler at the same time.

[...]

Re: [lisp-snmp] Faster ASN encode (reducing memory pressure)

[Chun T. (binghe) <bin...@gm...>]

[1] https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/asn.1/branches/faster-encode

> Hi, John
>
> Last night I did some basic test, and it's obviously that your patch
> works just fine and didn't change the interface between ASN.1 and SNMP
> package. It's a excellent work which towards a highly optimized ASN.1/
> SNMP package, and thank you again for look into my work deeply.
>
> I believe, sometimes to make Lisp code running fast, something have to
> be gave up, like code beauty and express, and only those greatest Lisp
> projects can keep both the efficiency and beauty at the same time (for
> example, CL-HTTP). And as you already told, "the best way forward
> would be to restructure the whole thing ...", I think you're totally
> right.
>
> So, I've made some decision:
>
> * Create a branch [1] for holding all your "fast encode" work,
> * Do a "restructure" work on trunk in next few weeks to try to achieve
> both speed and art (by learning your work mainly)
>
> Please let me know your comments.
>
> Best regards,
>
> Chun Tian (binghe)
>
>
>> Dear Chun Tian,
>>
>> Our primary platform is Allegro 8.1, and its garbage collector
>> struggles
>> awfully when garbage is generated.
>>
>> These changes take our time on one benchmark down from more than  
>> three
>> minutes to about two minutes, and only about 20% slower than SBCL.
>>
>> Therefore I have modified the ASN encode to reduce the garbage
>> generated. I used a variety of techniques to do it (compiler macro
>> constant folding, value caching, etc.) and modified quite a lot of  
>> the
>> code, so I guess you should look it over and give it a good
>> refactoring.
>>
>> In my opinion, the best way forward would be to restructure the whole
>> thing so that the encode functions have a nice scratch buffer and
>> don't
>> need to keep reallocating and copying memory all the time.
>>
>> Simply creating the SNMP message objects takes a lot of time and  
>> could
>> be avoided as far as I can see.
>>
>> However, the decode is now the juicier time waster. The weird
>> transitions between stream, lists and vectors make us suffer quite a
>> lot.
>>
>> Again, why not stick with a byte buffer.
>>
>> <asn.1-faster-encode.patch>Index: pdu.lisp
>> ===================================================================
>> --- pdu.lisp	(revision 769)
>> +++ pdu.lisp	(working copy)
>> @@ -108,29 +108,31 @@
>>      (format stream "(~D (~D ~D)) ~A"
>>              request-id non-repeaters max-repetitions variable-
>> bindings))))
>>
>> -(defmethod ber-encode-pdu ((value pdu) tag)
>> -  (declare (type fixnum tag))
>> -  (with-slots (request-id error-status error-index variable-
>> bindings) value
>> -    (let ((sub-encode (ber-encode-list (list request-id
>> -                                             error-status
>> -                                             error-index
>> -                                             variable-bindings))))
>> -      (concatenate 'vector
>> -                   (ber-encode-type 2 1 tag)
>> -                   (ber-encode-length (length sub-encode))
>> -                   sub-encode))))
>> +(defmacro do-ber-encode-pdu (value tag)
>> +  (flet ((encode (vars)
>> +	   `(with-slots ,vars ,value
>> +	      (let ,(loop for v in vars collect `(,v (ber-encode ,v)))
>> +		(let ((length (+ ,@(loop for v in vars collect `(length ,v)))))
>> +		  (asn.1::ber-array-from-seqs
>> +		       (ber-encode-type 2 1 ,tag)
>> +		       (ber-encode-length length)
>> +		       ,@vars))))))
>> +    (encode '(request-id
>> +	     error-status
>> +	     error-index
>> +	     variable-bindings))))
>>
>> (defmethod ber-encode ((value get-request-pdu))
>> -  (ber-encode-pdu value +get-request-pdu+))
>> +  (do-ber-encode-pdu value +get-request-pdu+))
>>
>> (defmethod ber-encode ((value get-next-request-pdu))
>> -  (ber-encode-pdu value +get-next-request-pdu+))
>> +  (do-ber-encode-pdu value +get-next-request-pdu+))
>>
>> (defmethod ber-encode ((value response-pdu))
>> -  (ber-encode-pdu value +response-pdu+))
>> +  (do-ber-encode-pdu value +response-pdu+))
>>
>> (defmethod ber-encode ((value set-request-pdu))
>> -  (ber-encode-pdu value +set-request-pdu+))
>> +  (do-ber-encode-pdu value +set-request-pdu+))
>>
>> (defmethod ber-encode ((value trap-pdu))
>>  (with-slots (enterprise agent-addr generic-trap specific-trap
>> timestamp
>> @@ -158,13 +160,13 @@
>>                   sub-encode))))
>>
>> (defmethod ber-encode ((value inform-request-pdu))
>> -  (ber-encode-pdu value +inform-request-pdu+))
>> +  (do-ber-encode-pdu value +inform-request-pdu+))
>>
>> (defmethod ber-encode ((value snmpv2-trap-pdu))
>> -  (ber-encode-pdu value +snmpv2-trap-pdu+))
>> +  (do-ber-encode-pdu value +snmpv2-trap-pdu+))
>>
>> (defmethod ber-encode ((value report-pdu))
>> -  (ber-encode-pdu value +report-pdu+))
>> +  (do-ber-encode-pdu value +report-pdu+))
>>
>> (defun ber-decode-pdu (stream length class)
>>  (declare (type stream stream)
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity
>> professionals. Meet
>> the minds behind Google Creative Lab, Visual Complexity,  
>> Processing, &
>> iPhoneDevCamp asthey present alongside digital heavyweights like
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
>> _______________________________________________
>> cl-net-snmp-general mailing list
>> cl-...@li...
>> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general
>
>
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com
> _______________________________________________
> cl-net-snmp-general mailing list
> cl-...@li...
> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general

Re: [lisp-snmp] Faster ASN encode (reducing memory pressure)

[Chun T. (binghe) <bin...@me...>]

Hi, John

Last night I did some basic test, and it's obviously that your patch  
works just fine and didn't change the interface between ASN.1 and SNMP  
package. It's a excellent work which towards a highly optimized ASN.1/ 
SNMP package, and thank you again for look into my work deeply.

I believe, sometimes to make Lisp code running fast, something have to  
be gave up, like code beauty and express, and only those greatest Lisp  
projects can keep both the efficiency and beauty at the same time (for  
example, CL-HTTP). And as you already told, "the best way forward  
would be to restructure the whole thing ...", I think you're totally  
right.

So, I've made some decision:

* Create a branch [1] for holding all your "fast encode" work,
* Do a "restructure" work on trunk in next few weeks to try to achieve  
both speed and art (by learning your work mainly)

Please let me know your comments.

Best regards,

Chun Tian (binghe)


> Dear Chun Tian,
>
> Our primary platform is Allegro 8.1, and its garbage collector  
> struggles
> awfully when garbage is generated.
>
> These changes take our time on one benchmark down from more than three
> minutes to about two minutes, and only about 20% slower than SBCL.
>
> Therefore I have modified the ASN encode to reduce the garbage
> generated. I used a variety of techniques to do it (compiler macro
> constant folding, value caching, etc.) and modified quite a lot of the
> code, so I guess you should look it over and give it a good  
> refactoring.
>
> In my opinion, the best way forward would be to restructure the whole
> thing so that the encode functions have a nice scratch buffer and  
> don't
> need to keep reallocating and copying memory all the time.
>
> Simply creating the SNMP message objects takes a lot of time and could
> be avoided as far as I can see.
>
> However, the decode is now the juicier time waster. The weird
> transitions between stream, lists and vectors make us suffer quite a
> lot.
>
> Again, why not stick with a byte buffer.
>
> <asn.1-faster-encode.patch>Index: pdu.lisp
> ===================================================================
> --- pdu.lisp	(revision 769)
> +++ pdu.lisp	(working copy)
> @@ -108,29 +108,31 @@
>       (format stream "(~D (~D ~D)) ~A"
>               request-id non-repeaters max-repetitions variable- 
> bindings))))
>
> -(defmethod ber-encode-pdu ((value pdu) tag)
> -  (declare (type fixnum tag))
> -  (with-slots (request-id error-status error-index variable- 
> bindings) value
> -    (let ((sub-encode (ber-encode-list (list request-id
> -                                             error-status
> -                                             error-index
> -                                             variable-bindings))))
> -      (concatenate 'vector
> -                   (ber-encode-type 2 1 tag)
> -                   (ber-encode-length (length sub-encode))
> -                   sub-encode))))
> +(defmacro do-ber-encode-pdu (value tag)
> +  (flet ((encode (vars)
> +	   `(with-slots ,vars ,value
> +	      (let ,(loop for v in vars collect `(,v (ber-encode ,v)))
> +		(let ((length (+ ,@(loop for v in vars collect `(length ,v)))))
> +		  (asn.1::ber-array-from-seqs
> +		       (ber-encode-type 2 1 ,tag)
> +		       (ber-encode-length length)
> +		       ,@vars))))))
> +    (encode '(request-id
> +	     error-status
> +	     error-index
> +	     variable-bindings))))
>
> (defmethod ber-encode ((value get-request-pdu))
> -  (ber-encode-pdu value +get-request-pdu+))
> +  (do-ber-encode-pdu value +get-request-pdu+))
>
> (defmethod ber-encode ((value get-next-request-pdu))
> -  (ber-encode-pdu value +get-next-request-pdu+))
> +  (do-ber-encode-pdu value +get-next-request-pdu+))
>
> (defmethod ber-encode ((value response-pdu))
> -  (ber-encode-pdu value +response-pdu+))
> +  (do-ber-encode-pdu value +response-pdu+))
>
> (defmethod ber-encode ((value set-request-pdu))
> -  (ber-encode-pdu value +set-request-pdu+))
> +  (do-ber-encode-pdu value +set-request-pdu+))
>
> (defmethod ber-encode ((value trap-pdu))
>   (with-slots (enterprise agent-addr generic-trap specific-trap  
> timestamp
> @@ -158,13 +160,13 @@
>                    sub-encode))))
>
> (defmethod ber-encode ((value inform-request-pdu))
> -  (ber-encode-pdu value +inform-request-pdu+))
> +  (do-ber-encode-pdu value +inform-request-pdu+))
>
> (defmethod ber-encode ((value snmpv2-trap-pdu))
> -  (ber-encode-pdu value +snmpv2-trap-pdu+))
> +  (do-ber-encode-pdu value +snmpv2-trap-pdu+))
>
> (defmethod ber-encode ((value report-pdu))
> -  (ber-encode-pdu value +report-pdu+))
> +  (do-ber-encode-pdu value +report-pdu+))
>
> (defun ber-decode-pdu (stream length class)
>   (declare (type stream stream)
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity  
> professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, &
> iPhoneDevCamp asthey present alongside digital heavyweights like  
> Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com  
> _______________________________________________
> cl-net-snmp-general mailing list
> cl-...@li...
> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general

[lisp-snmp] Faster ASN encode (reducing memory pressure)

[John F. <jf...@ms...>]

Dear Chun Tian,

Our primary platform is Allegro 8.1, and its garbage collector struggles
awfully when garbage is generated.

These changes take our time on one benchmark down from more than three
minutes to about two minutes, and only about 20% slower than SBCL.

Therefore I have modified the ASN encode to reduce the garbage
generated. I used a variety of techniques to do it (compiler macro
constant folding, value caching, etc.) and modified quite a lot of the
code, so I guess you should look it over and give it a good refactoring.

In my opinion, the best way forward would be to restructure the whole
thing so that the encode functions have a nice scratch buffer and don't
need to keep reallocating and copying memory all the time.

Simply creating the SNMP message objects takes a lot of time and could
be avoided as far as I can see.

However, the decode is now the juicier time waster. The weird
transitions between stream, lists and vectors make us suffer quite a
lot.

Again, why not stick with a byte buffer.

[lisp-snmp] Weak hashtables for OIDs

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
> 在 2009-5-11，10:42， John Fremlin 写道：
[...]
>> "Chun Tian (binghe)" <bin...@gm...> writes:
>> [...]
>>> Using a "weak hash table" as a cache for holding SIMPLE-OID instances
>>> which already created in current lisp image, so that few SIMPLE-OID
>>> created when repeat do SNMP-WALK on same MIB node. The key of this
>>> hash table would be a "oid number list" as a list, and the value
>>> would
>>> be the correspond SIMPLE-OID instance.
>>>
>>> Since "weak hash table" is not in CL standard, I'll support only a
>>> few
>>> implementations included at least LispWorks, AllegroCL and SBCL, and
>>> make a new "ASN.1 feature" (OID-CACHE) for this work. Do you think
>>> this features would be useful for you?
>>
>> I think this might be a bad idea and probably will decrease
>> performance
>> on our workload, as we have many OIDs that are never seen again, so
>> putting them into a weak hash-table just causes more problems for the
>> garbage collector (they are difficult to implement efficiently).
>
> OK ... so you don't need it in case you have so many one-time OIDs.

I tried this idea out.

It seems it saved about 10% of memory on Allegro, but did not make any
difference to the speed of our (rather bad) benchmark. 

Maybe it will help, maybe not ;-)

Anyway, here is the patch to use trivial-garbage which has portable weak
hashtables.

(It also fixes some wrong references to #+asn.1-yacc instead of
#+asn.1-features:cl-yacc)

[lisp-snmp] Optimizing the ASN.1 library

[John F. <jf...@ms...>]

Dear Chun Tian,

It seems that the biggest use of time (25%) is in the decode. I guess
that's to be expected. Having the simple vector OIDs should really help
for that.

[...]

Re: [lisp-snmp] Misc fiddles for new OID types

[Chun T. (binghe) <bin...@gm...>]

All right, I'll see what I can do.

--binghe

在 2009-5-11，12:36， John Fremlin 写道：

> Hi Chun Tian,
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>>> As I said before, the most sensible thing is probably just to use
>>> simple
>>> vectors as OIDs.
>>
>> I think this change wouldn't bring any essential performance boost
>> from the view of algorithm, right?
>
> If you are talking about time complexity in a naive way then that's
> true.
>
> But the time complexity in our workload is heavily affected by the GC.
>
>> A simple vector is "fast" than A list only when we need to access
>> specific elements in it. For simple vectors, we have O(1) performance
>> here (and O(n) for lists). But you should see, in ASN.1 package, any
>> time we need this vector/list, we need all elements from it. Here we
>> just waste some memory (which equals to the number of SIMPLE-OID
>> instances times a small factor), compare to the whole memory
>> requirements it's nothing.
>
> I think compared to the whole memory requirements and the strain on  
> the
> garbage collector it is very important.
>
> Why do you think that it is not important?
>
> In our profile, the biggest memory user was OID creation (instances)  
> and
> the oid-number-list creation.
>
> Both would be heavily reduced with simple vectors for OIDs
>
> [...]
>

Re: [lisp-snmp] Misc fiddles for new OID types

[John F. <jf...@ms...>]

Hi Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
>> As I said before, the most sensible thing is probably just to use
>> simple
>> vectors as OIDs.
>
> I think this change wouldn't bring any essential performance boost
> from the view of algorithm, right?

If you are talking about time complexity in a naive way then that's
true.

But the time complexity in our workload is heavily affected by the GC.

> A simple vector is "fast" than A list only when we need to access
> specific elements in it. For simple vectors, we have O(1) performance
> here (and O(n) for lists). But you should see, in ASN.1 package, any
> time we need this vector/list, we need all elements from it. Here we
> just waste some memory (which equals to the number of SIMPLE-OID
> instances times a small factor), compare to the whole memory
> requirements it's nothing.

I think compared to the whole memory requirements and the strain on the
garbage collector it is very important.

Why do you think that it is not important?

In our profile, the biggest memory user was OID creation (instances) and
the oid-number-list creation.

Both would be heavily reduced with simple vectors for OIDs

[...]

Re: [lisp-snmp] Misc fiddles for new OID types

[Chun T. (binghe) <bin...@gm...>]

在 2009-5-11，10:42， John Fremlin 写道：

> Dear Chun Tian,
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>> Using a "weak hash table" as a cache for holding SIMPLE-OID instances
>> which already created in current lisp image, so that few SIMPLE-OID
>> created when repeat do SNMP-WALK on same MIB node. The key of this
>> hash table would be a "oid number list" as a list, and the value  
>> would
>> be the correspond SIMPLE-OID instance.
>>
>> Since "weak hash table" is not in CL standard, I'll support only a  
>> few
>> implementations included at least LispWorks, AllegroCL and SBCL, and
>> make a new "ASN.1 feature" (OID-CACHE) for this work. Do you think
>> this features would be useful for you?
>
> I think this might be a bad idea and probably will decrease  
> performance
> on our workload, as we have many OIDs that are never seen again, so
> putting them into a weak hash-table just causes more problems for the
> garbage collector (they are difficult to implement efficiently).

OK ... so you don't need it in case you have so many one-time OIDs.

>
>
> As I said before, the most sensible thing is probably just to use  
> simple
> vectors as OIDs.

I think this change wouldn't bring any essential performance boost  
from the view of algorithm, right?

A simple vector is "fast" than A list only when we need to access  
specific elements in it. For simple vectors, we have O(1) performance  
here (and O(n) for lists). But you should see, in ASN.1 package, any  
time we need this vector/list, we need all elements from it. Here we  
just waste some memory (which equals to the number of SIMPLE-OID  
instances times a small factor), compare to the whole memory  
requirements it's nothing.

Comments?

--binghe

>
>
> [...]
>

Re: [lisp-snmp] Misc fiddles for new OID types

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
> Using a "weak hash table" as a cache for holding SIMPLE-OID instances
> which already created in current lisp image, so that few SIMPLE-OID
> created when repeat do SNMP-WALK on same MIB node. The key of this
> hash table would be a "oid number list" as a list, and the value would
> be the correspond SIMPLE-OID instance.
>
> Since "weak hash table" is not in CL standard, I'll support only a few
> implementations included at least LispWorks, AllegroCL and SBCL, and
> make a new "ASN.1 feature" (OID-CACHE) for this work. Do you think
> this features would be useful for you?

I think this might be a bad idea and probably will decrease performance
on our workload, as we have many OIDs that are never seen again, so
putting them into a weak hash-table just causes more problems for the
garbage collector (they are difficult to implement efficiently).

As I said before, the most sensible thing is probably just to use simple
vectors as OIDs.

[...]

[lisp-snmp] Experimental SIMPLE-OID cache support

[Chun T. <bin...@me...>]

(GMail is down, let me change to MobileMe)

Hi, John

I've commited a early version of the experimental SIMPLE-OID cache  
support for ASN.1 package, see r769.

If you think this feature is useful for you, please help me port it to  
Allegro CL.

Thanks,

Chun Tian (binghe)

Re: [lisp-snmp] Misc fiddles for new OID types

[Chun T. (binghe) <bin...@gm...>]

Hi, John

Thanks for these SIMPLE-OID related fixes, I'm appreciated about them.

Now, I'm thinking on another performance enhancement when we have  
SIMPLE-OID:

Using a "weak hash table" as a cache for holding SIMPLE-OID instances  
which already created in current lisp image, so that few SIMPLE-OID  
created when repeat do SNMP-WALK on same MIB node. The key of this  
hash table would be a "oid number list" as a list, and the value would  
be the correspond SIMPLE-OID instance.

Since "weak hash table" is not in CL standard, I'll support only a few  
implementations included at least LispWorks, AllegroCL and SBCL, and  
make a new "ASN.1 feature" (OID-CACHE) for this work. Do you think  
this features would be useful for you?

Regards,

Chun Tian (binghe)

在 2009-5-8，09:56， John Fremlin 写道：

> Dear Chun Tian,
>
> Here are the misc helpers I put in to deal with the new OID types.
>
> Maybe you would like to fix them up or correct them?
>
> ASN.1 patch
>
> Index: runtime/object-id.lisp
> ===================================================================
> --- runtime/object-id.lisp	(revision 766)
> +++ runtime/object-id.lisp	(working copy)
> @@ -113,8 +113,9 @@
>   (slot-boundp oid 'parent))
>
> (defun oid-module-p (oid)
> -  (declare (type object-id oid))
> -  (slot-boundp oid 'module))
> +  (typecase oid
> +    (object-id
> +     (slot-boundp oid 'module))))
>
> (defun oid-syntax-p (oid)
>   (declare (type object-id oid))
> @@ -124,17 +125,29 @@
>   (declare (type object-id oid))
>   (symbol-name (oid-name oid)))
>
> -(defun oid-name-list (oid)
> -  (declare (type object-id oid))
> +(defgeneric oid-name-list (oid))
> +(defmethod oid-name-list ((oid object-id))
>   (labels ((iter (o acc)
>              (if (slot-boundp o 'parent)
> -                 (iter (oid-parent o) (cons (if (slot-boundp o 'name)
> -                                                (oid-name-string o)
> -                                              (oid-value o))
> -                                            acc))
> -               acc)))
> +                 (iter (oid-parent o)
> +		       (cons (if (slot-boundp o 'name)
> +				 (oid-name-string o)
> +				 (oid-value o))
> +			     acc))
> +		 acc)))
>     (iter oid nil)))
>
> +(defun oid-number-sublist (oid)
> +  (nthcdr
> +   (if (oid-parent-p oid)
> +       (oid-length (oid-parent oid))
> +       0)
> +   (oid-number-list oid)))
> +
> +(defmethod oid-name-list ((oid simple-oid))
> +  (append (when (oid-parent-p oid) (oid-name-list (oid-parent oid)))
> +	  (oid-number-sublist oid)))
> +
> (defmethod plain-value ((object simple-oid) &key default)
>   (declare (ignore default))
>   (oid-number-list object))
> Index: runtime/oid-walk.lisp
> ===================================================================
> --- runtime/oid-walk.lisp	(revision 766)
> +++ runtime/oid-walk.lisp	(working copy)
> @@ -85,10 +85,25 @@
>        (zerop (hash-table-count (oid-children oid)))))
>
> (defun oid-trunk-p (oid)
> -  (declare (type object-id oid))
>   (and (slot-boundp oid 'name)
>        (plusp (hash-table-count (oid-children oid)))))
>
> +(defun oid-find-base (oid)
> +  (labels ((iter (o acc)
> +	     (typecase o
> +	       (null (values nil acc))
> +	       (object-id
> +		(if (slot-boundp o 'name)
> +		    (values o acc)
> +		    (let ((p (oid-parent o))
> +			  (v (oid-value o)))
> +		      (iter p (cons v acc)))))
> +	       (simple-oid
> +		(iter (when (oid-parent-p o) (oid-parent o))
> +		      (append (oid-number-sublist o) acc))))))
> +    (iter oid nil)))
> +
> +
> (defun oid-find-leaf (oid)
>   "Find the leaf node in a oid's all parents"
>   (declare (type object-id oid))
>
>
>
> PS. Thanks for fixing runtime/ipaddress to not barf on length != 4.
>
> BTW
> (loop for i from 1 upto length
>    collect (read-byte stream))
>
> is the same as
>
> (loop repeat length
>        collect (read-byte stream))
>

Re: [lisp-snmp] ASN.1 dynamic loading

[Chun T. (binghe) <bin...@gm...>]

Hi, John

在 2009-5-8，09:02， John Fremlin 写道：

> Dear Chun Tian,
>
> I have merged your current ASN.1 and SNMP so that we are up to
> date.
>
> You improved the mechanism I made for dynamically loading ASN.1
> files. What is your approved way of doing it now? I stupidly cannot  
> find
> the right function.

Sorry, this work is still on progress, never got a working  
implementation. I'll make sure you are informed when it's done.

>
>
> By the way, in your features lisp-expr, the name for yacc should be
> cl-yacc, as that is used everywhere in the #+asn.1-features:cl-yacc

Thanks, fixed as r767.

>
>
> [I'm still against using the bizarre little code I wrote to convert  
> the
> grammar and recommend sticking with just one parser]
>
> (
>
> ;;; Lalr Parser
> ;;; Note: choose only ONE between 'parsergen and 'yacc
>
> #+lispworks
> parsergen ; default on LispWorks
>
> #-lispworks
> yacc   ; (cl-yacc) default on other CL platform
>
> ^^^^ should be cl-yacc?
>
> [...]
>

[lisp-snmp] Misc fiddles for new OID types

[John F. <jf...@ms...>]

Dear Chun Tian,

Here are the misc helpers I put in to deal with the new OID types.

Maybe you would like to fix them up or correct them?

ASN.1 patch

Index: runtime/object-id.lisp
===================================================================
--- runtime/object-id.lisp	(revision 766)
+++ runtime/object-id.lisp	(working copy)
@@ -113,8 +113,9 @@
   (slot-boundp oid 'parent))
 
 (defun oid-module-p (oid)
-  (declare (type object-id oid))
-  (slot-boundp oid 'module))
+  (typecase oid
+    (object-id
+     (slot-boundp oid 'module))))
 
 (defun oid-syntax-p (oid)
   (declare (type object-id oid))
@@ -124,17 +125,29 @@
   (declare (type object-id oid))
   (symbol-name (oid-name oid)))
 
-(defun oid-name-list (oid)
-  (declare (type object-id oid))
+(defgeneric oid-name-list (oid))
+(defmethod oid-name-list ((oid object-id))
   (labels ((iter (o acc)
              (if (slot-boundp o 'parent)
-                 (iter (oid-parent o) (cons (if (slot-boundp o 'name)
-                                                (oid-name-string o)
-                                              (oid-value o))
-                                            acc))
-               acc)))
+                 (iter (oid-parent o) 
+		       (cons (if (slot-boundp o 'name)
+				 (oid-name-string o)
+				 (oid-value o))
+			     acc))
+		 acc)))
     (iter oid nil)))
 
+(defun oid-number-sublist (oid)
+  (nthcdr 
+   (if (oid-parent-p oid)
+       (oid-length (oid-parent oid))
+       0)
+   (oid-number-list oid)))
+
+(defmethod oid-name-list ((oid simple-oid))
+  (append (when (oid-parent-p oid) (oid-name-list (oid-parent oid))) 
+	  (oid-number-sublist oid)))
+
 (defmethod plain-value ((object simple-oid) &key default)
   (declare (ignore default))
   (oid-number-list object))
Index: runtime/oid-walk.lisp
===================================================================
--- runtime/oid-walk.lisp	(revision 766)
+++ runtime/oid-walk.lisp	(working copy)
@@ -85,10 +85,25 @@
        (zerop (hash-table-count (oid-children oid)))))
 
 (defun oid-trunk-p (oid)
-  (declare (type object-id oid))
   (and (slot-boundp oid 'name)
        (plusp (hash-table-count (oid-children oid)))))
 
+(defun oid-find-base (oid)
+  (labels ((iter (o acc)
+	     (typecase o
+	       (null (values nil acc))
+	       (object-id
+		(if (slot-boundp o 'name)
+		    (values o acc)
+		    (let ((p (oid-parent o))
+			  (v (oid-value o)))
+		      (iter p (cons v acc)))))
+	       (simple-oid
+		(iter (when (oid-parent-p o) (oid-parent o)) 
+		      (append (oid-number-sublist o) acc))))))
+    (iter oid nil)))
+
+
 (defun oid-find-leaf (oid)
   "Find the leaf node in a oid's all parents"
   (declare (type object-id oid))



PS. Thanks for fixing runtime/ipaddress to not barf on length != 4.

BTW 
(loop for i from 1 upto length
    collect (read-byte stream)) 

is the same as

(loop repeat length
        collect (read-byte stream)) 

[lisp-snmp] ASN.1 dynamic loading

[John F. <jf...@ms...>]

Dear Chun Tian,

I have merged your current ASN.1 and SNMP so that we are up to
date. 

You improved the mechanism I made for dynamically loading ASN.1
files. What is your approved way of doing it now? I stupidly cannot find
the right function.

By the way, in your features lisp-expr, the name for yacc should be
cl-yacc, as that is used everywhere in the #+asn.1-features:cl-yacc

[I'm still against using the bizarre little code I wrote to convert the
grammar and recommend sticking with just one parser]

(

 ;;; Lalr Parser
 ;;; Note: choose only ONE between 'parsergen and 'yacc

 #+lispworks
 parsergen ; default on LispWorks

 #-lispworks
 yacc   ; (cl-yacc) default on other CL platform

^^^^ should be cl-yacc?

[...]

Re: [lisp-snmp] Changing the representation of OIDs

[Chun T. (binghe) <bin...@gm...>]

在 2009-5-1，16:29， John Fremlin 写道：

> Dear Chun Tian,
>
> Thanks again, I will test your new patch after the Golden Week
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>> A SIMPLE-OID is a special ASN.1 object ID which have less slots  
>> than a
>> normal OBJECT-ID:
>>
>> (defclass base-oid (asn.1-type)
>>  ((parent      :type object-id
>>                :reader oid-parent
>>                :initarg :parent)))
>>
>> (defclass simple-oid (base-oid)
>>  ((values      :type list
>>                :reader oid-number-list
>>                :initarg :values)
>>   (length      :type fixnum
>>                :reader oid-length))
>
> Seeing as the complex OID also has these slots, why not include them  
> in
> base-oid?

SIMPLE-OID is a super class of OBJECT-ID :)

>
>
> With a vector representation you only need one slot as the length is  
> O(1)
>
>>  (:documentation "A simple OID implementation"))
>>
>> I still use a LIST to hold the oid number list because this can best
>> fit most of exist code.
>
> Would you accept a patch to convert it over to vectors

Yes, I would like to see this patch. But I hope the change be as small  
as possible.

>
>
> There are many places where we might better use vectors than lists
> (e.g. ber-decode-type which unfortunately does plenty of unnecessary
> consing).
>
> [...]
>
>> You can see, for (OID "sysDescr.0") and (OID '(1 3 6 1 2 1 1 1 0)) ,
>> it returns a SIMPLE-OID instance instead of OBJECT-ID now. The rule
>> is: if the object id ends with unnamed number lists, a SIMPLE-OID  
>> will
>> return instead. I think lots of work can be saved here.
>
> Thanks, I this is we have taken out the top memory wasters out of our
> program, which might be enough to make the customer a bit happier.
>
> Mr Kuroda was threatening to try to get me to specify the amount of
> memory required by the program -- perhaps that will be a bit simpler  
> now
> (but not much ;-).
>
> [...]
>

Re: [lisp-snmp] Changing the representation of OIDs

[John F. <jf...@ms...>]

Dear Chun Tian,

Thanks again, I will test your new patch after the Golden Week

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
> A SIMPLE-OID is a special ASN.1 object ID which have less slots than a
> normal OBJECT-ID:
>
> (defclass base-oid (asn.1-type)
>   ((parent      :type object-id
>                 :reader oid-parent
>                 :initarg :parent)))
>
> (defclass simple-oid (base-oid)
>   ((values      :type list
>                 :reader oid-number-list
>                 :initarg :values)
>    (length      :type fixnum
>                 :reader oid-length))

Seeing as the complex OID also has these slots, why not include them in
base-oid?

With a vector representation you only need one slot as the length is O(1)

>   (:documentation "A simple OID implementation"))
>
> I still use a LIST to hold the oid number list because this can best
> fit most of exist code.

Would you accept a patch to convert it over to vectors

There are many places where we might better use vectors than lists
(e.g. ber-decode-type which unfortunately does plenty of unnecessary
consing).

[...]

> You can see, for (OID "sysDescr.0") and (OID '(1 3 6 1 2 1 1 1 0)) ,
> it returns a SIMPLE-OID instance instead of OBJECT-ID now. The rule
> is: if the object id ends with unnamed number lists, a SIMPLE-OID will
> return instead. I think lots of work can be saved here.

Thanks, I this is we have taken out the top memory wasters out of our
program, which might be enough to make the customer a bit happier.

Mr Kuroda was threatening to try to get me to specify the amount of
memory required by the program -- perhaps that will be a bit simpler now
(but not much ;-).
 
[...]

Re: [lisp-snmp] Changing the representation of OIDs

[Chun T. (binghe) <bin...@gm...>]

Hi, John

Thank you for your suggestion and I took some time to finish it. Now ...

May I introduce a new feature on ASN.1 trunk: the SIMPLE-OID class.

A SIMPLE-OID is a special ASN.1 object ID which have less slots than a  
normal OBJECT-ID:

(defclass base-oid (asn.1-type)
   ((parent      :type object-id
                 :reader oid-parent
                 :initarg :parent)))

(defclass simple-oid (base-oid)
   ((values      :type list
                 :reader oid-number-list
                 :initarg :values)
    (length      :type fixnum
                 :reader oid-length))
   (:documentation "A simple OID implementation"))

I still use a LIST to hold the oid number list because this can best  
fit most of exist code.

A SIMPLE-OID instance can be BER encoded the same with normal OBJECT- 
ID instances. And below is some new feature when OID function is called:

ASN.1 21 > (oid "sysDescr")
#<OBJECT-ID SNMPv2-MIB::sysDescr (1) [0]>

ASN.1 22 > (oid "sysDescr.0")
#<SIMPLE-OID SNMPv2-MIB::sysDescr.0>

ASN.1 23 > (oid-number-list (oid "sysDescr"))
(1 3 6 1 2 1 1 1)

ASN.1 24 > (oid '(1 3 6 1 2 1 1 1))
#<OBJECT-ID SNMPv2-MIB::sysDescr (1) [0]>

ASN.1 25 > (oid '(1 3 6 1 2 1 1 1 0))
#<SIMPLE-OID SNMPv2-MIB::sysDescr.0>

You can see, for (OID "sysDescr.0") and (OID '(1 3 6 1 2 1 1 1 0)) ,  
it returns a SIMPLE-OID instance instead of OBJECT-ID now. The rule  
is: if the object id ends with unnamed number lists, a SIMPLE-OID will  
return instead. I think lots of work can be saved here.

Now a full SNMP-WALK on (OID "system") may look like this:

CL-USER 19 > (snmp:snmp-walk "localhost" "system")
((#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysDescr.0>
   "Darwin binghe-pro.local 9.6.2 Darwin Kernel Version 9.6.2: Tue Jan  
13 20:42:22 PST 2009; root:xnu-1228.9.80~1/RELEASE_I386 i386")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysObjectID.0> #<ASN.1:OBJECT-ID NET- 
SNMP-TC::unknown (255) [0]>)
  (#<ASN.1:OBJECT-ID DISMAN-EVENT-MIB::sysUpTimeInstance (0) [0]>  
#<ASN.1:TIMETICKS (43192102) 119:58:41.02>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysContact.0> "Administrator <pos...@ex... 
 >")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysName.0> "binghe-pro.local")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysLocation.0> "Right here, right  
now.")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysServices.0> 76)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORLastChange.0> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.1> #<ASN.1:OBJECT-ID SNMP- 
FRAMEWORK-MIB::snmpFrameworkMIBCompliance (1) [0]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.2> #<ASN.1:OBJECT-ID SNMP- 
MPD-MIB::snmpMPDCompliance (1) [0]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.3> #<ASN.1:OBJECT-ID SNMP- 
USER-BASED-SM-MIB::usmMIBCompliance (1) [0]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.4> #<ASN.1:OBJECT-ID SNMPv2- 
MIB::snmpMIB (1) [2]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.5> #<ASN.1:OBJECT-ID TCP- 
MIB::tcpMIB (49) [1]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.6> #<ASN.1:OBJECT-ID IP- 
MIB::ip (4) [38]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.7> #<ASN.1:OBJECT-ID UDP- 
MIB::udpMIB (50) [1]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORID.8> #<ASN.1:OBJECT-ID SNMP- 
VIEW-BASED-ACM-MIB::vacmBasicGroup (1) [0]>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.1> "The SNMP Management  
Architecture MIB.")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.2> "The MIB for Message  
Processing and Dispatching.")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.3> "The management  
information definitions for the SNMP User-based Security Model.")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.4> "The MIB module for  
SNMPv2 entities")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.5> "The MIB module for  
managing TCP implementations")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.6> "The MIB module for  
managing IP and ICMP implementations")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.7> "The MIB module for  
managing UDP implementations")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORDescr.8> "View-based Access  
Control Model for SNMP.")
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.1> #<ASN.1:TIMETICKS  
(65) 0:00:00.65>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.2> #<ASN.1:TIMETICKS  
(65) 0:00:00.65>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.3> #<ASN.1:TIMETICKS  
(65) 0:00:00.65>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.4> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.5> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.6> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.7> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>)
  (#<ASN.1:SIMPLE-OID SNMPv2-MIB::sysORUpTime.8> #<ASN.1:TIMETICKS  
(66) 0:00:00.66>))

You can see, as a "key", almost all OIDs are returned as SIMPLE-OID,  
but those OIDs as a "value", we still use original OBJECT-ID.

All my work for this feature has been commited on cl-net-snmp SVN (ASN. 
1 trunk, r762), you can find the diff here:

http://cl-net-snmp.svn.sourceforge.net/viewvc/cl-net-snmp?view=rev&revision=762

Due to its big change, this work won't appear in ASN.1 4.x

Hope you like it, and more comments are welcome.

Regards,

Chun Tian (binghe)

在 2009-5-1，14:08， John Fremlin 写道：

> Dear Chin Tian,
>
> Thanks for your rapid response.
>
> We seem to have reduced the memory usage significantly. Hopefully we
> will be demonstrably more efficient now too!
>
> However, according to SBCL, still the largest amount of consing is  
> going
> on in the object-id code.
>
> The problem is that there are many OIDs like this
>
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.7.1.1.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.7.1.2.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.7.1.3.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.7.1.4.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.7.1.5.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.1.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.2.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.3.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.4.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.5.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.6.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.7.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.8.127 = 255
> localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout. 
> 0.11.1.3.6.1.2.1.25.3.8.1.9.127 = 255
>
> Each step along the OID after the last known one
> (NET-SNMP-AGENT-MIB::nsModuleTimeout) causes a new instance of class
> object-id to be created.
>
> This is quite unnecessary.
>
> As I mentioned before, I rather like the idea of being able to pass
> around simple-vectors of integers as OIDs. Is there a limit to the  
> size
> of the integers in an OID's number list? (I.e. could we use simple
> arrays of fixnum.)
>
> This would mean that an OID could be of two kinds, either an actual
> instance of object-id or a vector.
>
> Alternatively we could make a defclass like this (rough sketch)
>
> (defclass object-id-unknown ()
>          ((parent) ; pointer to parent full oid
>           (values :type 'simple-vector)))
>
> At the same time I'd like to turn the number-list into a vector as a
> list is rather wasteful.
>
> What do you think?
>
> Would you accept a patch for either of the two approaches?
>
> [...]
>

[lisp-snmp] Changing the representation of OIDs

[John F. <jf...@ms...>]

Dear Chin Tian,

Thanks for your rapid response.

We seem to have reduced the memory usage significantly. Hopefully we
will be demonstrably more efficient now too!

However, according to SBCL, still the largest amount of consing is going
on in the object-id code.

The problem is that there are many OIDs like this

localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.7.1.1.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.7.1.2.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.7.1.3.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.7.1.4.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.7.1.5.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.1.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.2.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.3.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.4.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.5.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.6.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.7.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.8.127 = 255
localhost -- NET-SNMP-AGENT-MIB::nsModuleTimeout.0.11.1.3.6.1.2.1.25.3.8.1.9.127 = 255

Each step along the OID after the last known one
(NET-SNMP-AGENT-MIB::nsModuleTimeout) causes a new instance of class
object-id to be created.

This is quite unnecessary.

As I mentioned before, I rather like the idea of being able to pass
around simple-vectors of integers as OIDs. Is there a limit to the size
of the integers in an OID's number list? (I.e. could we use simple
arrays of fixnum.)

This would mean that an OID could be of two kinds, either an actual
instance of object-id or a vector.

Alternatively we could make a defclass like this (rough sketch)

(defclass object-id-unknown ()
          ((parent) ; pointer to parent full oid
           (values :type 'simple-vector)))

At the same time I'd like to turn the number-list into a vector as a
list is rather wasteful.

What do you think?

Would you accept a patch for either of the two approaches?

[...]

Re: [lisp-snmp] Memory usage of ASN.1 library

[Chun T. (binghe) <bin...@gm...>]

Your version equal to this one: (and it works)

(defun oid-< (oid-1 oid-2)
  "test if oid-1 is oid-2's child"
  (unless (<= (oid-length oid-1) (oid-length oid-2))
    (loop for x-1 in (oid-number-list oid-1)
          for x-2 in (oid-number-list oid-2)
          always (= x-1 x-2))))

I'm so stupid in this function ... :)

Thank you, I'll commit it as ASN.1 4.16 and into trunk.

--binghe

在 2009-5-1，13:47， John Fremlin 写道：

> "Chun Tian (binghe)" <bin...@gm...> writes:
>
>> Great!
>>
>> And, please use my more updated version of ASN.1:OID-< as a work  
>> base:
>
> Sorry, I already made a new version, is it okay?
>
> (defun oid-< (oid-1 oid-2)
>  "test if oid-1 is oid-2's child"
>  (let ((o-1 (oid-number-list oid-1))
>        (o-2 (oid-number-list oid-2))
>        (o-1-len (oid-length oid-1))
>        (o-2-len (oid-length oid-2)))
>    (unless (<= o-1-len o-2-len)
>      (loop for x-1 in o-1
> 	    for x-2 in o-2
> 	    always (equal x-1 x-2)))))
>
> In the old version with O(n) lengths this is suboptimal but in the new
> version with O(1) lengths we are better off with checking the length  
> first
>
>>
>> (defun oid-< (oid-1 oid-2)
>>  "test if oid-1 is oid-2's child"
>>  (let ((o-1-len (oid-length oid-1))
>>        (o-2-len (oid-length oid-2)))
>>    (if (<= o-1-len o-2-len) nil
>>      (let ((o-1 (reverse (oid-number-list oid-1)))
>>            (o-2 (reverse (oid-number-list oid-2))))
>>        (equal o-2 (nthcdr (- o-1-len o-2-len) o-1))))))
>>
>> I believe it's more sufficient than before.
>
> if test nil else === unless test else
>
> [...]
>

Re: [lisp-snmp] Memory usage of ASN.1 library

[John F. <jf...@ms...>]

"Chun Tian (binghe)" <bin...@gm...> writes:

> Great!
>
> And, please use my more updated version of ASN.1:OID-< as a work base:

Sorry, I already made a new version, is it okay?

(defun oid-< (oid-1 oid-2)
  "test if oid-1 is oid-2's child"
  (let ((o-1 (oid-number-list oid-1))
        (o-2 (oid-number-list oid-2))
        (o-1-len (oid-length oid-1))
        (o-2-len (oid-length oid-2)))
    (unless (<= o-1-len o-2-len)
      (loop for x-1 in o-1
	    for x-2 in o-2
	    always (equal x-1 x-2)))))

In the old version with O(n) lengths this is suboptimal but in the new
version with O(1) lengths we are better off with checking the length first

>
> (defun oid-< (oid-1 oid-2)
>   "test if oid-1 is oid-2's child"
>   (let ((o-1-len (oid-length oid-1))
>         (o-2-len (oid-length oid-2)))
>     (if (<= o-1-len o-2-len) nil
>       (let ((o-1 (reverse (oid-number-list oid-1)))
>             (o-2 (reverse (oid-number-list oid-2))))
>         (equal o-2 (nthcdr (- o-1-len o-2-len) o-1))))))
>
> I believe it's more sufficient than before.

if test nil else === unless test else

[...]