Re: [lisp-snmp] Faster ASN encode (reducing memory pressure)
Brought to you by:
binghe
Menu
▾
▴
Re: [lisp-snmp] Faster ASN encode (reducing memory pressure)
|
From: Chun T. (binghe) <bin...@gm...> - 2009-05-23 08:26:07
|
[1] https://cl-net-snmp.svn.sourceforge.net/svnroot/cl-net-snmp/asn.1/branches/faster-encode > Hi, John > > Last night I did some basic test, and it's obviously that your patch > works just fine and didn't change the interface between ASN.1 and SNMP > package. It's a excellent work which towards a highly optimized ASN.1/ > SNMP package, and thank you again for look into my work deeply. > > I believe, sometimes to make Lisp code running fast, something have to > be gave up, like code beauty and express, and only those greatest Lisp > projects can keep both the efficiency and beauty at the same time (for > example, CL-HTTP). And as you already told, "the best way forward > would be to restructure the whole thing ...", I think you're totally > right. > > So, I've made some decision: > > * Create a branch [1] for holding all your "fast encode" work, > * Do a "restructure" work on trunk in next few weeks to try to achieve > both speed and art (by learning your work mainly) > > Please let me know your comments. > > Best regards, > > Chun Tian (binghe) > > >> Dear Chun Tian, >> >> Our primary platform is Allegro 8.1, and its garbage collector >> struggles >> awfully when garbage is generated. >> >> These changes take our time on one benchmark down from more than >> three >> minutes to about two minutes, and only about 20% slower than SBCL. >> >> Therefore I have modified the ASN encode to reduce the garbage >> generated. I used a variety of techniques to do it (compiler macro >> constant folding, value caching, etc.) and modified quite a lot of >> the >> code, so I guess you should look it over and give it a good >> refactoring. >> >> In my opinion, the best way forward would be to restructure the whole >> thing so that the encode functions have a nice scratch buffer and >> don't >> need to keep reallocating and copying memory all the time. >> >> Simply creating the SNMP message objects takes a lot of time and >> could >> be avoided as far as I can see. >> >> However, the decode is now the juicier time waster. The weird >> transitions between stream, lists and vectors make us suffer quite a >> lot. >> >> Again, why not stick with a byte buffer. >> >> <asn.1-faster-encode.patch>Index: pdu.lisp >> =================================================================== >> --- pdu.lisp (revision 769) >> +++ pdu.lisp (working copy) >> @@ -108,29 +108,31 @@ >> (format stream "(~D (~D ~D)) ~A" >> request-id non-repeaters max-repetitions variable- >> bindings)))) >> >> -(defmethod ber-encode-pdu ((value pdu) tag) >> - (declare (type fixnum tag)) >> - (with-slots (request-id error-status error-index variable- >> bindings) value >> - (let ((sub-encode (ber-encode-list (list request-id >> - error-status >> - error-index >> - variable-bindings)))) >> - (concatenate 'vector >> - (ber-encode-type 2 1 tag) >> - (ber-encode-length (length sub-encode)) >> - sub-encode)))) >> +(defmacro do-ber-encode-pdu (value tag) >> + (flet ((encode (vars) >> + `(with-slots ,vars ,value >> + (let ,(loop for v in vars collect `(,v (ber-encode ,v))) >> + (let ((length (+ ,@(loop for v in vars collect `(length ,v))))) >> + (asn.1::ber-array-from-seqs >> + (ber-encode-type 2 1 ,tag) >> + (ber-encode-length length) >> + ,@vars)))))) >> + (encode '(request-id >> + error-status >> + error-index >> + variable-bindings)))) >> >> (defmethod ber-encode ((value get-request-pdu)) >> - (ber-encode-pdu value +get-request-pdu+)) >> + (do-ber-encode-pdu value +get-request-pdu+)) >> >> (defmethod ber-encode ((value get-next-request-pdu)) >> - (ber-encode-pdu value +get-next-request-pdu+)) >> + (do-ber-encode-pdu value +get-next-request-pdu+)) >> >> (defmethod ber-encode ((value response-pdu)) >> - (ber-encode-pdu value +response-pdu+)) >> + (do-ber-encode-pdu value +response-pdu+)) >> >> (defmethod ber-encode ((value set-request-pdu)) >> - (ber-encode-pdu value +set-request-pdu+)) >> + (do-ber-encode-pdu value +set-request-pdu+)) >> >> (defmethod ber-encode ((value trap-pdu)) >> (with-slots (enterprise agent-addr generic-trap specific-trap >> timestamp >> @@ -158,13 +160,13 @@ >> sub-encode)))) >> >> (defmethod ber-encode ((value inform-request-pdu)) >> - (ber-encode-pdu value +inform-request-pdu+)) >> + (do-ber-encode-pdu value +inform-request-pdu+)) >> >> (defmethod ber-encode ((value snmpv2-trap-pdu)) >> - (ber-encode-pdu value +snmpv2-trap-pdu+)) >> + (do-ber-encode-pdu value +snmpv2-trap-pdu+)) >> >> (defmethod ber-encode ((value report-pdu)) >> - (ber-encode-pdu value +report-pdu+)) >> + (do-ber-encode-pdu value +report-pdu+)) >> >> (defun ber-decode-pdu (stream length class) >> (declare (type stream stream) >> ------------------------------------------------------------------------------ >> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT >> is a gathering of tech-side developers & brand creativity >> professionals. Meet >> the minds behind Google Creative Lab, Visual Complexity, >> Processing, & >> iPhoneDevCamp asthey present alongside digital heavyweights like >> Barbarian >> Group, R/GA, & Big Spaceship. http://www.creativitycat.com >> _______________________________________________ >> cl-net-snmp-general mailing list >> cl-...@li... >> https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general > > > ------------------------------------------------------------------------------ > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT > is a gathering of tech-side developers & brand creativity > professionals. Meet > the minds behind Google Creative Lab, Visual Complexity, Processing, & > iPhoneDevCamp asthey present alongside digital heavyweights like > Barbarian > Group, R/GA, & Big Spaceship. http://www.creativitycat.com > _______________________________________________ > cl-net-snmp-general mailing list > cl-...@li... > https://lists.sourceforge.net/lists/listinfo/cl-net-snmp-general |
