cl-net-snmp-general — cl-net-snmp general

Re: [lisp-snmp] Memory usage of ASN.1 library

[Chun T. (binghe) <bin...@gm...>]

Great!

And, please use my more updated version of ASN.1:OID-< as a work base:

(defun oid-< (oid-1 oid-2)
   "test if oid-1 is oid-2's child"
   (let ((o-1-len (oid-length oid-1))
         (o-2-len (oid-length oid-2)))
     (if (<= o-1-len o-2-len) nil
       (let ((o-1 (reverse (oid-number-list oid-1)))
             (o-2 (reverse (oid-number-list oid-2))))
         (equal o-2 (nthcdr (- o-1-len o-2-len) o-1))))))

I believe it's more sufficient than before.

在 2009-5-1，13:41， John Fremlin 写道：

> "Chun Tian (binghe)" <bin...@gm...> writes:
>
>> I believe the bug involved in my last patch is in function ASN.1:OID-
>> <, below is a updated version:
>
> Thanks for finding that, I was confused about where it was happening.
>
> I will have a look at making a better oid-<
>
> Thanks again
>
> [...]
>

Re: [lisp-snmp] Memory usage of ASN.1 library

[John F. <jf...@ms...>]

"Chun Tian (binghe)" <bin...@gm...> writes:

> I believe the bug involved in my last patch is in function ASN.1:OID-
> <, below is a updated version:

Thanks for finding that, I was confused about where it was happening.

I will have a look at making a better oid-<

Thanks again

[...]

Re: [lisp-snmp] Memory usage of ASN.1 library

[Chun T. (binghe) <bin...@gm...>]

Hi, John

I believe the bug involved in my last patch is in function ASN.1:OID- 
<, below is a updated version:

(defun oid-< (oid-1 oid-2)
   "test if oid-1 is oid-2's child"
   (let ((o-1 (reverse (oid-number-list oid-1)))
         (o-2 (reverse (oid-number-list oid-2)))
         (o-1-len (oid-length oid-1))
         (o-2-len (oid-length oid-2)))
     (if (<= o-1-len o-2-len) nil
       (equal o-2 (nthcdr (- o-1-len o-2-len) o-1)))))

There're two REVERSE which were NREVERSE. This cause the cached OID- 
NUMBER-LIST list been incorrectly destroyed. By using above new  
version of OID-<, I think SNNP-WALK works now.

Would you help me finding a more optimized version of OID-< which  
didn't do any consing? Maybe you can do better than me here by using  
LOOP?

Regards,

Chun Tian (binghe)

在 2009-5-1，09:09， John Fremlin 写道：

> Dear Chun Tian,
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>> First, I should say sorry for my delay on the work of "runtime MIB
>> loading". I've been hired as a commercial Lisp programmer since one
>> month ago, and I don't have much time working on cl-net-snmp these
>> days. I hope I can have some time to finish it in next month.  
>> However,
>> you (and MSI) are still my big "client" :)
>
> Good luck and have fun dealing with your bosses in Bangalore.
>
> [...]
>
>> For optimize, I want to add two more slots in class OBJECT-ID: VALUES
>> (to hold the cache of "oid number list" and "oid length"). This is
>> reasonable because a OBJECT-ID instance won't change their "oid  
>> number
>> list" once created, so calculate it just once will work.
>
> Yes, as you saw from my email this was more or less my suggestion  
> and I
> had already tried it.
>
> The trouble is that for some reason it causes SNMP walks to no longer
> work. An object ID is created, its number-list is taken, then it is  
> modified.
>
>> (snmp:snmp-walk "127.0.0.1" ".1")
> #<SNMP::SMI NO-SUCH-OBJECT (0)>
>
> I guess there is no hope but to actually fix the problem and delve  
> even
> further into ASN.1.
>
> I was hoping that you might already have a plan to start using simple
> vectors of integers as OIDs. Using lists/instances is more
> complicated and wasteful.
>
> PS. Why do you hate loop and iterate so much?

Re: [lisp-snmp] Memory usage of ASN.1 library

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
> First, I should say sorry for my delay on the work of "runtime MIB
> loading". I've been hired as a commercial Lisp programmer since one
> month ago, and I don't have much time working on cl-net-snmp these
> days. I hope I can have some time to finish it in next month. However,
> you (and MSI) are still my big "client" :)

Good luck and have fun dealing with your bosses in Bangalore.

[...]

> For optimize, I want to add two more slots in class OBJECT-ID: VALUES
> (to hold the cache of "oid number list" and "oid length"). This is
> reasonable because a OBJECT-ID instance won't change their "oid number
> list" once created, so calculate it just once will work.

Yes, as you saw from my email this was more or less my suggestion and I
had already tried it.

The trouble is that for some reason it causes SNMP walks to no longer
work. An object ID is created, its number-list is taken, then it is modified.

> (snmp:snmp-walk "127.0.0.1" ".1")
#<SNMP::SMI NO-SUCH-OBJECT (0)>

I guess there is no hope but to actually fix the problem and delve even
further into ASN.1.

I was hoping that you might already have a plan to start using simple
vectors of integers as OIDs. Using lists/instances is more
complicated and wasteful.

PS. Why do you hate loop and iterate so much?

Re: [lisp-snmp] Memory usage of ASN.1 library

[Chun T. (binghe) <bin...@gm...>]

Hi, John

First, I should say sorry for my delay on the work of "runtime MIB  
loading". I've been hired as a commercial Lisp programmer since one  
month ago, and I don't have much time working on cl-net-snmp these  
days. I hope I can have some time to finish it in next month. However,  
you (and MSI) are still my big "client" :)

I understand heavily calls to OID-NUMBER-LIST will cause consing too  
much.

For optimize, I want to add two more slots in class OBJECT-ID: VALUES  
(to hold the cache of "oid number list" and "oid length"). This is  
reasonable because a OBJECT-ID instance won't change their "oid number  
list" once created, so calculate it just once will work.

Actual test shows that, by using this cache, repeated calls to OID- 
NUMBER-LIST on the "same" oid instance won't cons any more after the  
first call:

ASN.1 14 > (time (oid-number-list (oid "system")))
Timing the evaluation of (OID-NUMBER-LIST (OID "system"))

User time    =        0.000
System time  =        0.000
Elapsed time =        0.000
Allocation   = 900 bytes
0 Page faults
(1 3 6 1 2 1 1)

ASN.1 15 > (time (oid-number-list (oid "system")))
Timing the evaluation of (OID-NUMBER-LIST (OID "system"))

User time    =        0.000
System time  =        0.000
Elapsed time =        0.000
Allocation   = 24 bytes
0 Page faults
(1 3 6 1 2 1 1)

Please test the new ASN.1 patch (4-16.lisp) in attach, it's a loadable  
patch: you could load it (remember compile it first) into your running  
lisp image even AFTER all OID instances in your application are  
created, and you'll get the faster version of OID-NUMBER-LIST then.

I hope that helps.

Regards,

Chun Tian (binghe)

[lisp-snmp] Memory usage of ASN.1 library

[John F. <jf...@ms...>]

Dear Chun Tian,

We are struggling with the memory usage of our application.

It seems that our garbage collector is rather tiresome (we are using
Allegro CL 8.1) and struggles when large amounts of garbage are
generated.

In the ASN.1 parsing, according to a sensible profile (from SBCL, the
Allegro space profiler doesn't give useful information), one of the
biggest generators of garbage in our application is

784,921,696 |    72,584 |   0.000000 | ASN.1:OID-NUMBER-LIST

(about 50% of the consing).

You mention in the source code that this method could easily be improved
with a hash-table.

How about storing a number-list slot in each object-id instance?

Any advice or ideas would be helpful.

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

"Chun Tian (binghe)" <bin...@gm...> writes:
> On 2009-2-24, at 17:11, John Fremlin wrote:
>
>> Dear Chun Tian,
>>
>> Most of the short MIB package names are fine. If they are validated by
>> you at compile time that is doubly fine.
>>
>> The problem is in loading in user defined package names at runtime. It
>> is important that we don't cause name collisions if the user for some
>> reason wants to load in an MIB which might, for example, have the same
>> name as our application (which is already a Lisp package).
>>
>> Perhaps having an option to avoid polluting the global package
>> namespace
>> is a good idea. I guess we should rewrite the runtime loading in of
>> MIB
>> files anyway (unless you've already done that?), but after asking for
>> this feature the customer has shown absolutely no interest in it
>> whatsoever so I guess it isn't important.
>
>
> The "runtime loading in of MIB files" feature haven't been
> supported. And I do have a plan to implemented it without EVAL,
> COMPILE and any  other things which make delivery hard. No matter
> whether you and your  customer need it, this is a standard feature and
> I'll finish it in a  week.

Thanks! That's great. As far as I can see there was no need for eval at
all and this would be a tidy clean-up

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

Hi John,

On 2009-2-24, at 17:11, John Fremlin wrote:

> Dear Chun Tian,
>
> Most of the short MIB package names are fine. If they are validated by
> you at compile time that is doubly fine.
>
> The problem is in loading in user defined package names at runtime. It
> is important that we don't cause name collisions if the user for some
> reason wants to load in an MIB which might, for example, have the same
> name as our application (which is already a Lisp package).
>
> Perhaps having an option to avoid polluting the global package  
> namespace
> is a good idea. I guess we should rewrite the runtime loading in of  
> MIB
> files anyway (unless you've already done that?), but after asking for
> this feature the customer has shown absolutely no interest in it
> whatsoever so I guess it isn't important.


The "runtime loading in of MIB files" feature haven't been supported.  
And I do have a plan to implemented it without EVAL, COMPILE and any  
other things which make delivery hard. No matter whether you and your  
customer need it, this is a standard feature and I'll finish it in a  
week.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
> To respect your wishes and still keep the implementation of OID
> function simple, I has changed the behavior (short-package-name) into
> a compile-time option.
>
> If you update both ASN.1 and SNMP package to newest SVN trunk, you'll
> find file "features.lisp-expr" in both packages. There's
> one option in #p"ASN1:features.lisp-expr":
>
>  ;;; Short package name of MIB packages, disable for expert only.
>  short-package-name
>
> If you disable it (either prefix a #+ignre or comment it), and
> recompile the compiler part of ASN.1 packages, next time when you
> compile MIB files or just use (snmp:update-mib) to regenerate all
> MIBs, the new generated files will NOT have short nicknames at all.
>
> As before I don't think it's necessary to disable the short package
> names since it's quite convenient. But if you really don't like it, I
> hope you would like above solution.

Most of the short MIB package names are fine. If they are validated by
you at compile time that is doubly fine.

The problem is in loading in user defined package names at runtime. It
is important that we don't cause name collisions if the user for some
reason wants to load in an MIB which might, for example, have the same
name as our application (which is already a Lisp package).

Perhaps having an option to avoid polluting the global package namespace
is a good idea. I guess we should rewrite the runtime loading in of MIB
files anyway (unless you've already done that?), but after asking for
this feature the customer has shown absolutely no interest in it
whatsoever so I guess it isn't important.

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

Hi, John

To respect your wishes and still keep the implementation of OID  
function simple, I has changed the behavior (short-package-name) into  
a compile-time option.

If you update both ASN.1 and SNMP package to newest SVN trunk, you'll  
find file "features.lisp-expr" in both packages. There's
one option in #p"ASN1:features.lisp-expr":

  ;;; Short package name of MIB packages, disable for expert only.
  short-package-name

If you disable it (either prefix a #+ignre or comment it), and  
recompile the compiler part of ASN.1 packages, next time when you  
compile MIB files or just use (snmp:update-mib) to regenerate all  
MIBs, the new generated files will NOT have short nicknames at all.

As before I don't think it's necessary to disable the short package  
names since it's quite convenient. But if you really don't like it, I  
hope you would like above solution.

Regards,

Chun Tian (binghe)

On 2009-2-9, at 15:27, John Fremlin wrote:

> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>>> Will this work with the old function? I don't think so, because of
>>> the .
>>
>> No, the #\. is NOT the key, and (oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2-
>> MIB::system") do work with the old (current) OID function, you can  
>> try
>> it yourself. CL readtable is a very good builtin lexer.
>
> Sorry for the confusion.
>
> [...]
>> Again, #\. is not the key, it is how to add "ASN.1/" to every  
>> "package
>> name" in OID string.
>
> Maybe using the reader here is asking for a bit too much ;-)
>
> What are the OID sequences you want to parse?
>
> package::name
> name
> .name0.name1.name2
> .1.2.4.5.6
>
> This is quite easy to do without the Lisp reader . . .
>
> [...]
>

[lisp-snmp] cdr-assoc: alist-get and ralist-get

[John F. <jf...@ms...>]

Chun Tian (binghe) <bin...@gm...> writes:
> On 2009-2-7, at 00:15, Chun Tian (binghe) wrote:
[...]
> I've fixed this bug. I used a very dirty way to implement LispWorks'
> CDR-ASSOC ...

Here is an implementation of cdr-assoc (called alist-get) and also
cdr-rassoc (called ralist-get). It allows you to do something like (setf
(cdr (assoc ...)  ...) ...) but create the key and value if key does not
already exist.

It is from http://common-lisp.net/project/cl-irregsexp/

I'm trying to get it into Alexandria . . . does it look okay?

(macrolet ((define-alist-get (name get-pair get-value-from-pair add)
	     `(progn
		(declaim (inline ,name))
		(defun ,name (alist key &key (test 'eql))
		  (let ((pair (,get-pair key alist :test test)))
		    (values (,get-value-from-pair pair) pair)))
		(define-setf-expander ,name (place key &key (test ''eql)
					     &environment env)
		  (multiple-value-bind (dummies vals newvals setter getter)
		      (get-setf-expansion place env)
		    (when (cdr newvals)
		      (error "~A cannot store multiple values in one place" ',name))
		    (with-unique-names (store key-val test-val alist found)
		      (values 
		       `(,@dummies ,key-val ,test-val)
		       `(,@vals ,key ,test)
		       (list store)
		       `(let ((,alist ,getter))
			  (let ((,found (,',get-pair ,key-val ,alist :test ,test-val)))
			    (cond (,found 
				   (setf (,',get-value-from-pair ,found) ,store))
				  (t
				   (let ,newvals
				     (setf ,(first newvals) (,',add ,key ,store ,alist))
				     ,setter)))
			    ,store))
		       `(,',name ,getter ,key))))))))
  (define-alist-get alist-get assoc cdr acons)
  (define-alist-get ralist-get rassoc car racons))

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
>> Will this work with the old function? I don't think so, because of
>> the .
>
> No, the #\. is NOT the key, and (oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2-
> MIB::system") do work with the old (current) OID function, you can try
> it yourself. CL readtable is a very good builtin lexer.

Sorry for the confusion.

[...]
> Again, #\. is not the key, it is how to add "ASN.1/" to every "package
> name" in OID string.

Maybe using the reader here is asking for a bit too much ;-)

What are the OID sequences you want to parse?

package::name
name
.name0.name1.name2
.1.2.4.5.6

This is quite easy to do without the Lisp reader . . .

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

On 2009-2-9, at 15:03, John Fremlin wrote:
>>>>
>>>> ASN.1 10 > (oid "SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
>>>> #<OBJECT-ID SNMPv2-MIB::system (1) [9]>
>>>
>>> This syntax can easily be supported as follows
>>>
>>> (defmethod oid-parse ((source string))
>>> (or
>>>  (with-input-from-string (s source)
>>>    (oid-parse s))
>>>  (with-input-from-string (s (concatenate 'string *asn.1-package-
>>> prefix* source))
>>>    (oid-parse s))))
>>
>> I'm afraid this function cannot work for (oid "SNMPv2-
>> SMI::mib-2.SNMPv2-MIB::system") correctly, you just change it into:
>
> Why do think that?
>
> Look at the function.
>
> First it tries the old way, then it tries with an ASN.1/ prefix.
>
>> (oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
>
> Will this work with the old function? I don't think so, because of
> the .

No, the #\. is NOT the key, and (oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2- 
MIB::system") do work with the old (current) OID function, you can try  
it yourself. CL readtable is a very good builtin lexer.

>
>
>>
>> How can you solve the second package name "SNMPv2-MIB" in that  
>> string?
>>
>>>
>>>
>>> However, *asn.1-package-prefix* cannot contain a "." because of your
>>> strange readtable. So it must be changed to something other than
>>> "ASN.1/", perhaps "ASN1/"
>>
>> The strange *oid-readtable* is my way to avoid the use of SPLIT-
>> SEQUENCE, it just let #\. be treat as a separator. However, this
>> method can cause unnecessary synbol intern in ASN.1 package, not the
>> best way to read a OID string.
>>
>> Give me more time to think out a better solution please.
>
> Cool
>
>>
>>>
>>>> Well, contact me if you really found your customers use such a MIB
>>>> module name, I think we can talk about this issue later:)
>>>
>>> It's kind of strange that you talk about making your package perfect
>>> but
>>> have it so arranged that by loading in an MIB file one can overwrite
>>> your package itself . . .
>>>
>>> I guess it's not just C programmers who don't care about  
>>> security ;-)
>>
>> The *asn.1-package-prefix* was learnt from Common Lisp's CORBA
>> language mapping. I think you've persuaded me, but I cannot change it
>> until I can solve the OID reader syntax issue above.
>
> Yes, the "." problem is annoying but it can be avoided by using  
> "ASN1/"
> instead . . .

Again, #\. is not the key, it is how to add "ASN.1/" to every "package  
name" in OID string.

>
>
> Thanks for your time
>
> [...]
>

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

"Chun Tian (binghe)" <bin...@gm...> writes:

> On 2009-2-9, at 14:44, John Fremlin wrote:
>
>> Dear Chun Tian,
>>
>> "Chun Tian (binghe)" <bin...@gm...> writes:
>> [...]
>>>> To support the cool (oid ..) syntax you can easily add "asn.1/" to
>>>> the
>>>> start of the string processed by OID method.
>>>
>>> It's not so easy when you've found following syntax is also well
>>> supported:
>>>
>>> ASN.1 10 > (oid "SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
>>> #<OBJECT-ID SNMPv2-MIB::system (1) [9]>
>>
>> This syntax can easily be supported as follows
>>
>> (defmethod oid-parse ((source string))
>>  (or
>>   (with-input-from-string (s source)
>>     (oid-parse s))
>>   (with-input-from-string (s (concatenate 'string *asn.1-package-
>> prefix* source))
>>     (oid-parse s))))
>
> I'm afraid this function cannot work for (oid "SNMPv2-
> SMI::mib-2.SNMPv2-MIB::system") correctly, you just change it into:

Why do think that?

Look at the function.

First it tries the old way, then it tries with an ASN.1/ prefix.

> (oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2-MIB::system")

Will this work with the old function? I don't think so, because of
the .

>
> How can you solve the second package name "SNMPv2-MIB" in that string?
>
>>
>>
>> However, *asn.1-package-prefix* cannot contain a "." because of your
>> strange readtable. So it must be changed to something other than
>> "ASN.1/", perhaps "ASN1/"
>
> The strange *oid-readtable* is my way to avoid the use of SPLIT-
> SEQUENCE, it just let #\. be treat as a separator. However, this
> method can cause unnecessary synbol intern in ASN.1 package, not the
> best way to read a OID string.
>
> Give me more time to think out a better solution please.

Cool

>
>>
>>> Well, contact me if you really found your customers use such a MIB
>>> module name, I think we can talk about this issue later:)
>>
>> It's kind of strange that you talk about making your package perfect
>> but
>> have it so arranged that by loading in an MIB file one can overwrite
>> your package itself . . .
>>
>> I guess it's not just C programmers who don't care about security ;-)
>
> The *asn.1-package-prefix* was learnt from Common Lisp's CORBA
> language mapping. I think you've persuaded me, but I cannot change it
> until I can solve the OID reader syntax issue above.

Yes, the "." problem is annoying but it can be avoided by using "ASN1/"
instead . . .

Thanks for your time

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

On 2009-2-9, at 14:44, John Fremlin wrote:

> Dear Chun Tian,
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>>> To support the cool (oid ..) syntax you can easily add "asn.1/" to  
>>> the
>>> start of the string processed by OID method.
>>
>> It's not so easy when you've found following syntax is also well
>> supported:
>>
>> ASN.1 10 > (oid "SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
>> #<OBJECT-ID SNMPv2-MIB::system (1) [9]>
>
> This syntax can easily be supported as follows
>
> (defmethod oid-parse ((source string))
>  (or
>   (with-input-from-string (s source)
>     (oid-parse s))
>   (with-input-from-string (s (concatenate 'string *asn.1-package- 
> prefix* source))
>     (oid-parse s))))

I'm afraid this function cannot work for (oid "SNMPv2- 
SMI::mib-2.SNMPv2-MIB::system") correctly, you just change it into:

(oid "ASN.1/SNMPv2-SMI::mib-2.SNMPv2-MIB::system")

How can you solve the second package name "SNMPv2-MIB" in that string?

>
>
> However, *asn.1-package-prefix* cannot contain a "." because of your
> strange readtable. So it must be changed to something other than
> "ASN.1/", perhaps "ASN1/"

The strange *oid-readtable* is my way to avoid the use of SPLIT- 
SEQUENCE, it just let #\. be treat as a separator. However, this  
method can cause unnecessary synbol intern in ASN.1 package, not the  
best way to read a OID string.

Give me more time to think out a better solution please.

>
>> Well, contact me if you really found your customers use such a MIB
>> module name, I think we can talk about this issue later:)
>
> It's kind of strange that you talk about making your package perfect  
> but
> have it so arranged that by loading in an MIB file one can overwrite
> your package itself . . .
>
> I guess it's not just C programmers who don't care about security ;-)

The *asn.1-package-prefix* was learnt from Common Lisp's CORBA  
language mapping. I think you've persuaded me, but I cannot change it  
until I can solve the OID reader syntax issue above.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
>> To support the cool (oid ..) syntax you can easily add "asn.1/" to the
>> start of the string processed by OID method.
>
> It's not so easy when you've found following syntax is also well
> supported:
>
> ASN.1 10 > (oid "SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
> #<OBJECT-ID SNMPv2-MIB::system (1) [9]>

This syntax can easily be supported as follows

(defmethod oid-parse ((source string))
  (or
   (with-input-from-string (s source)
     (oid-parse s))
   (with-input-from-string (s (concatenate 'string *asn.1-package-prefix* source))
     (oid-parse s))))

However, *asn.1-package-prefix* cannot contain a "." because of your
strange readtable. So it must be changed to something other than
"ASN.1/", perhaps "ASN1/"

[...]

> Well, contact me if you really found your customers use such a MIB
> module name, I think we can talk about this issue later:)

It's kind of strange that you talk about making your package perfect but
have it so arranged that by loading in an MIB file one can overwrite
your package itself . . .

I guess it's not just C programmers who don't care about security ;-)

>
>> [...]
>>
>>> The best way to load MIB definitions at runtime, I think, is to
>>> compile the ASN.1 definitions into another form: the AST itself or a
>>> modified version. Then we can use a pure-lisp ASN.1 "interpreter" to
>>> read this ACT, and create correspond OID variables at runtime. This
>>> is
>>> a delivery-free solution, we only need something like MAKE-INSTANCE,
>>> and DEFCLASS sometimes.
>>>
>>> What your opinion here?
>>
>> I think that is extra work. I would not be against it if you did it
>> but
>> I don't think it is a good use of my time. In our images we keep eval,
>> but not the compiler, so eval is a good option for us and compile-file
>> is a very bad option.
>
> OK, I'll do the "extra" work, to make my packages perfect.
>
>>
>>
>>>
>>> -----
>>> |3) I'd very like your functions to convert LispWorks PARSERGEN
>>> syntax
>>> into CL-YACC syntax.
>>>
>>> The reason to keep PARSERGEN: The ASN.1 compiler based on LispWorks'
>>> parsergen must be faster than those based on CL-YACC, and on
>>> LispWorks
>>> the CL-YACC package is not needed.
>>
>> Are you sure? I don't think that the speed is important and I am not
>> sure that there will be a big difference in speed as they use very
>> similar algorithms.
>
> You're right. After actual test, no big performance difference between
> cl-yacc and parsergen on lispworks, but I have other reasons to keep
> the current way:
>
>>
>>
>>> PARSERGEN will compile LALR(1) syntax into a series of Lisp functions
>>> which do the actual parsing job, these functions will be compiled and
>>> saved into syntax.fasl and on runtime they can do things very fast
>>> (no
>>> depend on these LALR definitions any more). Compare to that, CL-YACC
>>> is NOT a LALR compiler, it just compile LALR syntax definitons into a
>>> private-format "parser" object, and use it on runtime, this method
>>> should be slower than LispWorks' primitive parser functions.
>>
>> Are you sure?
>>
>> LALR is just about building a table. Does Lispworks really compile
>> down
>> the table into separate functions? I doubt it.
>
> It's true. Both ZEBU and PARSERGEN will compile parsers into separate
> parser functions:
>
> ASN.1 13 > (apropos "ASN.1-PARSER")
> ASN.1-PARSER-ACTION103 (defined)
> ASN.1-PARSER-ACTION42 (defined)
> ASN.1-PARSER-ACTION93 (defined)
> ASN.1-PARSER-ACTION61 (defined)
> ASN.1-PARSER-ACTION6 (defined)
> ASN.1-PARSER (defined)
> ASN.1-PARSER-ACTION12 (defined)
> ASN.1-PARSER-ACTION13 (defined)
> ASN.1-PARSER-ACTION20 (defined)
> ASN.1-PARSER-ACTION11 (defined)
> ASN.1-PARSER-ACTION26 (defined)
> ASN.1-PARSER-ACTION37 (defined)
> ASN.1-PARSER-ACTION34 (defined)
> ASN.1-PARSER-ACTION55 (defined)
> ASN.1-PARSER-ACTION59 (defined)
> ASN.1-PARSER-ACTION0 (defined)
> ASN.1-PARSER-ACTION100 (defined)
> ASN.1-PARSER-ACTION101 (defined)
> ASN.1-PARSER-ACTION41 (defined)
> ASN.1-PARSER-ACTION4 (defined)
> ASN.1-PARSER-ACTION131 (defined)
> ASN.1-PARSER-ACTION32 (defined)
> ASN.1-PARSER-ACTION53 (defined)
> ASN.1-PARSER-ACTION31 (defined)
> ASN.1-PARSER-ACTION86 (defined)
> ASN.1-PARSER-ACTION94 (defined)
> ASN.1-PARSER-ACTION95 (defined)
> ASN.1-PARSER-ACTION81 (defined)
> ASN.1-PARSER-ACTION9 (defined)
> ASN.1-PARSER-ACTION104 (defined)
> ASN.1-PARSER-ACTION119 (defined)
> ASN.1-PARSER-ACTION22 (defined)
> ASN.1-PARSER-ACTION33 (defined)
> ASN.1-PARSER-ACTION80 (defined)
> ASN.1-PARSER-ACTION76 (defined)
> ASN.1-PARSER-ACTION54 (defined)
> ASN.1-PARSER-ACTION75 (defined)
> ASN.1-PARSER-ACTION79 (defined)
> ASN.1-PARSER-ACTION2 (defined)
> ASN.1-PARSER-ACTION62 (defined)
> ASN.1-PARSER-ACTION71 (defined)
> ASN.1-PARSER-ACTION7 (defined)
> ASN.1-PARSER-ACTION60 (defined)
> ASN.1-PARSER-ACTION21 (defined)
> ASN.1-PARSER-ACTION77 (defined)
> ASN.1-PARSER-ACTION44 (defined)
> ASN.1-PARSER-ACTION78 (defined)
> ASN.1-PARSER-ACTION69 (defined)
> ASN.1-PARSER-ACTION1 (defined)
>
>>
>>
>> Why is the speed of loading in MIB files important anyway? I think
>> that
>> cl-yacc is sufficiently fast and the extra complexity of trying to
>> maintain separate identical grammars is just going to cause trouble
>> down
>> the line.
>
> Well, I think I'm not maintaining two grammars, just one, and your
> translation function from parsergen to cl-yacc is excellent. I want to
> keep this way since it works well. I'm a LispWorks customer and love
> it. It's my responsibility to give LispWorks users best performance
> and minimum dependency when using my packages.

Fair enough!

[...]

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

Hi, John

On 2009-2-9, at 13:56, John Fremlin wrote:

> Dear Chun Tian,
>
> "Chun Tian (binghe)" <bin...@gm...> writes:
> [...]
>> |1) For the question about nicknames of generated packages, I think  
>> my
>> current solution is the best, my ideas on this design:
>>
>> On ASN.1 package trunk, the ASN.1 "module" is directly mapped into
>> Lisp "package", this change let us could have two OID instances with
>> the same name but in different modules. One example is
>> "netSnmpAgentOIDs.linux" (in NET-SNMP-TC) and  
>> "ucdSnmpAgent.linux" (in
>> UCD-SNMP-MIB). A new OID function syntax has been introduced:
>>
>> ASN.1 47 > (oid "NET-SNMP-TC::linux")
>> #<OBJECT-ID NET-SNMP-TC::linux (10) [0]>
>
> The problem with this design is that the MIB module name is made
> directly into a Lisp package.
>
> This means that you cannot safely have a MIB module named COMMON- 
> LISP or
> CFFI or PPCRE or anything which is already a Lisp package. This is
> particularly important when the end-user is able to load in arbitrary
> MIB files.
>
> That's why the "asn.1/" prefix to generated package names is a good
> idea. It is already there. Just drop the nickname and we are fine.
>
> To support the cool (oid ..) syntax you can easily add "asn.1/" to the
> start of the string processed by OID method.

It's not so easy when you've found following syntax is also well  
supported:

ASN.1 10 > (oid "SNMPv2-SMI::mib-2.SNMPv2-MIB::system")
#<OBJECT-ID SNMPv2-MIB::system (1) [9]>

>
>
> [...]
>
>> Due to this design, the implementation of OID function is very easy,
>> and highly depend on what Common Lisp already have, and it's delivery
>> friendly (need only symbols but the CL compiler/interpreter). So the
>> "short" package name is necessary, no matter if they may clash with
>> other CL packages. (Actually the clash chance is very small, because
>> most formally defined MIB module has a "MIB" as part of its module
>> name).
>
> Yes, it is fine for most MIBs. But suppose the end-user (our customer)
> loads in an MIB called SYSTEM, it would cause all sorts of mayhem in
> the SYSTEM package.

Well, contact me if you really found your customers use such a MIB  
module name, I think we can talk about this issue later:)

> [...]
>
>> The best way to load MIB definitions at runtime, I think, is to
>> compile the ASN.1 definitions into another form: the AST itself or a
>> modified version. Then we can use a pure-lisp ASN.1 "interpreter" to
>> read this ACT, and create correspond OID variables at runtime. This  
>> is
>> a delivery-free solution, we only need something like MAKE-INSTANCE,
>> and DEFCLASS sometimes.
>>
>> What your opinion here?
>
> I think that is extra work. I would not be against it if you did it  
> but
> I don't think it is a good use of my time. In our images we keep eval,
> but not the compiler, so eval is a good option for us and compile-file
> is a very bad option.

OK, I'll do the "extra" work, to make my packages perfect.

>
>
>>
>> -----
>> |3) I'd very like your functions to convert LispWorks PARSERGEN  
>> syntax
>> into CL-YACC syntax.
>>
>> The reason to keep PARSERGEN: The ASN.1 compiler based on LispWorks'
>> parsergen must be faster than those based on CL-YACC, and on  
>> LispWorks
>> the CL-YACC package is not needed.
>
> Are you sure? I don't think that the speed is important and I am not
> sure that there will be a big difference in speed as they use very
> similar algorithms.

You're right. After actual test, no big performance difference between  
cl-yacc and parsergen on lispworks, but I have other reasons to keep  
the current way:

>
>
>> PARSERGEN will compile LALR(1) syntax into a series of Lisp functions
>> which do the actual parsing job, these functions will be compiled and
>> saved into syntax.fasl and on runtime they can do things very fast  
>> (no
>> depend on these LALR definitions any more). Compare to that, CL-YACC
>> is NOT a LALR compiler, it just compile LALR syntax definitons into a
>> private-format "parser" object, and use it on runtime, this method
>> should be slower than LispWorks' primitive parser functions.
>
> Are you sure?
>
> LALR is just about building a table. Does Lispworks really compile  
> down
> the table into separate functions? I doubt it.

It's true. Both ZEBU and PARSERGEN will compile parsers into separate  
parser functions:

ASN.1 13 > (apropos "ASN.1-PARSER")
ASN.1-PARSER-ACTION103 (defined)
ASN.1-PARSER-ACTION42 (defined)
ASN.1-PARSER-ACTION93 (defined)
ASN.1-PARSER-ACTION61 (defined)
ASN.1-PARSER-ACTION6 (defined)
ASN.1-PARSER (defined)
ASN.1-PARSER-ACTION12 (defined)
ASN.1-PARSER-ACTION13 (defined)
ASN.1-PARSER-ACTION20 (defined)
ASN.1-PARSER-ACTION11 (defined)
ASN.1-PARSER-ACTION26 (defined)
ASN.1-PARSER-ACTION37 (defined)
ASN.1-PARSER-ACTION34 (defined)
ASN.1-PARSER-ACTION55 (defined)
ASN.1-PARSER-ACTION59 (defined)
ASN.1-PARSER-ACTION0 (defined)
ASN.1-PARSER-ACTION100 (defined)
ASN.1-PARSER-ACTION101 (defined)
ASN.1-PARSER-ACTION41 (defined)
ASN.1-PARSER-ACTION4 (defined)
ASN.1-PARSER-ACTION131 (defined)
ASN.1-PARSER-ACTION32 (defined)
ASN.1-PARSER-ACTION53 (defined)
ASN.1-PARSER-ACTION31 (defined)
ASN.1-PARSER-ACTION86 (defined)
ASN.1-PARSER-ACTION94 (defined)
ASN.1-PARSER-ACTION95 (defined)
ASN.1-PARSER-ACTION81 (defined)
ASN.1-PARSER-ACTION9 (defined)
ASN.1-PARSER-ACTION104 (defined)
ASN.1-PARSER-ACTION119 (defined)
ASN.1-PARSER-ACTION22 (defined)
ASN.1-PARSER-ACTION33 (defined)
ASN.1-PARSER-ACTION80 (defined)
ASN.1-PARSER-ACTION76 (defined)
ASN.1-PARSER-ACTION54 (defined)
ASN.1-PARSER-ACTION75 (defined)
ASN.1-PARSER-ACTION79 (defined)
ASN.1-PARSER-ACTION2 (defined)
ASN.1-PARSER-ACTION62 (defined)
ASN.1-PARSER-ACTION71 (defined)
ASN.1-PARSER-ACTION7 (defined)
ASN.1-PARSER-ACTION60 (defined)
ASN.1-PARSER-ACTION21 (defined)
ASN.1-PARSER-ACTION77 (defined)
ASN.1-PARSER-ACTION44 (defined)
ASN.1-PARSER-ACTION78 (defined)
ASN.1-PARSER-ACTION69 (defined)
ASN.1-PARSER-ACTION1 (defined)

>
>
> Why is the speed of loading in MIB files important anyway? I think  
> that
> cl-yacc is sufficiently fast and the extra complexity of trying to
> maintain separate identical grammars is just going to cause trouble  
> down
> the line.

Well, I think I'm not maintaining two grammars, just one, and your  
translation function from parsergen to cl-yacc is excellent. I want to  
keep this way since it works well. I'm a LispWorks customer and love  
it. It's my responsibility to give LispWorks users best performance  
and minimum dependency when using my packages.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:
[...]
> |1) For the question about nicknames of generated packages, I think my
> current solution is the best, my ideas on this design:
>
> On ASN.1 package trunk, the ASN.1 "module" is directly mapped into
> Lisp "package", this change let us could have two OID instances with
> the same name but in different modules. One example is
> "netSnmpAgentOIDs.linux" (in NET-SNMP-TC) and "ucdSnmpAgent.linux" (in
> UCD-SNMP-MIB). A new OID function syntax has been introduced:
>
> ASN.1 47 > (oid "NET-SNMP-TC::linux")
> #<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

The problem with this design is that the MIB module name is made
directly into a Lisp package.

This means that you cannot safely have a MIB module named COMMON-LISP or
CFFI or PPCRE or anything which is already a Lisp package. This is
particularly important when the end-user is able to load in arbitrary
MIB files.

That's why the "asn.1/" prefix to generated package names is a good
idea. It is already there. Just drop the nickname and we are fine.

To support the cool (oid ..) syntax you can easily add "asn.1/" to the
start of the string processed by OID method.

[...]

> Due to this design, the implementation of OID function is very easy,
> and highly depend on what Common Lisp already have, and it's delivery
> friendly (need only symbols but the CL compiler/interpreter). So the
> "short" package name is necessary, no matter if they may clash with
> other CL packages. (Actually the clash chance is very small, because
> most formally defined MIB module has a "MIB" as part of its module
> name).

Yes, it is fine for most MIBs. But suppose the end-user (our customer)
loads in an MIB called SYSTEM, it would cause all sorts of mayhem in
the SYSTEM package.

[...]

> |2) For the "EVAL" issue and your related questions. I think, the only
> issue you have solved is to use cl-yacc instead of lispworks'
> parsergen, then all supported CL platform can use the ASN.1 compiler.
>
> I don't quite like the use of function EVAL at runtime, because this
> will cause some delivery issues. I don't know much about Allegro CL,
> but to support EVAL on runtime in delivered LispWorks images, the
> whole Common Lisp runtime and interpreter will be hold in target
> executions, this may increase the file size. And EVAL itself also
> suffer from environment-related problems, one of which is the symbol
> intern issues you have mentioned.

The symbol intern issues can easily be fixed, and it is wrong for extra
symbols to be interned in the ASN.1 package whether or not we use eval.

> The better way to load MIB definitions at runtime may be to use
> COMPILE-FILE and LOAD instead of EVAL, if a new lisp file is generated
> from the ASN.1 definitions, we should be able to COMPILE-FILE it
> first, and then LOAD it.

Compile file is more tricky than eval because it requires the compiler.

[...]

> The best way to load MIB definitions at runtime, I think, is to
> compile the ASN.1 definitions into another form: the AST itself or a
> modified version. Then we can use a pure-lisp ASN.1 "interpreter" to
> read this ACT, and create correspond OID variables at runtime. This is
> a delivery-free solution, we only need something like MAKE-INSTANCE,
> and DEFCLASS sometimes.
>
> What your opinion here?

I think that is extra work. I would not be against it if you did it but
I don't think it is a good use of my time. In our images we keep eval,
but not the compiler, so eval is a good option for us and compile-file
is a very bad option.

>
> -----
> |3) I'd very like your functions to convert LispWorks PARSERGEN syntax
> into CL-YACC syntax.
>
> The reason to keep PARSERGEN: The ASN.1 compiler based on LispWorks'
> parsergen must be faster than those based on CL-YACC, and on LispWorks
> the CL-YACC package is not needed.

Are you sure? I don't think that the speed is important and I am not
sure that there will be a big difference in speed as they use very
similar algorithms.

> PARSERGEN will compile LALR(1) syntax into a series of Lisp functions
> which do the actual parsing job, these functions will be compiled and
> saved into syntax.fasl and on runtime they can do things very fast (no
> depend on these LALR definitions any more). Compare to that, CL-YACC
> is NOT a LALR compiler, it just compile LALR syntax definitons into a
> private-format "parser" object, and use it on runtime, this method
> should be slower than LispWorks' primitive parser functions.

Are you sure?

LALR is just about building a table. Does Lispworks really compile down
the table into separate functions? I doubt it.

Why is the speed of loading in MIB files important anyway? I think that
cl-yacc is sufficiently fast and the extra complexity of trying to
maintain separate identical grammars is just going to cause trouble down
the line.

[...]

Re: [lisp-snmp] [patch] CL-YACC support in ASN.1

[John F. <jf...@ms...>]

Dear Chun Tian,

"Chun Tian (binghe)" <bin...@gm...> writes:

> I've merged your patch in a quite different way:
>
> 1) Define a variable *asn.1-syntax* to hold all LALR syntax rules.
> 2) Convert your CL-YACC changes into loadable patch and put them in
> #p"ASN1:vendor;"
> 3) On LispWorks, still use PARSERGEN
> 4) On non-LispWorks, use your DEFPARSER-TO-YACC to convert *asn.1-
> syntax* into CL-YACC format.
>
> 5) The "eval-compile-asn.1" funtion haven't merged.
>
> I think it's beautiful way to made this big changes and only generate
> minimum diff to original code, but ... it seems that something goes
> wrong, either in CL-YACC patch side, or in DEFPARSER-TO-YACC. Can you
> find it?

I don't like this way.

Please don't do it like this.

The defparser-to-yacc function was a quick hack for one time use. It was
not designed to be used like this.

It is only luck that this particular grammar can be transformed by this
function. It is quite likely that more complex grammars cannot be
transformed. The function would have to be extended to support more
advanced grammars.

The reason I kept both grammars active, was so that you could test that
the transform was correct. That is the only reason. It is difficult to
make a general grammar translator. The function I made is not a general
grammar translator. It only works on simple grammars.

Having two different parser generators does not bring any benefits, as
far as I can see, and means that there is quite likely to be different
behaviour on different Lisps. Why do you want to keep using the
Lispworks generator?

I recommend having a transition period until Juliusz releases a new
version of cl-yacc, and then moving over to cl-yacc entirely.

PS. I have a working portable version of cdr-assoc in my personal
development tree of cl-irregsexp (the one in the released version is
borked). When I get the Internet at home I will push it to the cl.net
repo.

[...]

[lisp-snmp] New ASN.1:OID documentation

[Chun T. (b. <bin...@gm...>]

ASN.1:OID (general function)

Convert anything to OBJECT-ID.

1. For single names, return all possible OIDs as multiple values:

    > (oid "linux")
    #<OBJECT-ID NET-SNMP-TC::linux (10) [0]>
    #<OBJECT-ID UCD-SNMP-MIB::linux (10) [0]>

2. For names plus numbers, create new OID instances:

    > (oid "sysDescr.0")
    #<OBJECT-ID SNMPv2-MIB::sysDescr.0>

3. For names list, validating every names:

    > (oid "ucdSnmpAgent.linux")
    #<OBJECT-ID UCD-SNMP-MIB::linux (10) [0]>

4. For pure number list, search and build OID from 'zero':

    > (oid "0.2.3.4")
    #<OBJECT-ID zero.2.3.4>
    > (oid "1.2.3")
    #<OBJECT-ID iso.2.3>
    > (oid ".1.2.3")
    #<OBJECT-ID iso.2.3>

5. OID name with module is supported:

    > (oid "NET-SNMP-TC::linux")
    #<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

[lisp-snmp] ASN.1 compiler runs on every supported CL

[Chun T. (binghe) <bin...@gm...>]

Hi, cl-net-snmp users

Thanks to John Fremlin from Mathematical Systems Inc (MSI), the ASN.1  
compiler in cl-net-snmp has been port to non-LispWorks CL.

Now snmp-dev.asd and asn.1-dev.asd on trunk can be loaded on all  
supported CL, and SNMP:UPDATE-MIB can also be used.

This is a big new feature in our next cl-net-snmp release.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] [patch] CL-YACC support in ASN.1 (was Re: SF.net SVN: cl-net-snmp:[737] asn.1/trunk)

[Chun T. (b. <bin...@gm...>]

On 2009-2-7, at 00:15, Chun Tian (binghe) wrote:

> Hi, John
>
> I've merged your patch in a quite different way:
>
> 1) Define a variable *asn.1-syntax* to hold all LALR syntax rules.
> 2) Convert your CL-YACC changes into loadable patch and put them in  
> #p"ASN1:vendor;"
> 3) On LispWorks, still use PARSERGEN
> 4) On non-LispWorks, use your DEFPARSER-TO-YACC to convert *asn.1- 
> syntax* into CL-YACC format.
>
> 5) The "eval-compile-asn.1" funtion haven't merged.
>
> I think it's beautiful way to made this big changes and only  
> generate minimum diff to original code, but ... it seems that  
> something goes wrong, either in CL-YACC patch side, or in DEFPARSER- 
> TO-YACC. Can you find it?
>

I've fixed this bug. I used a very dirty way to implement LispWorks'  
CDR-ASSOC ...

> Regards,
>
> --binghe

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

[lisp-snmp] [patch] CL-YACC support in ASN.1 (was Re: SF.net SVN: cl-net-snmp:[737] asn.1/trunk)

[Chun T. (binghe) <bin...@gm...>]

Hi, John

I've merged your patch in a quite different way:

1) Define a variable *asn.1-syntax* to hold all LALR syntax rules.
2) Convert your CL-YACC changes into loadable patch and put them in  
#p"ASN1:vendor;"
3) On LispWorks, still use PARSERGEN
4) On non-LispWorks, use your DEFPARSER-TO-YACC to convert *asn.1- 
syntax* into CL-YACC format.

5) The "eval-compile-asn.1" funtion haven't merged.

I think it's beautiful way to made this big changes and only generate  
minimum diff to original code, but ... it seems that something goes  
wrong, either in CL-YACC patch side, or in DEFPARSER-TO-YACC. Can you  
find it?

Regards,

--binghe

On 2009-2-7, at 00:04, bi...@us... wrote:

> Revision: 737
>          http://cl-net-snmp.svn.sourceforge.net/cl-net-snmp/?rev=737&view=rev
> Author:   binghe
> Date:     2009-02-06 16:04:09 +0000 (Fri, 06 Feb 2009)
>
> Log Message:
> -----------
> [asn.1] merge YACC patch from John Fremlin (MSI)
>
> Modified Paths:
> --------------
>    asn.1/trunk/asn.1-dev.asd
>    asn.1/trunk/compiler/compiler.lisp
>    asn.1/trunk/compiler/parser.lisp
>    asn.1/trunk/compiler/syntax.lisp
>
> Added Paths:
> -----------
>    asn.1/trunk/vendor/yacc.lisp
>
> Modified: asn.1/trunk/asn.1-dev.asd
> ===================================================================
> --- asn.1/trunk/asn.1-dev.asd	2009-02-06 16:00:46 UTC (rev 736)
> +++ asn.1/trunk/asn.1-dev.asd	2009-02-06 16:04:09 UTC (rev 737)
> @@ -10,7 +10,6 @@
> #+lispworks
> (require "parsergen")
>
> -#+lispworks
> (defsystem asn.1-dev
>   :description "ASN.1 Compiler"
>   :version "5.0-dev"
> @@ -18,6 +17,7 @@
>   :author "Chun Tian (binghe) <bin...@gm...>"
>   :depends-on (:asn.1 #-lispworks :yacc)
>   :components ((:module "compiler"
> +                :depends-on ("vendor")
> 		:components ((:file "reader")
> 			     (:file "syntax")
> 			     (:file "parser"
> @@ -27,4 +27,6 @@
>                              (:file "compiler"
>                               :depends-on ("sort" "parser" "compile- 
> type"))
>                              (:file "interpreter"
> -                              :depends-on ("compiler"))))))
> +                              :depends-on ("compiler"))))
> +               (:module "vendor"
> +                :components (#-lispworks (:file "yacc")))))
>
> Modified: asn.1/trunk/compiler/compiler.lisp
> ===================================================================
> --- asn.1/trunk/compiler/compiler.lisp	2009-02-06 16:00:46 UTC (rev  
> 736)
> +++ asn.1/trunk/compiler/compiler.lisp	2009-02-06 16:04:09 UTC (rev  
> 737)
> @@ -3,7 +3,7 @@
>
> (in-package :asn.1)
>
> -(defvar *mib-name-map-table* (make-hash-table :test #'string=))
> +(defvar *mib-name-map-table* (make-hash-table :test #'equal))
>
> ;;; Empty and old MIB module
> (defvar *mib-name-map* nil)
>
> Modified: asn.1/trunk/compiler/parser.lisp
> ===================================================================
> --- asn.1/trunk/compiler/parser.lisp	2009-02-06 16:00:46 UTC (rev 736)
> +++ asn.1/trunk/compiler/parser.lisp	2009-02-06 16:04:09 UTC (rev 737)
> @@ -3,13 +3,28 @@
>
> (in-package :asn.1)
>
> +#+lispworks
> +(macrolet ((define-parser (name syntax)
> +             `(parsergen:defparser ,name ,@syntax)))
> +  (define-parser asn.1-parser #.*asn.1-syntax*))
> +
> +#-lispworks
> +(macrolet ((define-parser (name reserved-words start-symbol syntax)
> +             `(yacc:define-parser ,name
> +                (:terminals ,reserved-words)
> +                (:start-symbol ,start-symbol)
> +                ,@syntax)))
> +  (define-parser *asn.1-parser*
> +                 #.(list* :id :number :string *reserved-words*)
> +                 %module-definition
> +                 #.(defparser-to-yacc *asn.1-syntax*)))
> +
> (defun asn.1-lexer (stream)
>   (let ((*readtable* *asn.1-readtable*)
>         (*package* (find-package :asn.1)))
>     (let ((token (read stream nil nil nil)))
> -      (if token
> -        (values (detect-token token) token)
> -        (values nil nil)))))
> +      (when token
> +        (values (detect-token token) token)))))
>
> (defgeneric detect-token (token))
>
> @@ -33,7 +48,10 @@
>     (parse s)))
>
> (defmethod parse ((source stream))
> -  (asn.1-parser #'(lambda () (asn.1-lexer source))))
> +  (funcall #+lispworks #'asn.1-parser
> +           #-lispworks #'yacc:parse-with-lexer
> +           #'(lambda () (asn.1-lexer source))
> +           #-lispworks *asn.1-parser*))
>
> (defmethod parse ((source t))
>   (error "Unknown Parser Source"))
>
> Modified: asn.1/trunk/compiler/syntax.lisp
> ===================================================================
> --- asn.1/trunk/compiler/syntax.lisp	2009-02-06 16:00:46 UTC (rev 736)
> +++ asn.1/trunk/compiler/syntax.lisp	2009-02-06 16:04:09 UTC (rev 737)
> @@ -44,9 +44,9 @@
> (eval-when (:load-toplevel :execute)
>   (fill-reserved-words))
>
> -(parsergen:defparser asn.1-parser
> +(defparameter *asn.1-syntax* '(
>   ((%root %module-definition) $1)
> -  ((root assignment) $1)
> +  ;; ((root assignment) $1)
>   ((%module-definition :id
>                        DEFINITIONS
>                        %tag-default
> @@ -244,3 +244,61 @@
>   ((%general IDENTIFIER) $1)
>   ((%general INTEGER) $1)
>   ((%general |IA5String|) $1))
> +) ;; defvar *asn.1-syntax*
> +
> +;;; binghe: flatten is from alexandria project
> +(defun flatten (tree)
> +  "Traverses the tree in order, collecting non-null leaves into a  
> list."
> +  (let (list)
> +    (labels ((traverse (subtree)
> +               (when subtree
> +                 (if (consp subtree)
> +                     (progn
> +                       (traverse (car subtree))
> +                       (traverse (cdr subtree)))
> +                     (push subtree list)))))
> +      (traverse tree))
> +    (nreverse list)))
> +
> +;;; binghe: following funtions is contributed by John Fremlin from  
> MSI <jf...@ms...>
> +
> +(defun defparser-production-to-yacc (grammar-symbols forms)
> +  (cond ((not (and forms (or (not (listp forms)) (some 'identity  
> forms))))
> +	 nil)
> +	(t
> +	 (labels ((make-var (i)
> +		    (intern (format nil "$~D" i)))
> +		  (vars ()
> +		    (loop for i from 1
> +                          for x in grammar-symbols collect (make- 
> var i)))
> +		  (used-vars ()
> +                    (remove-duplicates
> +                     (loop for sym in (flatten forms)
> +                           when (and (symbolp sym) (eql #\$ (elt  
> (symbol-name sym) 0))
> +                                     (ignore-errors (parse-integer  
> (symbol-name sym) :start 1)))
> +                           collect (make-var (parse-integer (symbol- 
> name sym) :start 1)))
> +                     :test 'eql)))
> +	   (let ((unused-vars (set-difference (vars) (used-vars))))
> +	     (list `#'(lambda(,@(vars))
> +                        ,@(when unused-vars  (list `(declare  
> (ignore ,@unused-vars))))
> +                        ,forms)))))))
> +
> +(defun defparser-to-yacc (rules)
> +  (let (grouped-rules)
> +    (loop for rule in rules do
> +	  (destructuring-bind ((non-terminal &rest grammar-symbols)  
> &optional forms)
> +	      rule
> +	    (assert non-terminal)
> +            #+ignore
> +            (push (append grammar-symbols (defparser-production-to- 
> yacc grammar-symbols forms))
> +		  (sys:cdr-assoc non-terminal grouped-rules))
> +            ;;; binghe: I don't know if this rewrite is correct ...
> +            (let ((temp (assoc non-terminal grouped-rules))
> +                  (item (append grammar-symbols
> +                                (defparser-production-to-yacc  
> grammar-symbols forms))))
> +              (if (null temp)
> +                  (setf grouped-rules (acons non-terminal item  
> grouped-rules))
> +                (rplacd temp
> +                        (push item (cdr temp)))))))
> +    (loop for (name . alternatives) in (nreverse grouped-rules)
> +	  collect `(,name ,@(reverse alternatives)))))
>
> Added: asn.1/trunk/vendor/yacc.lisp
> ===================================================================
> --- asn.1/trunk/vendor/yacc.lisp	                        (rev 0)
> +++ asn.1/trunk/vendor/yacc.lisp	2009-02-06 16:04:09 UTC (rev 737)
> @@ -0,0 +1,75 @@
> +;;;; -*- Mode: Lisp -*-
> +;;;; $Id$
> +
> +;;;; Patch to CL-YACC, made by John Fremlin from MSI  
> <jf...@ms...>, waiting for merge
> +
> +(in-package :yacc)
> +
> +(define-condition conflict-warning (yacc-compile-warning simple- 
> warning)
> +  ((kind :initarg :kind :reader conflict-warning-kind)
> +   (state :initarg :state :reader conflict-warning-state)
> +   (terminal :initarg :terminal :reader conflict-warning-terminal)
> +   (chosen-action :initarg :chosen-action :reader conflict-warning- 
> chosen-action :initform nil))
> +  (:report (lambda (w stream)
> +             (format stream "~A conflict on terminal ~S in state  
> ~A, ~_~?, ~@[taking action ~A~]"
> +                     (case (conflict-warning-kind w)
> +                       (:shift-reduce "Shift/Reduce")
> +                       (:reduce-reduce "Reduce/Reduce")
> +                       (t (conflict-warning-kind w)))
> +                     (conflict-warning-terminal w)
> +                     (conflict-warning-state w)
> +                     (simple-condition-format-control w)
> +                     (simple-condition-format-arguments w)
> +		     (conflict-warning-chosen-action w)))))
> +
> +(defun handle-conflict (a1 a2 grammar action-productions id s
> +                        &optional muffle-conflicts)
> +  "Decide what to do with a conflict between A1 and A2 in state ID  
> on symbol S.
> +Returns three actions: the chosen action, the number of new sr and  
> rr."
> +  (declare (type action a1 a2) (type grammar grammar)
> +           (type index id) (symbol s))
> +  (when (action-equal-p a1 a2)
> +    (return-from handle-conflict (values a1 0 0)))
> +  (when (and (shift-action-p a2) (reduce-action-p a1))
> +    (psetq a1 a2 a2 a1))
> +  (let ((p1 (cdr (assoc a1 action-productions)))
> +        (p2 (cdr (assoc a2 action-productions))))
> +    ;; operator precedence and associativity
> +    (when (and (shift-action-p a1) (reduce-action-p a2))
> +      (let* ((op1 (find-single-terminal (production-derives p1)  
> grammar))
> +             (op2 (find-single-terminal (production-derives p2)  
> grammar))
> +             (op1-tail (find-precedence op1 (grammar-precedence  
> grammar)))
> +             (op2-tail (find-precedence op2 (grammar-precedence  
> grammar))))
> +        (when (and (eq s op1) op1-tail op2-tail)
> +          (cond
> +            ((eq op1-tail op2-tail)
> +             (return-from handle-conflict
> +               (ecase (caar op1-tail)
> +                 ((:left) (values a2 0 0))
> +                 ((:right) (values a1 0 0))
> +                 ((:nonassoc) (values nil 0 0)))))
> +            (t
> +             (return-from handle-conflict
> +               (if (tailp op2-tail (cdr op1-tail))
> +                   (values a1 0 0)
> +                   (values a2 0 0))))))))
> +    ;; default: prefer first defined production
> +    (let ((kind (typecase a1
> +                  (shift-action :shift-reduce)
> +                  (t :reduce-reduce)))
> +	  (chosen-action
> +	   (if (production< p1 p2)
> +	       a1
> +             a2)))
> +      (unless muffle-conflicts
> +	(warn (make-condition
> +               'conflict-warning
> +               :kind kind
> +               :state id :terminal s
> +               :format-control "~S and ~S~@[ ~_~A~]~@[ ~_~A~]"
> +               :format-arguments (list a1 a2 p1 p2)
> +               :chosen-action chosen-action)))
> +      (values chosen-action
> +	      (if (eq kind :shift-reduce) 1 0)
> +	      (if (eq kind :reduce-reduce) 1 0)))))
> +
>
>
> Property changes on: asn.1/trunk/vendor/yacc.lisp
> ___________________________________________________________________
> Added: svn:keywords
>   + Id
>
>
> This was sent by the SourceForge.net collaborative development  
> platform, the world's largest Open Source development site.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (binghe) <bin...@gm...>]

Hi, John

Thanks for your work, and I would like to merge it. Read following  
words please:

-----
|1) For the question about nicknames of generated packages, I think my  
current solution is the best, my ideas on this design:

On ASN.1 package trunk, the ASN.1 "module" is directly mapped into  
Lisp "package", this change let us could have two OID instances with  
the same name but in different modules. One example is  
"netSnmpAgentOIDs.linux" (in NET-SNMP-TC) and "ucdSnmpAgent.linux" (in  
UCD-SNMP-MIB). A new OID function syntax has been introduced:

ASN.1 47 > (oid "NET-SNMP-TC::linux")
#<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

ASN.1 48 > (oid "UCD-SNMP-MIB::linux")
#<OBJECT-ID UCD-SNMP-MIB::linux (10) [0]>

Or using following method:

ASN.1 49 > (oid "netSnmpAgentOIDs.linux")
#<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

ASN.1 50 > (oid "ucdSnmpAgent.linux")
#<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

The "NET-SNMP-TC" in form (oid "NET-SNMP-TC::linux") is just a package  
name, and the "linux" is just a variable in that package, so following  
method can also works:

ASN.1 53 > NET-SNMP-TC::|linux|
#<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

ASN.1 54 > UCD-SNMP-MIB::|linux|
#<OBJECT-ID UCD-SNMP-MIB::linux (10) [0]>

Due to this design, the implementation of OID function is very easy,  
and highly depend on what Common Lisp already have, and it's delivery  
friendly (need only symbols but the CL compiler/interpreter). So the  
"short" package name is necessary, no matter if they may clash with  
other CL packages. (Actually the clash chance is very small, because  
most formally defined MIB module has a "MIB" as part of its module  
name).

The "long" package name is used to identify these packages when a  
Lisper look at the output of (LIST-ALL-PACKAGES), because the main  
name of a package will be shown when it's PRINT-OBJECTed.

I hope you can understand above explanation, I really don't want to  
change this part unless there're better solutions founded on it.

-----
|2) For the "EVAL" issue and your related questions. I think, the only  
issue you have solved is to use cl-yacc instead of lispworks'  
parsergen, then all supported CL platform can use the ASN.1 compiler.

I don't quite like the use of function EVAL at runtime, because this  
will cause some delivery issues. I don't know much about Allegro CL,  
but to support EVAL on runtime in delivered LispWorks images, the  
whole Common Lisp runtime and interpreter will be hold in target  
executions, this may increase the file size. And EVAL itself also  
suffer from environment-related problems, one of which is the symbol  
intern issues you have mentioned.

The better way to load MIB definitions at runtime may be to use  
COMPILE-FILE and LOAD instead of EVAL, if a new lisp file is generated  
from the ASN.1 definitions, we should be able to COMPILE-FILE it  
first, and then LOAD it. This could also solve the rest  issues. But  
unfortunately this approach only works on Open Source CL platforms  
which COMPILE-FILE is free. In LispWorks, COMPILE-FILE function is  
removed after delivery. I think Allegro CL have more strict rules on  
this part due to its complex license options.

The best way to load MIB definitions at runtime, I think, is to  
compile the ASN.1 definitions into another form: the AST itself or a  
modified version. Then we can use a pure-lisp ASN.1 "interpreter" to  
read this ACT, and create correspond OID variables at runtime. This is  
a delivery-free solution, we only need something like MAKE-INSTANCE,  
and DEFCLASS sometimes.

What your opinion here?

-----
|3) I'd very like your functions to convert LispWorks PARSERGEN syntax  
into CL-YACC syntax.

The reason to keep PARSERGEN: The ASN.1 compiler based on LispWorks'  
parsergen must be faster than those based on CL-YACC, and on LispWorks  
the CL-YACC package is not needed.

PARSERGEN will compile LALR(1) syntax into a series of Lisp functions  
which do the actual parsing job, these functions will be compiled and  
saved into syntax.fasl and on runtime they can do things very fast (no  
depend on these LALR definitions any more). Compare to that, CL-YACC  
is NOT a LALR compiler, it just compile LALR syntax definitons into a  
private-format "parser" object, and use it on runtime, this method  
should be slower than LispWorks' primitive parser functions.

I'll try to modify your patch to keep the ASN.1 syntax definitions  
only one copy, and let both PARSERGEN and CL-YACC can use it.

Opinions are welcome.

On 2009-2-6, at 18:26, John Fremlin wrote:

> Hi Chun Tian,
>
> This patch allows one to use cl-yacc to parse ASN.1 files instead of
> only Lispworks' parsergen. We intend to allow users to load in ASN.1  
> MIB
> definitions for our SNMP project. It requires a patch to cl-yacc,
> attached at the bottom of this mail.
>
>
> A new function eval-compile-asn.1 was introduced, to immediately  
> load a
> compiled ASN.1.
>
> There are several problems relating to the use of Common Lisp packages
> for each ASN module definition.
>
> Would it be possible to stop giving the package a nickname that
> is entirely specified by the user? The long-package-name made with
>
> (defvar *asn.1-package-prefix* "ASN.1/")
>
> (defun module->package (module)
>  (declare (type symbol module))
>  (let ((package-symbol (intern (concatenate 'string
>                                             *asn.1-package-prefix*
>                                             (symbol-name module))
>                                :keyword)))
>    (the symbol package-symbol)))
>
> is just fine, and it is annoying to have a :nickname that may clash  
> with
> a normal Lisp package.
>
>
> Secondly, it is not possible to eval the output of the compiler  
> directly
> because it incorrectly interns symbols in the current package, when  
> they
> should be interned in the module package. The module-specific  
> package must
> created at compile time to support this.
>
> Would you accept a patch for these changes?
>
> Running on Lispworks, the patch will use both the cl-yacc parser and  
> the
> parsergen parser simultaneously and asserts that the output of both is
> equalp. When enough testing has been done, this can be removed and  
> only
> one parser used.
>
> Please review this
>
> Index: package.lisp
> ===================================================================
> --- package.lisp	(revision 706)
> +++ package.lisp	(working copy)
> @@ -27,6 +27,7 @@
>            #:defoid
>            #:delete-object
>            #:ensure-oid
> +	   #:eval-compile-asn.1
>            #:gauge
> 	   #:general-type
>            #:get-asn.1-type
> Index: asn.1-dev.asd
> ===================================================================
> --- asn.1-dev.asd	(revision 706)
> +++ asn.1-dev.asd	(working copy)
> @@ -10,17 +10,17 @@
> #+lispworks
> (require "parsergen")
>
> -#+lispworks
> (defsystem asn.1-dev
>   :description "ASN.1 Compiler"
>   :version "5.0-dev"
>   :licence "MIT"
>   :author "Chun Tian (binghe) <bin...@gm...>"
> -  :depends-on (:asn.1)
> +  :depends-on (:asn.1 :yacc)
>   :components ((:module "compiler"
> 		:components ((:file "reader")
> 			     (:file "syntax")
> -			     (:file "parser" :depends-on ("reader" "syntax"))
> +			     (:file "yacc" :depends-on ("syntax"))
> +			     (:file "parser" :depends-on ("yacc" "reader"))
>                              (:file "compile-type")
>                              (:file "sort")
>                              (:file "compiler" :depends-on ("sort"  
> "parser" "compile-type"))
> Index: compiler/compiler.lisp
> ===================================================================
> --- compiler/compiler.lisp	(revision 706)
> +++ compiler/compiler.lisp	(working copy)
> @@ -3,7 +3,7 @@
>
> (in-package :asn.1)
>
> -(defvar *mib-name-map-table* (make-hash-table :test #'string=))
> +(defvar *mib-name-map-table* (make-hash-table :test #'equal))
>
> ;;; Empty and old MIB module
> (defvar *mib-name-map* nil)
> @@ -223,3 +223,23 @@
> (defmethod compile-dfo-internal ((key (eql 'MAX-ACCESS)) (value  
> symbol))
>   (declare (ignore key))
>   `(:max-access ',value))
> +
> +
> +(defun eval-compile-asn.1 (pathname)
> +;; Unfortunately it is not possible to eval the generated list
> +;; directly, because the package in which the symbols should actually
> +;; reside is not created at compile time.
> +
> +;; To move into the correct package, the list is written to a string
> +;; and then read in.
> +
> +;; This is very messy and should be fixed.
> +  (with-standard-io-syntax
> +    (let ((*package* (find-package #.(package-name *package*))))
> +      (handler-case
> +	  (with-input-from-string
> +	      (*standard-input*
> +	       (with-output-to-string (*standard-output*)
> +		 (mapcar 'print (sort-definitions (cdr (compile-asn.1  
> pathname))))))
> +	    (loop (eval (read))))
> +	(end-of-file ())))))
> Index: compiler/parser.lisp
> ===================================================================
> --- compiler/parser.lisp	(revision 706)
> +++ compiler/parser.lisp	(working copy)
> @@ -7,9 +7,8 @@
>   (let ((*readtable* *asn.1-readtable*)
>         (*package* (find-package :asn.1)))
>     (let ((token (read stream nil nil nil)))
> -      (if token
> -        (values (detect-token token) token)
> -        (values nil nil)))))
> +      (when token
> +        (values (detect-token token) token)))))
>
> (defgeneric detect-token (token))
>
> @@ -32,8 +31,27 @@
>   (with-open-file (s source :direction :input)
>     (parse s)))
>
> -(defmethod parse ((source stream))
> -  (asn.1-parser #'(lambda () (asn.1-lexer source))))
> +(defmethod parse ((stream stream))
> +  #-lispworks
> +  (yacc:parse-with-lexer
> +   (lambda() (asn.1-lexer stream))
> +   *asn-1-yacc-parser*)
>
> +  #+lispworks
> +  (progn
> +    (let ((forms (loop for pair = (multiple-value-list (asn.1-lexer  
> stream))
> +		       collect pair
> +		       until (every 'not pair))))
> +      (flet ((lexer ()
> +	       (let ((forms forms))
> +		 (lambda()(values-list (pop forms))))))
> +	(let ((parsergen (asn.1-parser (lexer)))
> +	      (yacc
> +	       (yacc:parse-with-lexer
> +		(lexer)
> +		*asn-1-yacc-parser*)))
> +	  (assert (equalp yacc parsergen))
> +	  yacc)))))
> +
> (defmethod parse ((source t))
>   (error "Unknown Parser Source"))
> Index: compiler/yacc.lisp
> ===================================================================
> --- compiler/yacc.lisp	(revision 0)
> +++ compiler/yacc.lisp	(revision 0)
> @@ -0,0 +1,431 @@
> +(in-package :asn.1)
> +
> +(yacc:define-parser *asn-1-yacc-parser*
> +  (:terminals
> +   #.`(,@*reserved-words*
> +	   :id
> +	   :number
> +	   :string))
> +  (:start-symbol
> +   %module-definition)
> +
> +  (root (assignment #'(lambda ($1) $1)))
> +  (%module-definition
> +   (:id
> +    definitions
> +    %tag-default
> +    %extension-default
> +    \:\:=
> +    begin
> +    %module-body
> +    end
> +    #'(lambda ($1 $2 $3 $4 $5 $6 $7 $8)
> +	(declare (ignore $8 $6 $5 $4 $3 $2))
> +	`(:module ,$1 ,$7))))
> +  (%tag-default nil)
> +  (%extension-default nil)
> +  (%module-body
> +   (%exports
> +    %imports
> +    %assignment-list
> +    #'(lambda ($1 $2 $3) `((,$1 ,$2) ,$3)))
> +   nil)
> +  (%exports
> +   (exports
> +    all
> +    \;
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $2 $1)) '(export :all)))
> +   (exports
> +    %symbol*
> +    \;
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $1)) `(export ,@$2)))
> +   nil)
> +  (%imports
> +   (imports
> +    %symbols-from-modules
> +    \;
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $1)) `(import ,@$2)))
> +   nil)
> +  (%symbols-from-modules
> +   (%symbols-from-module #'(lambda ($1) `(,$1)))
> +   (%symbols-from-modules
> +    %symbols-from-module
> +    #'(lambda ($1 $2) `(,@$1 ,$2))))
> +  (%symbols-from-module
> +   (%symbol+
> +    from
> +    :id
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,$1 :from ,$3))))
> +  (%assignment-list
> +   (%assignment-list %assignment #'(lambda ($1 $2) `(,@$1 ,$2)))
> +   (%assignment #'(lambda ($1) `(,$1)))
> +   nil)
> +  (%assignment
> +   (%macro-definition #'(lambda ($1) $1))
> +   (%type-assignment #'(lambda ($1) $1))
> +   (%value-assignment #'(lambda ($1) $1)))
> +  (%macro-definition
> +   (%macro-name
> +    macro
> +    \:\:=
> +    begin
> +    %general-list
> +    end
> +    #'(lambda ($1 $2 $3 $4 $5 $6)
> +	(declare (ignore $6 $5 $4 $3 $2))
> +	`(:macro ,$1))))
> +  (%value-assignment
> +   (:id
> +    %type
> +    \:\:=
> +    %value
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $3))
> +	`(:value-assignment ,$2 (,$1 ,$4))))
> +   (:id
> +    %macro-name
> +    %macro-arguments+
> +    \:\:=
> +    %object-identifier-value
> +    #'(lambda ($1 $2 $3 $4 $5)
> +	(declare (ignore $4))
> +	`(:define ,$2 (,$1 ,$3) ,$5)))
> +   (:id
> +    %macro-name
> +    %macro-arguments+
> +    \:\:=
> +    :number
> +    #'(lambda ($1 $2 $3 $4 $5)
> +	(declare (ignore $4))
> +	`(:define ,$2 (,$1 ,$3) ,$5))))
> +  (%macro-arguments+
> +   (%macro-arguments #'(lambda ($1) `(,$1)))
> +   (%macro-arguments+ %macro-arguments #'(lambda ($1 $2) `(,@$1 , 
> $2))))
> +  (%macro-arguments
> +   (:id :id #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (:id :string #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (syntax %type #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (write-syntax %type #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (object :id #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (module
> +    :id
> +    %macro-arguments+
> +    #'(lambda ($1 $2 $3) (declare (ignore $1)) `(:module ,$2 ,$3)))
> +   (module
> +    %macro-arguments+
> +    #'(lambda ($1 $2) (declare (ignore $1)) `(:module nil ,$2)))
> +   (:id
> +    {
> +    {
> +    %symbol*
> +    }
> +    }
> +    #'(lambda ($1 $2 $3 $4 $5 $6)
> +	(declare (ignore $6 $5 $3 $2))
> +	`(,$1 ,$4)))
> +   (:id
> +    {
> +    :string
> +    }
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3)))
> +   (:id
> +    {
> +    :number
> +    }
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3)))
> +   (:id
> +    {
> +    %implied-symbol+
> +    }
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3))))
> +  (%type-assignment
> +   (:id
> +    \:\:=
> +    %type
> +    #'(lambda ($1 $2 $3)
> +	(declare (ignore $2))
> +	`(:type-assignment ,$1 ,$3))))
> +  (%type
> +   (%builtin-type #'(lambda ($1) $1))
> +   (%tagged-type #'(lambda ($1) $1))
> +   (:id #'(lambda ($1) $1))
> +   (:id
> +    \(
> +    size
> +    \(
> +    %numbers+
> +    \)
> +    \)
> +    #'(lambda ($1 $2 $3 $4 $5 $6 $7)
> +	(declare (ignore $7 $6 $4 $3 $2))
> +	`(:general-string ,$1 ,$5)))
> +   (:id
> +    \(
> +    %numbers+
> +    \)
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $4 $2))
> +	`(:general-integer ,$1 ,$3)))
> +   (:id
> +    {
> +    %named-number+
> +    }
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $4 $2))
> +	`(:general-integer ,$1 ,$3)))
> +   (%named-type #'(lambda ($1) `(:named-type ,$1))))
> +  (%named-type (:id %type #'(lambda ($1 $2) `(,$1 ,$2))))
> +  (%builtin-type
> +   (%object-identifier-type #'(lambda ($1) $1))
> +   (%choice-type #'(lambda ($1) $1))
> +   (%string-type #'(lambda ($1) $1))
> +   (%integer-type #'(lambda ($1) $1))
> +   (%sequence-of-type #'(lambda ($1) $1))
> +   (%sequence-type #'(lambda ($1) $1))
> +   (%textual-convention-type #'(lambda ($1) $1))
> +   (null #'(lambda ($1) (declare (ignore $1)) '(:null))))
> +  (%object-identifier-type
> +   (object
> +    identifier
> +    #'(lambda ($1 $2) (declare (ignore $2 $1)) :object-identifier)))
> +  (%choice-type
> +   (choice
> +    {
> +    %alternative-type-lists
> +    }
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $4 $2 $1))
> +	`(:choice ,$3))))
> +  (%alternative-type-lists
> +   (%root-alternative-type-list #'(lambda ($1) $1)))
> +  (%root-alternative-type-list
> +   (%alternative-type-list #'(lambda ($1) $1)))
> +  (%alternative-type-list
> +   (%named-type #'(lambda ($1) `(,$1)))
> +   (%alternative-type-list
> +    \,
> +    %named-type
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3))))
> +  (%string-type
> +   (octet
> +    string
> +    %string-options
> +    #'(lambda ($1 $2 $3)
> +	(declare (ignore $2 $1))
> +	`(:octet-string ,$3))))
> +  (%string-options
> +   (\(
> +    size
> +    \(
> +    %numbers+
> +    \)
> +    \)
> +    #'(lambda ($1 $2 $3 $4 $5 $6)
> +	(declare (ignore $6 $5 $3 $2 $1))
> +	`(:size ,$4)))
> +   (\(
> +    %numbers+
> +    \)
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $1)) `(:size ,$2)))
> +   nil)
> +  (%numbers+
> +   (%numbers+
> +    \|
> +    %splited-numbers
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3)))
> +   (%splited-numbers #'(lambda ($1) `(,$1))))
> +  (%integer-type
> +   (%integer-type-name
> +    \(
> +    %numbers+
> +    \)
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3)))
> +   (%integer-type-name
> +    {
> +    %named-number+
> +    }
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3)))
> +   (%integer-type-name #'(lambda ($1) $1)))
> +  (%integer-type-name
> +   (integer #'(lambda ($1) (declare (ignore $1)) :integer)))
> +  (%splited-numbers
> +   (:number #'(lambda ($1) $1))
> +   (:number
> +    |..|
> +    :number
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,$1 ,$3))))
> +  (%named-number+
> +   (%named-number #'(lambda ($1) `(,$1)))
> +   (%named-number+
> +    \,
> +    %named-number
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3))))
> +  (%named-number
> +   (:id
> +    \(
> +    :number
> +    \)
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3))))
> +  (%tagged-type
> +   (%tag
> +    implicit
> +    %builtin-type
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(:implicit ,$1 ,$3)))
> +   (%tag
> +    explicit
> +    %builtin-type
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(:explicit ,$1 ,$3)))
> +   (%tag %builtin-type #'(lambda ($1 $2) `(:tag ,$1 ,$2))))
> +  (%tag
> +   ([
> +    %class
> +    :number
> +    ]
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $1)) `(,$2 ,$3))))
> +  (%class
> +   (universal #'(lambda ($1) (declare (ignore $1)) :universal))
> +   (application #'(lambda ($1) (declare (ignore $1)) :application))
> +   (private #'(lambda ($1) (declare (ignore $1)) :private))
> +   nil)
> +  (%value
> +   (%object-identifier-value #'(lambda ($1) $1))
> +   (:string #'(lambda ($1) $1))
> +   (:number #'(lambda ($1) $1)))
> +  (%object-identifier-value
> +   ({
> +    %obj-id-component+
> +    }
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $1)) `(,@$2))))
> +  (%obj-id-component+
> +   (%obj-id-component+ %obj-id-component #'(lambda ($1 $2) `(,@$1 , 
> $2)))
> +   (%obj-id-component #'(lambda ($1) `(,$1))))
> +  (%obj-id-component
> +   (%name-and-number-form #'(lambda ($1) $1))
> +   (:id #'(lambda ($1) $1))
> +   (:number #'(lambda ($1) $1)))
> +  (%name-and-number-form
> +   (:id
> +    \(
> +    :number
> +    \)
> +    #'(lambda ($1 $2 $3 $4) (declare (ignore $4 $2)) `(,$1 ,$3))))
> +  (%sequence-of-type
> +   (sequence
> +    of
> +    %type
> +    #'(lambda ($1 $2 $3) (declare (ignore $2 $1)) `(:sequence-of , 
> $3))))
> +  (%sequence-type
> +   (sequence
> +    {
> +    }
> +    #'(lambda ($1 $2 $3) (declare (ignore $3 $2 $1)) '(:sequence)))
> +   (sequence
> +    {
> +    %component-type-lists
> +    }
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $4 $2 $1))
> +	`(:sequence ,@$3))))
> +  (%component-type-lists (%root-component-type-list #'(lambda ($1)  
> $1)))
> +  (%root-component-type-list (%component-type-list #'(lambda ($1)  
> $1)))
> +  (%component-type-list
> +   (%component-type #'(lambda ($1) `(,$1)))
> +   (%component-type-list
> +    \,
> +    %component-type
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3))))
> +  (%component-type
> +   (%named-type
> +    optional
> +    #'(lambda ($1 $2) (declare (ignore $2)) `(,$1 :optional)))
> +   (%named-type
> +    default
> +    %value
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,$1 :default ,$3)))
> +   (%named-type #'(lambda ($1) $1))
> +   (components
> +    of
> +    %type
> +    #'(lambda ($1 $2 $3)
> +	(declare (ignore $2 $1))
> +	`(:components-of ,$3))))
> +  (%textual-convention-type
> +   (textual-convention
> +    %tc-args
> +    syntax
> +    %type
> +    #'(lambda ($1 $2 $3 $4)
> +	(declare (ignore $3 $1))
> +	`(:textual-convention ,$2 (:syntax ,$4)))))
> +  (%tc-args
> +   (%tc-arg #'(lambda ($1) `(,$1)))
> +   (%tc-args %tc-arg #'(lambda ($1 $2) `(,@$1 ,$2))))
> +  (%tc-arg
> +   (:id :id #'(lambda ($1 $2) `(,$1 ,$2)))
> +   (:id :string #'(lambda ($1 $2) `(,$1 ,$2))))
> +  (%symbol+
> +   (%symbol #'(lambda ($1) `(,$1)))
> +   (%symbol+
> +    \,
> +    %symbol
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3))))
> +  (%implied-symbol+
> +   (%implied-symbol #'(lambda ($1) `(,$1)))
> +   (%implied-symbol+
> +    \,
> +    %implied-symbol
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3))))
> +  (%symbol*
> +   (%symbol #'(lambda ($1) `(,$1)))
> +   (%symbol*
> +    \,
> +    %symbol
> +    #'(lambda ($1 $2 $3) (declare (ignore $2)) `(,@$1 ,$3)))
> +   nil)
> +  (%symbol (%macro-name #'(lambda ($1) $1)) (:id #'(lambda ($1) $1)))
> +  (%implied-symbol
> +   (:id #'(lambda ($1) $1))
> +   (implied
> +    :id
> +    #'(lambda ($1 $2) (declare (ignore $1)) `(:implied ,$2))))
> +  (%macro-name
> +   (module-identity #'(lambda ($1) $1))
> +   (object-type #'(lambda ($1) $1))
> +   (notification-type #'(lambda ($1) $1))
> +   (textual-convention #'(lambda ($1) $1))
> +   (module-compliance #'(lambda ($1) $1))
> +   (object-group #'(lambda ($1) $1))
> +   (notification-group #'(lambda ($1) $1))
> +   (object-identity #'(lambda ($1) $1))
> +   (agent-capabilities #'(lambda ($1) $1))
> +   (trap-type #'(lambda ($1) $1)))
> +  (%general-list
> +   (%general #'(lambda ($1) `(,$1)))
> +   (%general-list
> +    \,
> +    %general
> +    #'(lambda ($1 $2 $3) (declare (ignore $3)) `(,@$1 ,$2)))
> +   (%general-list %general #'(lambda ($1 $2) `(,@$1 ,$2))))
> +  (%general
> +   (:number #'(lambda ($1) $1))
> +   (:id #'(lambda ($1) $1))
> +   (:string #'(lambda ($1) $1))
> +   (\:\:= #'(lambda ($1) $1))
> +   (\( #'(lambda ($1) $1))
> +   (\) #'(lambda ($1) $1))
> +   (\| #'(lambda ($1) $1))
> +   ({ #'(lambda ($1) $1))
> +   (} #'(lambda ($1) $1))
> +   (< #'(lambda ($1) $1))
> +   (> #'(lambda ($1) $1))
> +   (type #'(lambda ($1) $1))
> +   (value #'(lambda ($1) $1))
> +   (notation #'(lambda ($1) $1))
> +   (sequence #'(lambda ($1) $1))
> +   (object #'(lambda ($1) $1))
> +   (identifier #'(lambda ($1) $1))
> +   (integer #'(lambda ($1) $1))
> +   (|IA5String| #'(lambda ($1) $1))))
> +
> +
> +
> Index: compiler/syntax.lisp
> ===================================================================
> --- compiler/syntax.lisp	(revision 706)
> +++ compiler/syntax.lisp	(working copy)
> @@ -44,6 +44,7 @@
> (eval-when (:load-toplevel :execute)
>   (fill-reserved-words))
>
> +#+lispworks
> (defparser asn.1-parser
>   ((%root %module-definition) $1)
>   ((root assignment) $1)
>
>
> For reference here is the patch to cl-yacc needed to resolve the
> ambiguities in the same way as parsergen does.
>
> diff -rN -u old-cl-yacc/yacc.lisp new-cl-yacc/yacc.lisp
> --- old-cl-yacc/yacc.lisp	2009-02-06 14:04:14.000000000 +0900
> +++ new-cl-yacc/yacc.lisp	2009-02-06 14:04:14.000000000 +0900
> @@ -808,9 +808,10 @@
> (define-condition conflict-warning (yacc-compile-warning simple- 
> warning)
>   ((kind :initarg :kind :reader conflict-warning-kind)
>    (state :initarg :state :reader conflict-warning-state)
> -   (terminal :initarg :terminal :reader conflict-warning-terminal))
> +   (terminal :initarg :terminal :reader conflict-warning-terminal)
> +   (chosen-action :initarg :chosen-action :reader conflict-warning- 
> chosen-action :initform nil))
>   (:report (lambda (w stream)
> -             (format stream "~A conflict on terminal ~S in state  
> ~A, ~_~?"
> +             (format stream "~A conflict on terminal ~S in state  
> ~A, ~_~?, ~@[taking action ~A~]"
>                      (case (conflict-warning-kind w)
>                        (:shift-reduce "Shift/Reduce")
>                        (:reduce-reduce "Reduce/Reduce")
> @@ -818,7 +819,8 @@
>                      (conflict-warning-terminal w)
>                      (conflict-warning-state w)
>                      (simple-condition-format-control w)
> -                     (simple-condition-format-arguments w)))))
> +                     (simple-condition-format-arguments w)
> +		     (conflict-warning-chosen-action w)))))
>
> (define-condition conflict-summary-warning (yacc-compile-warning)
>   ((shift-reduce :initarg :shift-reduce
> @@ -885,19 +887,26 @@
>                (if (tailp op2-tail (cdr op1-tail))
>                    (values a1 0 0)
>                    (values a2 0 0))))))))
> -    ;; default: prefer shift or first production
> -    (unless muffle-conflicts
> -      (warn (make-condition
> -             'conflict-warning
> -             :kind (typecase a1
> +    ;; default: prefer first defined production
> +    (let ((kind (typecase a1
>                      (shift-action :shift-reduce)
> -                     (t :reduce-reduce))
> +                     (t :reduce-reduce)))
> +	  (chosen-action
> +	   (if (production< p1 p2)
> +	       a1
> +	      a2)))
> +      (unless muffle-conflicts
> +	(warn (make-condition
> +             'conflict-warning
> +             :kind kind
>              :state id :terminal s
>              :format-control "~S and ~S~@[ ~_~A~]~@[ ~_~A~]"
> -             :format-arguments (list a1 a2 p1 p2))))
> -    (typecase a1
> -      (shift-action (values a1 1 0))
> -      (t (values a1 0 1)))))
> +             :format-arguments (list a1 a2 p1 p2)
> +	     :chosen-action chosen-action)))
> +      (values chosen-action
> +	      (if (eq kind :shift-reduce) 1 0)
> +	      (if (eq kind :reduce-reduce) 1 0)))))
> +
>
> (defun compute-parsing-tables (kernels grammar
>                                &key muffle-conflicts)
>
>
>
> And if anybody is interested in converting from parsergen syntax to
> cl-yacc syntax, here are some helper functions
>
> (defun defparser-production-to-yacc (grammar-symbols forms)
>  (cond ((not (and forms (or (not (listp forms)) (some 'identity  
> forms))))
> 	 nil)
> 	(t
> 	 (labels ((make-var (i)
> 		    (intern (format nil "$~D" i)))
> 		  (vars ()
> 		    (loop for i from 1 for x in grammar-symbols collect (make-var  
> i)))
> 		  (used-vars ()
> 	     (remove-duplicates
> 	      (loop for sym in (alexandria:flatten forms)
> 		    when (and (symbolp sym) (eql #\$ (elt (symbol-name sym) 0))
> 			      (ignore-errors (parse-integer (symbol-name sym) :start 1)))
> 		    collect (make-var (parse-integer (symbol-name sym) :start 1)))
> 	      :test 'eql)))
> 	   (let ((unused-vars (set-difference (vars) (used-vars))))
> 	     (list `#'(lambda(,@(vars))
> 		   ,@(when unused-vars  (list `(declare (ignore ,@unused-vars))))
> 		   ,forms)))))))
>
> (defun defparser-to-yacc (rules)
>  (let (grouped-rules)
>    (loop for rule in rules do
> 	  (destructuring-bind ((non-terminal &rest grammar-symbols)  
> &optional forms)
> 	      rule
> 	    (assert non-terminal)
> 	    (push (append grammar-symbols (defparser-production-to-yacc  
> grammar-symbols forms))
> 		  (sys:cdr-assoc non-terminal grouped-rules))))
>    (loop for (name . alternatives) in (nreverse grouped-rules)
> 	  collect `(,name ,@(reverse alternatives)))))

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China

Re: [lisp-snmp] Patch to asn.1 to change from parsergen to a modified cl-yacc

[Chun T. (b. <bin...@gm...>]

Hi, myself

On 2009-2-6, at 20:01, Chun Tian (binghe) wrote:
>
> ASN.1 50 > (oid "ucdSnmpAgent.linux")
> #<OBJECT-ID NET-SNMP-TC::linux (10) [0]>

This is a bug ... I'll fix it.

--
Chun Tian (binghe)
NetEase.com, Inc.
P. R. China