Re: [CEDET-devel] macros in .[bw]y files

[David P. <dav...@wa...>]

Hi Eric,

[...]
 > I think I've finally finished eradicating all old poor uses of
 > "token", so I'll do that next.

Thanks!  I saw you did a great work!

[...]
 >>>This seems problematic though.  Perhaps the association of the string
 >>>"TAG" to 'foo-tag needs to somehow be in the grammar since it is the
 >>>parser-generator which uses that string, and nothing else.
 >>
 >>I thought to that too.  Perhaps could we use a new %expandmacro
 >>keyword like this:
 >>
 >>%expandmacro TAG foo-TAG
 >>
 >>To generate association of a grammar macro with an expander?
 >
 >
 > That sounds like a good idea.  It implies the need for a "%include",
 > or some other cleverness indicating what set of macro expand keys
 > should be loaded.
 >
 > Perhaps it would instead be declarative, such as "%gererate tags" or
 > "%macroset font-lock" or "%import indent"

I worked a little more on that and implemented a solution that seems
to work :-)

For now I created a new %expanders statement that looks like this:

%expanders (
             (ASSOC          . semantic-grammar-ASSOC)
             (EXPAND         . wisent-grammar-EXPAND)
             (EXPANDFULL     . wisent-grammar-EXPANDFULL)
             (TAG            . wisent-grammar-TAG)
             (VARIABLE-TAG   . wisent-grammar-VARIABLE-TAG)
             (FUNCTION-TAG   . wisent-grammar-FUNCTION-TAG)
             (TYPE-TAG       . wisent-grammar-TYPE-TAG)
             (INCLUDE-TAG    . wisent-grammar-INCLUDE-TAG)
             (PACKAGE-TAG    . wisent-grammar-PACKAGE-TAG)
             (EXPANDTAG      . wisent-grammar-EXPANDTAG)
             (CODE-TAG       . wisent-grammar-CODE-TAG)
             (AST-ADD        . wisent-grammar-AST-ADD)
             (AST-PUT        . wisent-grammar-AST-PUT)
             (AST-GET        . wisent-grammar-AST-GET)
             (AST-GET1       . wisent-grammar-AST-GET1)
             (AST-GET-STRING . wisent-grammar-AST-GET-STRING)
             (AST-MERGE      . wisent-grammar-AST-MERGE)
             )

It associates grammar macro names with function expanders.  At
generation these macros are expanded using the given functions.  I
also hacked the generation process to evaluate the prologue before
expansion, so it is possible to simply use a `require' statement to
include macro expanders:

%{
   (require 'wisent-grammar-macros)
   %}

A little inconvenient with that scheme, is that macro definitions are
also required at run time (copied from prologue in the generated lisp
file), even if not necessary.

I also provided a new `semantic-grammar-macros' variable that can be
used to provide default definitions for commonly used macros.  For
example I defined the following default for WY grammars:

(defvar-mode-local wisent-grammar-mode semantic-grammar-macros
   '(
     (ASSOC          . semantic-grammar-ASSOC)
     (EXPAND         . wisent-grammar-EXPAND)
     (EXPANDFULL     . wisent-grammar-EXPANDFULL)
     (TAG            . wisent-grammar-TAG)
     (VARIABLE-TAG   . wisent-grammar-VARIABLE-TAG)
     (FUNCTION-TAG   . wisent-grammar-FUNCTION-TAG)
     (TYPE-TAG       . wisent-grammar-TYPE-TAG)
     (INCLUDE-TAG    . wisent-grammar-INCLUDE-TAG)
     (PACKAGE-TAG    . wisent-grammar-PACKAGE-TAG)
     (EXPANDTAG      . wisent-grammar-EXPANDTAG)
     (CODE-TAG       . wisent-grammar-CODE-TAG)
     (AST-ADD        . wisent-grammar-AST-ADD)
     (AST-PUT        . wisent-grammar-AST-PUT)
     (AST-GET        . wisent-grammar-AST-GET)
     (AST-GET1       . wisent-grammar-AST-GET1)
     (AST-GET-STRING . wisent-grammar-AST-GET-STRING)
     (AST-MERGE      . wisent-grammar-AST-MERGE)
     )
   "Definitions of macros used in WY grammars.")

I enhanced grammar font-locking to automagically highlight macro names
defined in both `semantic-grammar-macros' and in the %expanders clause
in the current buffer!

Now that macros are no more hard coded, I would like to replace the
expanders statement with something like this:

%with-macros (TAG EXPAND ...) wisent-grammar-macro
...
%with-macros (ASSOC) semantic-grammar

The %with-macros statement would indicate to expand the given list of
macros using definitions loaded from the indicated library.  The
names of expander functions would be the name of the macro prefixed
with the library name.  In the above examples that would give these
associations:

(TAG    . wisent-grammar-macro-TAG)
(EXPAND . wisent-grammar-macro-EXPAND)
...
(ASSOC  . semantic-grammar-ASSOC)

The advantage would be simplicity, standardized naming, and possible
load of macro libraries at generation time only (should be easy to
implement).

It could be worth adopting a similar consistent form to setup default
macro expanders by customizing `semantic-grammar-macros':

(defvar-mode-local wisent-grammar-mode semantic-grammar-macros
   '(
     ((TAG EXPAND ...) wisent-grammar-macro)
     ...
     ((ASSOC) semantic-grammar)
     )
   )

Finally, I must look at the bovine grammar generator to see how it
could benefit of a such improved macro architecture ;-)

What do you think?

[...]
 >>I terminated conversion to the new grammar layout, and checked-in a
 >>bunch of changes!  Following is the change log.  Hope these changes
 >>didn't break things ;-)
 >
 >
 > I need to updated the EDE project to build a Makefile to create the
 > intermediate files, such as:
 >
 > %.el: %.wy
 >       $(EMACS) -f wisent-something $@
 >
 > so that wisent-awk.el is created from wisent-awk.wy.
 >
 > Do you have somethingalready?  If not, I can create something to get
 > the Makefiles working again.

Oops! I forgot the EDE stuff.  Sorry!

I see a problem here with the %.el: %.wy rule because the file names
are not necessarily identical.  The current rule is that the generated
lisp file name is the one indicated in the grammar %package statement
if provided, or a default generated name otherwise.  The default name
is generated like this:

foo.[wb]y -> foo-[wb]y.el

More generally:

foo.<ext> -> foo-<ext>.el

The problem is with an explicit %package name, like used in bovine
grammars.  For example c.by uses a "%package bovine-c" and generates a
bovine-c-by.el file.  How it is possible to handle that with make?

What I have to do too, is to allow `semantic-grammar-create-package'
to work in batch mode, to allow:

$(EMACS) -f semantic-grammar-create-package $@

Unfortunately, I will be very busy during the rest of August and probably
September too.  So I probably will not be able to do things quickly :-(

David