CEDET

Hi Eric,

[...]
 > The big question here, however, is if wisent can use these out of the
 > box, or does it need a wrapper.
 >
 > I'd prefer it if wisent can use it out of the box.  If we need to
 > change the bovine format some, that's ok.  This will let us change
 > the token format safely in the future.

For now wisent's grammars use `wisent-token' and `wisent-cooked-token'
to produce semantic tokens.  These functions are one level higher than
the semantic-token API, close to the `semantic-lambda' feature of the
bovinator.

However, I think it would be possible for wisent to take benefit of
the new semantic-token API, if we add the following new functions in
wisent-bovine.el.  Contrary to `wisent-token' and
`wisent-cooked-token', these new functions are passed a semantic
token instead of a list of values.

(defsubst wisent-raw-token (semantic-token)
   "Return a raw token from given SEMANTIC-TOKEN.
Should be used in semantic actions to build the Semantic parse tree."
   (nconc semantic-token
          (if (or $region
                  (setq $region (nthcdr 2 wisent-input)))
              (list (car $region) (cdr $region))
            (list (point-max) (point-max)))))

(defsubst wisent-cook-token (semantic-token)
   "Cook SEMANTIC-TOKEN and return a list of cooked tokens.
Should be used in semantic actions to build the Semantic parse tree."
   (let* ((cooked (semantic-raw-to-cooked-token
                   (wisent-raw-token semantic-token)))
          (l cooked))
     (while l
       (semantic-token-put (car l) 'reparse-symbol $nterm)
       (setq l (cdr l)))
     cooked))

So it would be possible to replace things like:

(wisent-token $3 'variable $2 nil
               (semantic-bovinate-make-assoc-list 'typemodifiers $1)
               nil)

(wisent-cooked-token $3 'variable $2 nil
                      (semantic-bovinate-make-assoc-list 'typemodifiers $1)
                      nil)

By respectively the equivalent better forms:

(wisent-raw-token
   (semantic-token-new-variable $3 $2 nil 'typemodifiers $1))

(wisent-cook-token
   (semantic-token-new-variable $3 $2 nil 'typemodifiers $1))

If we change the internal format of semantic tokens, we just have to
adapt the above two functions to take it into account.

What do you think?

David

>>> David Ponce <david.ponce@...> seems to think that:
  [ ... ]
>So it would be possible to replace things like:
>
>(wisent-token $3 'variable $2 nil
>               (semantic-bovinate-make-assoc-list 'typemodifiers $1)
>               nil)
>
>(wisent-cooked-token $3 'variable $2 nil
>                      (semantic-bovinate-make-assoc-list 'typemodifiers $1)
>                      nil)
>
>By respectively the equivalent better forms:
>
>(wisent-raw-token
>   (semantic-token-new-variable $3 $2 nil 'typemodifiers $1))
>
>(wisent-cook-token
>   (semantic-token-new-variable $3 $2 nil 'typemodifiers $1))
>
>If we change the internal format of semantic tokens, we just have to
>adapt the above two functions to take it into account.
>
>What do you think?
  [ ... ]

Thanks for looking into this.

I had been looking at wisent-token to see what needed to happen, and
wasn't sure.  Your solution is certainly quite simple, and could
probably be made to handle both situations for a short time.

What I am hoping to do is simplify the code in grammar files as much
as possible.  That's how the .bnf file format was eventually built.
I wanted the lisp code in there as simple as possible.

Here are examples:
* semantic-lambda auto-adds start/end.
* All rules return the default return of the action
* ASSOC, EXPAND, EXPANDFULL macros

As it stands, typing in `semantic-token-new-variable' seems long to
me.  Perhaps we can have some sort of macro like `TOKEN-VARIABLE' or
`TOKEN' that expands to either what .by or .wy needs.  This will
simplify token code, allow for eldoc help via existing mechanisms
(as ASSOC was done in .bnf files) and let us keep using some common
piece of code.

Does that muddy things up in wisent too much?
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

 > As it stands, typing in `semantic-token-new-variable' seems long to
 > me.  Perhaps we can have some sort of macro like `TOKEN-VARIABLE' or
 > `TOKEN' that expands to either what .by or .wy needs.  This will
 > simplify token code, allow for eldoc help via existing mechanisms
 > (as ASSOC was done in .bnf files) and let us keep using some common
 > piece of code.

That looks like a very good idea!  As macro expanders are specific
to each grammar mode, it would not be hard to expand macro to what
the parser need.

Maybe using names like VARIABLE-TAG instead of TOKEN-VARIABLE would
be less implementation dependent?

Also, when using rule based iteration in LALR grammars, it can be
necessary to produce cooked tokens instead of raw ones.  In such cases
we could use an `EXPANDTAG' macro like this:

(EXPANDTAG (VARIABLE-TAG $3 nil $2 'const t))

In WY grammars the macro expansion would be straightforward:

VARIABLE-TAG and such would expand to:

(wisent-raw-token
  (semantic-token-new-variable $3 nil $2 'const t))

And,

(EXPANDTAG (VARIABLE-TAG $3 nil $2 'const t))

would expand to:

(wisent-cook-token
  (wisent-raw-token
   (semantic-token-new-variable $3 nil $2 'const t)))

What do you think?

David

>>> David Ponce <david.ponce@...> seems to think that:
>Hi Eric,
>
> > As it stands, typing in `semantic-token-new-variable' seems long to
> > me.  Perhaps we can have some sort of macro like `TOKEN-VARIABLE' or
> > `TOKEN' that expands to either what .by or .wy needs.  This will
> > simplify token code, allow for eldoc help via existing mechanisms
> > (as ASSOC was done in .bnf files) and let us keep using some common
> > piece of code.
>
>That looks like a very good idea!  As macro expanders are specific
>to each grammar mode, it would not be hard to expand macro to what
>the parser need.
>
>Maybe using names like VARIABLE-TAG instead of TOKEN-VARIABLE would
>be less implementation dependent?
>
>Also, when using rule based iteration in LALR grammars, it can be
>necessary to produce cooked tokens instead of raw ones.  In such cases
>we could use an `EXPANDTAG' macro like this:
>
>(EXPANDTAG (VARIABLE-TAG $3 nil $2 'const t))
  [ ... ]

Hmmm.  Perhaps we need to think about our nomenclature more.
Unfortunately when I first wrote semantic I wasn't all that familiar
with the nomenclature of parsers.  Right now we have:

token - A lexical entity
token - A semantic parsed entity, like a tag.
tag - used in descriptions and doc to describe a semantic parsed entity
nonterminal - another semantic parsed entity
nonterminal - a parser element in wisent
stream - a list of semantic parsed entities
stream - a list of lexical entities

It seems to me we could settle on some names for anything new:

token - lexical
stream - list of lexical entities
tag - semantic
nonterminal - a parser element in wisent
table, tag-table, database? - list of semantic tokens.


So I would go back and change semantic-token.el to be -tag.el
instead.

We could also migrate some of the -nontermial APIs over to tag as
well, or live with the double meaning.

I was realizing the other day that the semanticdb search routines took
arguments in a way that was flexible (from a tool sense) and
inflexible from a user sense.  Changing everything to -tag, and
obsoleting the -nonterminal version of the searches would give me an
opportunity to rectify the problem.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>>>> "EL" == Eric M Ludlam <eric@...> writes:

    ...

    EL> It seems to me we could settle on some names for anything new:
    EL> 
    EL> token - lexical
    EL> stream - list of lexical entities
    EL> tag - semantic
    EL> nonterminal - a parser element in wisent
    EL> table, tag-table, database? - list of semantic tokens.

Hi Eric,

I think naming is terribly important in improving code
readability.  I think using the terms "token" and "tag" to
distinguish the two rather than "lexical token" and
"semantic token" respectively would significantly improve
code readability. 

PS: I've been swamped with multiple jobs and personal
related matters in the past week or two.  I am still working
on the semantic manual.  I'll get back to you guys ASAP.