CEDET

>>> "David PONCE" <David.Ponce@...> seems to think that:
>Hi Eric,
>
  [ ... ]
>> * Why does semantic's grammar need such a complicated number regexp?
>
>No more used now :-)

Good.

>> * Why does it need it's own symbol matcher?  Why not use the
>>   symbol-keyword matcher?
>
>First, it let me customize the regexp to suit grammar lexer needs (I
>just added the '.' in the regexp to fix your problem below).  Second,
>I prefer to use uppercase for nonterminal symbols ;-)
[.. < ]
>* semantic-grammar.el:
>
>(semantic-grammar-lex-symbol): Include '.' in symbol.

Why not make . a symbol constituent in the syntax table, or in the
syntax table modifiers for the .wy grammar?

Also, your regex is the same as `semantic-lex-symbol-or-keyword'.  If
you have preferenes with case sensitivity, we should
re-introduce/familiarize ourselves with the case-fold-search modifier
used during parsing.

>> * Why is the comment-start-skip so complicated?  This is used in the
>>   lexer, and that complexity probably slows things down.  Perhaps
>>   semantic-lex-comment-regex could be specified with something
>>   simpler?
>
>I just used the regexp from `emacs-lisp-mode'.

I thought I recognized it.  ;)

I recommend rewriting it, perhaps like this:

(setq semantic-lex-comment-regex ";;")

since grammars now require two semicolons to make a comment.
We can probably ignore the ?;; case if you modify the syntax table to
include ? as a quote character in the syntax table.

>>   After wondering these things, I discovered the real problem.
>> The lexer actually had a progress hang as soon as it hit the valid
>> bison/yacc rule:
>> 
>> .hush_warning: 
>> ;; action
>> ;
>> 
>> or any reference to .hush_warning.  This lead me to wonder about
>> `wisent-lex-punctuation' which just sort of confused me when I tried
>> to understand its purpose.
>
>This problem is solved now :-)

Yay!

>`wisent-lex-punctuation' automatically return lexical tokens with the
>terminal symbols associated to punctuations defined like this:
>
>%token <punctuation> SYMBOL "value"
>
>For example in semantic-grammar.wy the following punctuations are
>defined:
>
>%token <punctuation>   COLON       ":"
>%token <punctuation>   SEMI        ";"
>%token <punctuation>   OR          "|"
>%token <punctuation>   LT          "<"
>%token <punctuation>   GT          ">"
>%token <punctuation>   PERCENT     "%"
>
>Using `wisent-lex-punctuation', the lexer automatically return a
>COLON, SEMI, OR, LT, GT, PERCENT lexical token when it encounters a
>punctuation that respectively matches ":", ";", "|", "<", ">", "%".
>This is very convenient for LALR grammars, that often need to
>distinguish the different kind of punctuations.
>
>> In general, however, would it make sense to move this and others into
>> semantic-grammar.el or semantic-lex.el instead of wisent-bovine.el?
>
>As the `semantic-lex-token' stuff is already in semantic-lex, it makes
>sense to move `wisent-lex-punctuation' (as `semantic-lex-separator'?)
>in semantic-lex too.  If you are OK with that, I am willing to do it
>;-)

Sounds good.   Why the name `separator'?  I need to look more closely
at how lexical tokens are handled I think.  Following
`semantic-lex-symbol-or-keyword', perhaps the name
`semantic-lex-punctuation-or-... what?' would match.

>Anyway, I checked some changes in to improve the grammar lexer (the
>change log is below).

Yay!

>I converted the g++-parse.y file from the gcc sources, using the
>`bison->wisent' command.  Then it took me about 1.5 sec. (with my slow
>Celeron 366 Mhz) to successfully parse the resulting WY grammar :-)

Huzzah!  That's more like it.  Your parser totally rocks.

the g++ parse.y file has rules in it like this:

namespace_alias:
          NAMESPACE identifier '='
;; Action
          any_id ';'
;; Action
	;

where '=' and ';' are token literals.  Does wisent support this?  I
was pretty sure you had said you had removed this capability.  If it
is not supported, two options are to add support, or to make
bison->wisent perform a translation to names for you.

  [ ... ]
>* wisent/wisent-bovine.el:
>
>(wisent-lex-punctuation): Continue lexical analysis if a punctuation
>don't match.
  [ ... ]

Does this mean that '.' did not show up with a matched token name, so
the character was not skipped?

I think it would be good to have this create a generic punctuation
token when there isn't a punctuation keyword.  (See lame name above.)

Thanks!
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

>>(semantic-grammar-lex-symbol): Include '.' in symbol.
> 
> Why not make . a symbol constituent in the syntax table, or in the
> syntax table modifiers for the .wy grammar=3F

Good idea!  I am going to try that.

> Also, your regex is the same as `semantic-lex-symbol-or-keyword'.  If
> you have preferenes with case sensitivity, we should
> re-introduce/familiarize ourselves with the case-fold-search modifier
> used during parsing.

What I meant is that `semantic-grammar-lex-symbol' returns 'SYMBOL' as
token symbol instead of 'symbol'.  I prefer to use uppercase names for
lexical symbols as it is recommended for Bison grammars.

>>>* Why is the comment-start-skip so complicated=3F  This is used in the=

>>>  lexer, and that complexity probably slows things down.  Perhaps
>>>  semantic-lex-comment-regex could be specified with something
>>>  simpler=3F
>>
>>I just used the regexp from `emacs-lisp-mode'.
> 
> 
> I thought I recognized it.  ;)
> 
> I recommend rewriting it, perhaps like this:
> 
> (setq semantic-lex-comment-regex ";;")
> 
> since grammars now require two semicolons to make a comment.
> We can probably ignore the =3F;; case if you modify the syntax table to=

> include =3F as a quote character in the syntax table.

I am afraid, I don't understand your purpose here :-(
Currently `comment-start-skip' is:

"\\(\\(^\\|[^\\\\\n]\\)\\(\\\\\\\\\\)*\\);+ *"

I don't see any '=3F' in it.  As the comment says, the
regexp takes care of backslashes:

;; Look within the line for a ; following an even number of
;; backslashes after either a non-backslash or the line beginning.

>>As the `semantic-lex-token' stuff is already in semantic-lex, it makes
>>sense to move `wisent-lex-punctuation' (as `semantic-lex-separator'=3F)=

>>in semantic-lex too.  If you are OK with that, I am willing to do it
>>;-)
> 
> 
> Sounds good.   Why the name `separator'=3F  I need to look more closely=

> at how lexical tokens are handled I think.  Following
> `semantic-lex-symbol-or-keyword', perhaps the name
> `semantic-lex-punctuation-or-... what=3F' would match.

Just because `semantic-lex-punctuation' was already used ;-)
Maybe `semantic-lex-operator' is a better name=3F
Or `semantic-lex-punctuation-token'=3F

> the g++ parse.y file has rules in it like this:
> 
> namespace_alias:
>           NAMESPACE identifier '=3D'
> ;; Action
>           any_id ';'
> ;; Action
> =09;
> 
> where '=3D' and ';' are token literals.  Does wisent support this=3F  I=

> was pretty sure you had said you had removed this capability.  If it
> is not supported, two options are to add support, or to make
> bison->wisent perform a translation to names for you.

Character literals are completely supported as lexical tokens
(wisent-calc.wy/el is a working example).

>>* wisent/wisent-bovine.el:
>>
>>(wisent-lex-punctuation): Continue lexical analysis if a punctuation
>>don't match.
> 
>   [ ... ]
> 
> Does this mean that '.' did not show up with a matched token name, so
> the character was not skipped=3F

I am not sure to understand, but I fixed a nasty bug in the previous
version of `wisent-lex-punctuation' which returned an invalid
(nil START . END) token when no punctuation matched.  Now it returns
false to the match condition so the lexer can try the other analyzers.

> I think it would be good to have this create a generic punctuation
> token when there isn't a punctuation keyword.  (See lame name
> above.)

I don't think so.  IMO it is better to consider that the analyzer
failed to match, so it is possible to try other match with other
analyzers that can follow this one.  For example in semantic-grammar
if no regexp analyzer matched, the default analyzer is entered to
return a generic SEXP token that matches any s-expression, so it is
not necessary to handle specifically extra punctuations nor any other
kind of s-expressions like numbers :-)

Also, to do what you suggest, it is possible that
`wisent-lex-punctuation' and `semantic-lex-punctuation' follow each
other, with the [moderate] extra cost of a second (looking-at
"\\(\\s.\\|\\s$\\|\\s'\\)")).

Thanks!
David

Hi,

  I got your changes from CVS, and now my parse fails in the lexer on
  this rule:

new_initializer:
	  '(' nonnull_exprlist ')'
;; Action
	| LEFT_RIGHT
;; Action
	| '(' typespec ')'
;; Action
	;; GNU extension so people can use initializer lists.  Note that
	;; this alters the meaning of `new int = 1', which was previously
	;; syntactically valid but semantically invalid.
	;; This feature is now deprecated and will be removed in a future
	;; release.
	|;; '=' init
;; Action
	;

in particular, it dies on the '=' in the comment.
I changed the comment regexp for semantic only as I had specified in
my previous message and it started working again.

>>> "David PONCE" <David.Ponce@...> seems to think that:
  [ ... ]
>> Also, your regex is the same as `semantic-lex-symbol-or-keyword'.  If
>> you have preferenes with case sensitivity, we should
>> re-introduce/familiarize ourselves with the case-fold-search modifier
>> used during parsing.
>
>What I meant is that `semantic-grammar-lex-symbol' returns 'SYMBOL' as
>token symbol instead of 'symbol'.  I prefer to use uppercase names for
>lexical symbols as it is recommended for Bison grammars.

I see.  It makes sense, perhaps, to shift from lower to upper case.
It would be nice to have consistency.

As .y files do not have these conveniences with built in names,
it might make sense to eliminate them.  In the same vein, it is nice
to have conveniences when developing something new.

As I work on the C++ grammar, I'll see if there are other translations
to make, perhaps from some identifiable %tokens to built in names.
I'd have to look in the .lex file to figure that out though.  Ugh.

>>>>* Why is the comment-start-skip so complicated?  This is used in the
>>>>  lexer, and that complexity probably slows things down.  Perhaps
>>>>  semantic-lex-comment-regex could be specified with something
>>>>  simpler?
>>>
>>>I just used the regexp from `emacs-lisp-mode'.
>> 
>> 
>> I thought I recognized it.  ;)
>> 
>> I recommend rewriting it, perhaps like this:
>> 
>> (setq semantic-lex-comment-regex ";;")
>> 
>> since grammars now require two semicolons to make a comment.
>> We can probably ignore the ?;; case if you modify the syntax table to
>> include ? as a quote character in the syntax table.
>
>I am afraid, I don't understand your purpose here :-(
>Currently `comment-start-skip' is:
>
>"\\(\\(^\\|[^\\\\\n]\\)\\(\\\\\\\\\\)*\\);+ *"
>
>I don't see any '?' in it.  As the comment says, the
>regexp takes care of backslashes:

I refer to ?; in that that sequence does not start a comment in Emacs
Lisp.

>;; Look within the line for a ; following an even number of
>;; backslashes after either a non-backslash or the line beginning.

This makes some sense, but seems overly obtuse.  I seem to recall
submitting something like this as a bug in font-lock about 6 years
ago.  I was mostly worried about the lexical speed.

>>>As the `semantic-lex-token' stuff is already in semantic-lex, it makes
>>>sense to move `wisent-lex-punctuation' (as `semantic-lex-separator'?)
>>>in semantic-lex too.  If you are OK with that, I am willing to do it
>>>;-)
>> 
>> 
>> Sounds good.   Why the name `separator'?  I need to look more closely
>> at how lexical tokens are handled I think.  Following
>> `semantic-lex-symbol-or-keyword', perhaps the name
>> `semantic-lex-punctuation-or-... what?' would match.
>
>Just because `semantic-lex-punctuation' was already used ;-)
>Maybe `semantic-lex-operator' is a better name?
>Or `semantic-lex-punctuation-token'?

As we seem to be moving away from the old bovinator style of:

rule: punctuation "." ;

i think it makes sense to have the default `semantic-lex-punctuation'
return translated punctuation keyword things as part of it's job.

It may someday be necessary to just have a flex->semantic-lex
translator, and build a lexical analyzer that way.  I designed
Semantic's lexical analyzer to speak in syntax-table form.  This made
it easy for me to transition my Emacs knowledge into the realm of
parsers.  This also provided one analyzer for all languages.  If
general types as currently specified in the original bovinator aren't
useful anymore, perhaps it is time to abandon such thinking.

  [ ... ]
>> I think it would be good to have this create a generic punctuation
>> token when there isn't a punctuation keyword.  (See lame name
>> above.)
>
>I don't think so.  IMO it is better to consider that the analyzer
>failed to match, so it is possible to try other match with other
>analyzers that can follow this one.  For example in semantic-grammar
>if no regexp analyzer matched, the default analyzer is entered to
>return a generic SEXP token that matches any s-expression, so it is
>not necessary to handle specifically extra punctuations nor any other
>kind of s-expressions like numbers :-)
>
>Also, to do what you suggest, it is possible that
>`wisent-lex-punctuation' and `semantic-lex-punctuation' follow each
>other, with the [moderate] extra cost of a second (looking-at
>"\\(\\s.\\|\\s$\\|\\s'\\)")).
  [ ... ]

This makes sense.  Your two tier approach seems like a pretty good
one.  It could be used for keywords too I suppose. I babbled about
this a little bit above too.  I have no particular preferences here
other than to understand your reasoning.

Have fun
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

>   I got your changes from CVS, and now my parse fails in the lexer on
>   this rule:
[...]
> in particular, it dies on the '=3D' in the comment.
> I changed the comment regexp for semantic only as I had specified in
> my previous message and it started working again.

I committed a new version of semantic-grammar.el/wy that fixes that.
I finally followed your suggestions and set
`semantic-lex-comment-regex' to ";;", and added '=3F' and '\' as quote
characters in the character table ;-)

I also fixed some font lock regexps.  The change log is at end.

[...]
> I see.  It makes sense, perhaps, to shift from lower to upper case.
> It would be nice to have consistency.
> 
> As .y files do not have these conveniences with built in names,
> it might make sense to eliminate them.  In the same vein, it is nice
> to have conveniences when developing something new.

Agreed.

[...]
> As we seem to be moving away from the old bovinator style of:
> 
> rule: punctuation "." ;
> 
> i think it makes sense to have the default `semantic-lex-punctuation'
> return translated punctuation keyword things as part of it's job.

OK.  Maybe could we provide the current `semantic-lex-punctuation'
function as `semantic-lex-generic-punctuation'=3F

I will try to hack something this week-end ;-)

Thanks!
David

---------- Change log:

* semantic-grammar.el:

(semantic-grammar-lex-symbol): Undo previous change.
(semantic-grammar-automaton)
(semantic-grammar-keywords)
(semantic-grammar-tokens)
(semantic-grammar-setup-semantic): Re-generated.
(semantic-grammar-syntax-table): Added '.' as symbol constituent, and
'=3F' and '\' as quote characters.
(semantic-grammar-mode-keywords-1): Fixed symbol regexps.

* semantic-grammar.wy:

(setupcode): Setup `semantic-lex-comment-regex'.

Hi,

>>> "David PONCE" <David.Ponce@...> seems to think that:
>Hi Eric,
>
>>   I got your changes from CVS, and now my parse fails in the lexer on
>>   this rule:
>[...]
>> in particular, it dies on the '=' in the comment.
>> I changed the comment regexp for semantic only as I had specified in
>> my previous message and it started working again.
>
>I committed a new version of semantic-grammar.el/wy that fixes that.
>I finally followed your suggestions and set
>`semantic-lex-comment-regex' to ";;", and added '?' and '\' as quote
>characters in the character table ;-)
>
>I also fixed some font lock regexps.  The change log is at end.

Thanks!

  [ ... ]
>[...]
>> As we seem to be moving away from the old bovinator style of:
>> 
>> rule: punctuation "." ;
>> 
>> i think it makes sense to have the default `semantic-lex-punctuation'
>> return translated punctuation keyword things as part of it's job.
>
>OK.  Maybe could we provide the current `semantic-lex-punctuation'
>function as `semantic-lex-generic-punctuation'?

That makes sense.

  [ ... ]
>* semantic-grammar.el:
>
>(semantic-grammar-lex-symbol): Undo previous change.

What if there is a new punctuation problem we have not yet found?
I think you may want to throw an error, like "unexpected punctuation
character %c" instead.

>(semantic-grammar-automaton)
>(semantic-grammar-keywords)
>(semantic-grammar-tokens)
>(semantic-grammar-setup-semantic): Re-generated.
>(semantic-grammar-syntax-table): Added '.' as symbol constituent, and
>'?' and '\' as quote characters.
>(semantic-grammar-mode-keywords-1): Fixed symbol regexps.

Wouldn't making . be a symbol constituent fix this problem
automatically?

Thanks!
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org