cedet-devel Mailing List for CEDET (Page 290)
Brought to you by:
zappo
You can subscribe to this list here.
2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(2) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2002 |
Jan
(5) |
Feb
|
Mar
(3) |
Apr
|
May
(32) |
Jun
(75) |
Jul
(45) |
Aug
(77) |
Sep
(28) |
Oct
(10) |
Nov
(18) |
Dec
(49) |
2003 |
Jan
(98) |
Feb
(116) |
Mar
(111) |
Apr
(99) |
May
(29) |
Jun
(8) |
Jul
(48) |
Aug
(85) |
Sep
(61) |
Oct
(16) |
Nov
(70) |
Dec
(31) |
2004 |
Jan
(50) |
Feb
(74) |
Mar
(151) |
Apr
(76) |
May
(36) |
Jun
(91) |
Jul
(42) |
Aug
(26) |
Sep
(32) |
Oct
(38) |
Nov
(21) |
Dec
(35) |
2005 |
Jan
(78) |
Feb
(46) |
Mar
(25) |
Apr
(68) |
May
(47) |
Jun
(36) |
Jul
(42) |
Aug
(13) |
Sep
(12) |
Oct
(11) |
Nov
(12) |
Dec
(5) |
2006 |
Jan
(15) |
Feb
(9) |
Mar
(9) |
Apr
(10) |
May
(15) |
Jun
(29) |
Jul
(32) |
Aug
(17) |
Sep
(14) |
Oct
(5) |
Nov
(1) |
Dec
(4) |
2007 |
Jan
(17) |
Feb
(60) |
Mar
(39) |
Apr
(7) |
May
(23) |
Jun
(30) |
Jul
(28) |
Aug
(34) |
Sep
(9) |
Oct
(9) |
Nov
(9) |
Dec
(9) |
2008 |
Jan
(18) |
Feb
(38) |
Mar
(33) |
Apr
(35) |
May
(39) |
Jun
(68) |
Jul
(31) |
Aug
(32) |
Sep
(16) |
Oct
(19) |
Nov
(17) |
Dec
(33) |
2009 |
Jan
(83) |
Feb
(104) |
Mar
(214) |
Apr
(156) |
May
(104) |
Jun
(55) |
Jul
(127) |
Aug
(58) |
Sep
(58) |
Oct
(58) |
Nov
(48) |
Dec
(28) |
2010 |
Jan
(46) |
Feb
(135) |
Mar
(97) |
Apr
(52) |
May
(75) |
Jun
(31) |
Jul
(35) |
Aug
(51) |
Sep
(52) |
Oct
(107) |
Nov
(71) |
Dec
(15) |
2011 |
Jan
(24) |
Feb
(49) |
Mar
(107) |
Apr
(110) |
May
(28) |
Jun
(63) |
Jul
(28) |
Aug
(37) |
Sep
(29) |
Oct
(24) |
Nov
(46) |
Dec
(44) |
2012 |
Jan
(79) |
Feb
(103) |
Mar
(67) |
Apr
(81) |
May
(29) |
Jun
(70) |
Jul
(39) |
Aug
(21) |
Sep
(54) |
Oct
(75) |
Nov
(72) |
Dec
(86) |
2013 |
Jan
(72) |
Feb
(38) |
Mar
(131) |
Apr
(8) |
May
(31) |
Jun
(3) |
Jul
(5) |
Aug
(39) |
Sep
(38) |
Oct
(41) |
Nov
(43) |
Dec
(37) |
2014 |
Jan
(12) |
Feb
(47) |
Mar
(36) |
Apr
(9) |
May
(24) |
Jun
(50) |
Jul
(19) |
Aug
(26) |
Sep
(27) |
Oct
(21) |
Nov
(12) |
Dec
(26) |
2015 |
Jan
(83) |
Feb
(58) |
Mar
(34) |
Apr
(26) |
May
(6) |
Jun
(8) |
Jul
(2) |
Aug
(18) |
Sep
(1) |
Oct
(27) |
Nov
(7) |
Dec
(2) |
2016 |
Jan
(9) |
Feb
(4) |
Mar
(17) |
Apr
(3) |
May
|
Jun
(8) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
(4) |
Nov
|
Dec
|
2017 |
Jan
(2) |
Feb
(5) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2018 |
Jan
|
Feb
(1) |
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2019 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
(5) |
Dec
|
2020 |
Jan
(4) |
Feb
|
Mar
(4) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2021 |
Jan
|
Feb
|
Mar
(12) |
Apr
(7) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: <pon...@ne...> - 2002-06-18 07:08:50
|
Hi Eric, >This seems like a good idea. I also like his idea of removing the >file expansion. I may do that for myself too. ;) OK! Attached you will find a patch to senator. I finally used `eval-after-load' to setup the hook. Also I introduced a new flag `senator-enabled-try-expand-semantic' to avoid removing of `senator-try-expand-semantic' from `hippie-expand-try-functions-list' if it was previously setup via customization for example! If you are fine with this patch, I will check it in this evening and, of course, give credit to P. Lord ;-) David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: Eric M. L. <er...@si...> - 2002-06-17 20:07:46
|
This seems like a good idea. I also like his idea of removing the file expansion. I may do that for myself too. ;) Eric >>> pon...@ne... (David Ponce) seems to think that: >Hi Eric, > >What do you think of the following enhancement to senator that P. Lord proposed? > >As you first implemented senator hippie-expand extension I would like to read your opinion first before changing anything ;-) > >Thanks in advance! >David > >-------- Original Message -------- >Subject: senator >Date: Mon, 17 Jun 2002 13:24:34 +0100 >From: Phillip Lord <p....@ru...> >Reply-To: p....@ru... >To: David Ponce <da...@dp...> > > > > > >I have a suggestion for the senator package. In it you advice the >hippie-expand function like so.... > > >(defadvice hippie-expand (before senator activate) > "Add senator completion to hippie expand try method. >This setting is local to Semantic enabled buffers." > (Or (not (semantic-active-p)) > (memq #'senator-try-expand-semantic > hippie-expand-try-functions-list) > (set (make-local-variable 'hippie-expand-try-functions-list) > (cons #'senator-try-expand-semantic > (default-value 'hippie-expand-try-functions-list))))) > > >The problem is with this is that I have already set >hippie-expand-try-functions-list to a buffer local value for Java >mode (I like the try-complete-file-name functions lots, but they are a >pain in java mode). Because you use default-value the above >function over-writes the changes that I've tried to make in my >jde-mode-hook. > > >Why not something like... > >(if (fboundp 'hippie-expand) > (add-hook 'senator-minor-mode > (lambda() > (make-local-variable 'hippie-expand-try-functions-list) > (if senator-minor-mode > (add-to-list 'hippie-expand-try-functions-list > 'senator-try-expand-semantic) > (setq hippie-expand-try-functions-list > (delq 'senator-try-expand-semantic hippie-expand-try-functions-list)))))) > >Or maybe (eval-after-load "hippie-exp") instead of the (if (fboundp)) >form, as hippie-expand is autoloaded. > > >Cheers > >Phil > > > >__________________________________________________________________ >Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ > >Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ > > >_______________________________________________________________ > >Sponsored by: >ThinkGeek at http://www.ThinkGeek.com/ >_______________________________________________ >Cedet-devel mailing list >Ced...@li... >https://lists.sourceforge.net/lists/listinfo/cedet-devel > |
From: <pon...@ne...> - 2002-06-17 12:51:40
|
Hi Eric, What do you think of the following enhancement to senator that P. Lord proposed? As you first implemented senator hippie-expand extension I would like to read your opinion first before changing anything ;-) Thanks in advance! David -------- Original Message -------- Subject: senator Date: Mon, 17 Jun 2002 13:24:34 +0100 From: Phillip Lord <p....@ru...> Reply-To: p....@ru... To: David Ponce <da...@dp...> I have a suggestion for the senator package. In it you advice the hippie-expand function like so.... (defadvice hippie-expand (before senator activate) "Add senator completion to hippie expand try method. This setting is local to Semantic enabled buffers." (Or (not (semantic-active-p)) (memq #'senator-try-expand-semantic hippie-expand-try-functions-list) (set (make-local-variable 'hippie-expand-try-functions-list) (cons #'senator-try-expand-semantic (default-value 'hippie-expand-try-functions-list))))) The problem is with this is that I have already set hippie-expand-try-functions-list to a buffer local value for Java mode (I like the try-complete-file-name functions lots, but they are a pain in java mode). Because you use default-value the above function over-writes the changes that I've tried to make in my jde-mode-hook. Why not something like... (if (fboundp 'hippie-expand) (add-hook 'senator-minor-mode (lambda() (make-local-variable 'hippie-expand-try-functions-list) (if senator-minor-mode (add-to-list 'hippie-expand-try-functions-list 'senator-try-expand-semantic) (setq hippie-expand-try-functions-list (delq 'senator-try-expand-semantic hippie-expand-try-functions-list)))))) Or maybe (eval-after-load "hippie-exp") instead of the (if (fboundp)) form, as hippie-expand is autoloaded. Cheers Phil __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: David P. <da...@dp...> - 2002-06-14 17:33:27
|
Hi Eric, [...] > I know wisent has Bison style error correction. Will a bison style > grammar recover from errors as reliably as the token collecting loop > of a bovine style grammar? Yes, but with a bison style grammar a good error recovery scheme is not straightforward :-( It requires to minutely tweak the grammar to put the 'error terminals at the right places, and to write the clever semantic actions that will digest bad syntax and allow to continue parsing! I wrote the functions `wisent-skip-token' and `wisent-skip-block' to help to skip bad syntax in semantic actions. Nevertheless the hard part of work is to find where to catch syntax errors. Just compare the wisent-java (bison style) and wisent-java-tags (bovine style) grammars and you will see what I mean ;-) > For tagging purposes, a bovine style grammar seems like a good way to > capture as much information while avoiding errors. The bison style > grammars can sometimes have an irrecoverable heart attack on bad > syntax. This would be useful if highlighting all unparsable syntax > in a file. [...] I completely agree! The bison style is well suited to parse whole sources. By default bison (and wisent) stops on the first syntax error it encounters after issuing a relevant message. The bovine style is far better to do parsing in an interactive environment like Emacs. The semantic mechanism that automagically skip bad syntax is simple, efficient (it try to parse as much code as possible), very robust, and built-in! So it simplifies a lot the writing of grammars, where error recovery rules are almost unnecessary (there is no one in the wisent-java-tags grammar!). The developer only needs to be concentrated on grammar rules required to parse correct language syntax. Semantic will do the rest :-) > Have you experimented much with error recovery in wisent? This would > be a very useful topic to document clearly as it will make the > difference between parsing closely while a user types (where there > are lots of errors) and only once in a while. I'd love to see bad > syntax highlighting occurring the way font lock does as you type. My main experiment is in wisent-java grammar. Using a bison style grammar I managed to parse even very bad Java syntax :-) But I prefer to use the bovine style (like in wisent-java-tags) or, at the worst, a mix of the bovine and bison style (like in wisent-wy). It is far easier to maintain ;-) David |
From: Eric M. L. <er...@si...> - 2002-06-14 13:00:35
|
Hi David, Thanks for the updates. >>> pon...@ne... (David Ponce) seems to think that: >More news about my work on the parser plug-in API... > >Finally, I managed to migrate the wisent-java.wy/el stuff to use the >new parser API :-) > >I think I found a good and quite simple solution to use traditional >bison's style grammar with the semantic parser API! This is good news. >The issue with that sort of grammar is that it not use at all the >`semantic-bovinate-nonterminals' loop to parse the main >nonterminals. Only the start nonterminal is handled by >`semantic-bovinate-nonterminals', and cooked by >`semantic-raw-to-cooked-token' :-( > >To solve that I defined a new `wisent-cooked-token' function in >wisent-bovine.el. As its name means, it returns a cooked semantic >token (it calls `semantic-raw-to-cooked-token') contrary to >`wisent-token' which returns a raw token then post-processed by the >`semantic-bovinate-nonterminals' loop. > >Using that new function combined with the standard >`semantic-expand-nonterminal' mechanism it is quite easy to migrate >such grammars from the old wisent-bovine API to the new one! > >As a benefit it is still possible to develop both "orthodox" and >"bison traditional" style grammars ;-) This is a clever idea. Enabling someone to put a bison grammar from a real compiler into Emacs (after removing the C code, and replacing it with Emacs Lisp, of course) would help lots of new languages become supported. I know wisent has Bison style error correction. Will a bison style grammar recover from errors as reliably as the token collecting loop of a bovine style grammar? For tagging purposes, a bovine style grammar seems like a good way to capture as much information while avoiding errors. The bison style grammars can sometimes have an irrecoverable heart attack on bad syntax. This would be useful if highlighting all unparsable syntax in a file. Have you experimented much with error recovery in wisent? This would be a very useful topic to document clearly as it will make the difference between parsing closely while a user types (where there are lots of errors) and only once in a while. I'd love to see bad syntax highlighting occurring the way font lock does as you type. [ ... ] >P.S.: As it seems not required for 1.4 I would prefer to delay the >check-in of my change to `semantic-bovinate-nonterminals', related to >the 'reparse-symbol handling, until 1.4 release. [ ... ] Ok. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: David P. <da...@dp...> - 2002-06-13 18:26:18
|
Eric, I just checked a small typo fix in for semantic.texi and INSTALL. David |
From: Eric M. L. <er...@si...> - 2002-06-13 11:47:22
|
>>> "Richard Y. Kim" <ry...@ds...> seems to think that: >David and Erid, > >I made lots of little fixes. >Some of them were quite embarssing ones, so I had to post another >version to prevent one or both of you from spending time on >already-fixed things. > >I've been keeping this track of versions on my own RCS >archive. Whenever this things gets to be actually half-way useful, we >can worry about checking this into sf.net CVS database along with >other semantic files. [ ... ] Thanks for the update. It is Ok to check something like this into the sf CVS repository. I track which files are, or are not a part of a given distribution, so it would not cause a problem. If you are uncomfortable with that, that's ok too. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: <pon...@ne...> - 2002-06-13 09:24:45
|
Hi Eric, More news about my work on the parser plug-in API... Finally, I managed to migrate the wisent-java.wy/el stuff to use the new parser API :-) I think I found a good and quite simple solution to use traditional bison's style grammar with the semantic parser API! The issue with that sort of grammar is that it not use at all the `semantic-bovinate-nonterminals' loop to parse the main nonterminals. Only the start nonterminal is handled by `semantic-bovinate-nonterminals', and cooked by `semantic-raw-to-cooked-token' :-( To solve that I defined a new `wisent-cooked-token' function in wisent-bovine.el. As its name means, it returns a cooked semantic token (it calls `semantic-raw-to-cooked-token') contrary to `wisent-token' which returns a raw token then post-processed by the `semantic-bovinate-nonterminals' loop. Using that new function combined with the standard `semantic-expand-nonterminal' mechanism it is quite easy to migrate such grammars from the old wisent-bovine API to the new one! As a benefit it is still possible to develop both "orthodox" and "bison traditional" style grammars ;-) The couple wisent-java-tags/wisent-java is a good illustration of these two styles! You will find the updated wisent-bovine.el and wisent-java.wy/el files in the attached tarball ;-) What do you think? I hope to read you soon ;-) Thanks! David P.S.: As it seems not required for 1.4 I would prefer to delay the check-in of my change to `semantic-bovinate-nonterminals', related to the 'reparse-symbol handling, until 1.4 release. __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: Richard Y. K. <ry...@ds...> - 2002-06-13 08:37:46
|
David and Erid, I made lots of little fixes. Some of them were quite embarssing ones, so I had to post another version to prevent one or both of you from spending time on already-fixed things. I've been keeping this track of versions on my own RCS archive. Whenever this things gets to be actually half-way useful, we can worry about checking this into sf.net CVS database along with other semantic files. ;;; wisent-python.wy -- LALR grammar for Python ;; ;; Copyright (C) 2002 Richard Kim ;; ;; Author: Richard Kim <ry...@ds...> ;; Maintainer: Richard Kim <ry...@ds...> ;; Created: June 2002 ;; Keywords: syntax ;; X-RCS: $Id: wisent-python.wy,v 1.104 2002/06/13 08:29:03 ryk Exp $ ;; ;; This file is not part of GNU Emacs. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2, or (at ;; your option) any later version. ;; ;; This software is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ;; General Public License for more details. ;; ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs; see the file COPYING. If not, write to the ;; Free Software Foundation, Inc., 59 Temple Place - Suite 330, ;; Boston, MA 02111-1307, USA. ;;; Commentary: ;; ;; This is still a work in progress. ;; The official Grammar file from the Python source code distribution ;; was the starting point of this wisent version. ;; -------- ;; Settings ;; -------- %{ (setq ;; How `semantic-flex' will setup the lexer input stream. semantic-flex-depth nil ;; python grammar requires BOL tokens to compute indent/dedent semantic-flex-enable-bol t ;; python grammar requires NEWLINE tokens! semantic-flex-enable-newlines t ;; Tell `semantic-flex' to handle Python numbers semantic-number-expression semantic-java-number-regexp ;; Character used to separation a parent/child relationship semantic-type-relation-separator-character '(".") semantic-command-separation-character ";" ;; Init indentation stack wisent-python-lexer-indent-stack nil ) %} %outputfile "wisent-python.el" %parsetable wisent-python-parser-tables %keywordtable wisent-python-keywords %tokentable wisent-python-tokens %languagemode python-mode %setupfunction wisent-python-default-setup %start goal %token <newline> NEWLINE %token <punctuation> LTLTEQ "<<=" %token <punctuation> GTGTEQ ">>=" %token <punctuation> EXPEQ "**=" %token <punctuation> DIVDIVEQ "//=" %token <punctuation> DIVDIV "//" %token <punctuation> LTLT "<<" %token <punctuation> GTGT ">>" %token <punctuation> EXPONENT "**" %token <punctuation> EQ "==" %token <punctuation> GE ">=" %token <punctuation> LE "<=" %token <punctuation> PLUSEQ "+=" %token <punctuation> MINUSEQ "-=" %token <punctuation> MULTEQ "*=" %token <punctuation> DIVEQ "/=" %token <punctuation> MODEQ "%=" %token <punctuation> AMPEQ "&=" %token <punctuation> OREQ "|=" %token <punctuation> HATEQ "^=" %token <punctuation> LTGT "<>" %token <punctuation> NE "!=" %token <punctuation> HAT "^" %token <punctuation> LT "<" %token <punctuation> GT ">" %token <punctuation> AMP "&" %token <punctuation> MULT "*" %token <punctuation> DIV "/" %token <punctuation> MOD "%" %token <punctuation> PLUS "+" %token <punctuation> MINUS "-" %token <punctuation> PERIOD "." %token <punctuation> TILDE "~" %token <punctuation> BAR "|" %token <punctuation> COLON ":" %token <punctuation> SEMICOLON ";" %token <punctuation> COMMA "," %token <punctuation> POPEN "(" %token <punctuation> PCLOSE ")" %token <punctuation> BOPEN "[" %token <punctuation> BCLOSE "]" %token <punctuation> BRACEOPEN "{" %token <punctuation> BRACECLOSE "}" %token <punctuation> ASSIGN "=" %token <punctuation> BACKQUOTE "`" %token <string> STRING_LITERAL %token <number> NUMBER_LITERAL %token <symbol> NAME ;; Customize `wisent-flex' match algorithm ;; - Use string comparison for: ;; - An operator can be made of multiple successive punctuations %put punctuation {string t multiple t} ;; Handle 'bol tokens via function `wisent-python-lex-bol' %token <bol> INDENT DEDENT %put bol handler wisent-python-lex-bol ;; ----------------- ;; Literal terminals ;; ----------------- ;; ----------------- ;; Keyword terminals ;; ----------------- %token AND "and" %put AND summary " ... " %token ASSERT "assert" %put ASSERT summary " ... " %token BREAK "break" %put BREAK summary " ... " %token CLASS "class" %put CLASS summary " ... " %token CONTINUE "continue" %put CONTINUE summary " ... " %token DEF "def" %put DEF summary " ... " %token DEL "del" %put DEL summary " ... " %token ELIF "elif" %put ELIF summary " ... " %token ELSE "else" %put ELSE summary " ... " %token EXCEPT "except" %put EXCEPT summary " ... " %token EXEC "exec" %put EXEC summary " ... " %token FINALLY "finally" %put FINALLY summary " ... " %token FOR "for" %put FOR summary " ... " %token FROM "from" %put FROM summary " ... " %token GLOBAL "global" %put GLOBAL summary " ... " %token IF "if" %put IF summary " ... " %token IMPORT "import" %put IMPORT summary " ... " %token IN "in" %put IN summary " ... " %token IS "is" %put IS summary " ... " %token LAMBDA "lambda" %put LAMDA summary "..." %token NOT "not" %put NOT summary "..." %token OR "or" %put OR summary "..." %token PASS "pass" %put PASS summary "..." %token PRINT "print" %put PRINT summary "..." %token RAISE "raise" %put RAISE summary "..." %token RETURN "return" %put RETURN summary "..." %token TRY "try" %put TRY summary "..." %token WHILE "while" %put WHILE summary "..." %token YIELD "yield" %put YIELD summary "..." %% goal : single_input ;; | file_input ;; | eval_input ; ;; single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE single_input : NEWLINE | simple_stmt | compound_stmt NEWLINE ; ;; file_input: (NEWLINE | stmt)* ENDMARKER file_input : stmt_list ENDMARKER (wisent-token $1 'stmt_list nil nil) ; ;; What is ENDMARKER? ;; Make it empty non-terminal until I figure out what it is. -ryk 6/11/02 10:10pm. ENDMARKER : ;;EMPTY ; ;; (NEWLINE | stmt)* stmt_list : ;;EMPTY | stmt_list newline_or_stmt (wisent-token (concat $1 $2) 'stmt_list nil nil) ; ;; NEWLINE | stmt newline_or_stmt : NEWLINE | stmt (wisent-token $1 'stmt nil nil) ; ;; eval_input: testlist NEWLINE* ENDMARKER eval_input : testlist newline_list ENDMARKER (wisent-token $1 'testlist nil nil) ; ;; NEWLINE* newline_list : ;;EMPTY | newline_list NEWLINE ; ;; funcdef: 'def' NAME parameters ':' suite funcdef : DEF NAME parameters COLON suite ; ;; parameters: '(' [varargslist] ')' parameters : POPEN varargslist_opt PCLOSE ; ;; [varargslist] varargslist_opt : ;;EMPTY | varargslist ; ;; varargslist: (fpdef ['=' test] ',')* ('*' NAME [',' '**' NAME] | '**' NAME) | fpdef ['=' test] (',' fpdef ['=' test])* [','] varargslist : fpdef_opt_test_zom mult_name | fpdef eq_test_opt fpdef_opt_test_trailer_zom comma_opt ; ;; (fpdef ['=' test] ',')* fpdef_opt_test_zom : ;;EMPTY | fpdef_opt_test_zom fpdef eq_test_opt COMMA ; ;; ['=' test] eq_test_opt : ;;EMPTY | ASSIGN test ; ;; ('*' NAME [',' '**' NAME] | '**' NAME) mult_name : MULT NAME multmult_name_opt | EXPONENT NAME ; ;; [',' '**' NAME] multmult_name_opt : ;;EMPTY | COMMA EXPONENT NAME ; ;; (',' fpdef ['=' test])* fpdef_opt_test_trailer_zom : ;;EMPTY | fpdef_opt_test_trailer_zom COMMA fpdef eq_test_opt ; ;; fpdef: NAME | '(' fplist ')' fpdef : NAME | POPEN fplist PCLOSE ; ;; fplist: fpdef (',' fpdef)* [','] fplist : fpdef fpdef_list comma_opt ; ;; (',' fpdef)* fpdef_list : ;;EMPTY | fpdef_list COMMA fpdef ; ;; stmt: simple_stmt | compound_stmt stmt : simple_stmt | compound_stmt (wisent-token $1 'compound_stmt nil nil) ; ;; simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE simple_stmt : small_stmt_list semicolon_opt NEWLINE (wisent-token $1 'small_stmt_list nil nil) ; ;; small_stmt (';' small_stmt)* small_stmt_list : small_stmt | small_stmt_list SEMICOLON small_stmt (format "%s; %s" $1 $3) ; ;; [';'] semicolon_opt : ;;EMPTY | SEMICOLON ; ;; small_stmt: expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt ;; | import_stmt | global_stmt | exec_stmt | assert_stmt small_stmt : expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt | import_stmt | global_stmt | exec_stmt | assert_stmt ; ;; expr_stmt: testlist (augassign testlist | ('=' testlist)*) expr_stmt : testlist expr_stmt_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (augassign testlist | ('=' testlist)*) expr_stmt_trailer : augassign testlist (format "%s %s" $1 $2) | eq_testlist_zom ; ;; ('=' testlist)* eq_testlist_zom : ;;EMPTY | eq_testlist_zom ASSIGN testlist (format "%s %s %s" $1 $2 $3) ; ;; augassign: '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' ;; | '<<=' | '>>=' | '**=' | '//=' augassign : PLUSEQ | MINUSEQ | MULTEQ | DIVEQ | MODEQ | AMPEQ | OREQ | HATEQ | LTLTEQ | GTGTEQ | EXPEQ | DIVDIVEQ ; ;; # For normal assignments, additional restrictions enforced by the interpreter ;; print_stmt: 'print' ( [ test (',' test)* [','] ] | '>>' test [ (',' test)+ [','] ] ) print_stmt : PRINT print_stmt_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; ( [ test (',' test)* [','] ] | '>>' test [ (',' test)+ [','] ] ) print_stmt_trailer : test_list_with_opt_comma_opt | GTGT test trailing_test_list_with_opt_comma_opt (if $3 (format "%s %s %s" $1 $2 $3) (format "%s %s" $1 $2)) ; ;; [ (',' test)+ [','] ] trailing_test_list_with_opt_comma_opt : ;;EMPTY | trailing_test_list comma_opt ; ;; (',' test)+ trailing_test_list : COMMA test (format ", %s" $2) | trailing_test_list COMMA test (format "%s, %s" $1 $3) ; ;; [ test (',' test)* [','] ] test_list_with_opt_comma_opt : ;;EMPTY | testlist ; ;; del_stmt: 'del' exprlist del_stmt : DEL exprlist (format "del %s" $2) ; ;; pass_stmt: 'pass' pass_stmt : PASS ; ;; flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt flow_stmt : break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt ; ;; break_stmt: 'break' break_stmt : BREAK ; ;; continue_stmt: 'continue' continue_stmt : CONTINUE ; ;; return_stmt: 'return' [testlist] return_stmt : RETURN testlist_opt (if $2 (format "return %s" $2) "return") ; ;; [testlist] testlist_opt : ;;EMPTY | testlist ; ;; yield_stmt: 'yield' testlist yield_stmt : YIELD testlist (format "%s %s" $1 $2) ; ;; raise_stmt: 'raise' [test [',' test [',' test]]] raise_stmt : RAISE zero_one_two_or_three_tests ; ;; [test [',' test [',' test]]] zero_one_two_or_three_tests : ;;EMPTY | test zero_one_or_two_tests ; ;; [',' test [',' test]] zero_one_or_two_tests : ;;EMPTY | COMMA test zero_or_one_comma_test ; ;; [',' test] zero_or_one_comma_test : ;;EMPTY | COMMA test ; ;; import_stmt : 'import' dotted_as_name (',' dotted_as_name)* ;; | 'from' dotted_name 'import' ('*' | import_as_name (',' import_as_name)*) import_stmt : IMPORT dotted_as_name_list | FROM dotted_name IMPORT star_or_import_as_name_list ; ;; dotted_as_name (',' dotted_as_name)* dotted_as_name_list : dotted_as_name | dotted_as_name_list COMMA dotted_as_name ; ;; ('*' | import_as_name (',' import_as_name)*) star_or_import_as_name_list : MULT | import_as_name_list ; ;; import_as_name (',' import_as_name)* import_as_name_list : import_as_name | import_as_name_list COMMA import_as_name ; ;; import_as_name: NAME [NAME NAME] import_as_name : NAME name_name_opt ; ;; dotted_as_name: dotted_name [NAME NAME] dotted_as_name : dotted_name name_name_opt ; ;; [NAME NAME] name_name_opt : ;;EMPTY | NAME NAME (format "%s %s" $1 $2) ; ;; dotted_name: NAME ('.' NAME)* dotted_name : NAME | dotted_name PERIOD NAME (format "%s %s %s" $1 $2 $3) ; ;; global_stmt: 'global' NAME (',' NAME)* global_stmt : GLOBAL comma_sep_name_list ; ;; NAME (',' NAME)* comma_sep_name_list : NAME | comma_sep_name_list COMMA NAME (format "%s %s %s" $1 $2 $3) ; ;; exec_stmt: 'exec' expr ['in' test [',' test]] exec_stmt : EXEC expr exec_trailer ; ;; ['in' test [',' test]] exec_trailer : ;;EMPTY | IN test comma_test_opt (format "%s %s" $1 $2) ; ;; [',' test] comma_test_opt : ;;EMPTY | COMMA test ; ;; assert_stmt: 'assert' test [',' test] assert_stmt : ASSERT test comma_test_opt (format "%s %s" $1 $2) ; ;; compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | funcdef | classdef compound_stmt : if_stmt | while_stmt | for_stmt | try_stmt | funcdef | classdef ; ;; if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] if_stmt : IF test COLON suite elif_suite_pair_list else_suite_pair_opt ; ;; ('elif' test ':' suite)* elif_suite_pair_list : ;;EMPTY | elif_suite_pair_list ELIF test COLON suite ; ;; while_stmt: 'while' test ':' suite ['else' ':' suite] while_stmt : WHILE test COLON suite else_suite_pair_opt ; ;; for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite] for_stmt : FOR exprlist IN testlist COLON suite else_suite_pair_opt ; ;; try_stmt: ('try' ':' suite (except_clause ':' suite)+ #diagram:break ;; ['else' ':' suite] | 'try' ':' suite 'finally' ':' suite) try_stmt : TRY COLON suite except_clause_suite_pair_list else_suite_pair_opt | TRY COLON suite FINALLY COLON suite ; ;; (except_clause ':' suite)+ except_clause_suite_pair_list : except_clause COLON suite | except_clause_suite_pair_list except_clause COLON suite ; ;; ['else' ':' suite] else_suite_pair_opt : ;;EMPTY | ELSE COLON suite ; ;; # NB compile.c makes sure that the default except clause is last ;; except_clause: 'except' [test [',' test]] except_clause : EXCEPT zero_one_or_two_test ; ;; [test [',' test]] zero_one_or_two_test : ;;EMPTY | test zero_or_one_comma_test ; ;; suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT suite : simple_stmt | NEWLINE INDENT stmt_oom DEDENT ; ;; stmt+ stmt_oom : stmt | stmt_oom stmt ; ;; test: and_test ('or' and_test)* | lambdef test : and_test test_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) | lambdef ; ;; ('or' and_test)* test_trailer : ;;EMPTY | test_trailer OR and_test (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; and_test: not_test ('and' not_test)* and_test : not_test | and_test AND not_test (format "%s %s %s" $1 $2 $3) ; ;; not_test: 'not' not_test | comparison not_test : NOT not_test (format "%s %s" $1 $2) | comparison ; ;; comparison: expr (comp_op expr)* comparison : expr | comparison comp_op expr (format "%s %s %s" $1 $2 $3) ; ;; comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comp_op : LT | GT | EQ | GE | LE | LTGT | NE | IN | NOT IN | IS | IS NOT ; ;; expr: xor_expr ('|' xor_expr)* expr : xor_expr | expr BAR xor_expr (format "%s %s %s" $1 $2 $3) ; ;; xor_expr: and_expr ('^' and_expr)* xor_expr : and_expr | xor_expr HAT and_expr (format "%s %s %s" $1 $2 $3) ; ;; and_expr: shift_expr ('&' shift_expr)* and_expr : shift_expr | and_expr AMP shift_expr (format "%s %s %s" $1 $2 $3) ; ;; shift_expr: arith_expr (('<<'|'>>') arith_expr)* shift_expr : arith_expr | shift_expr shift_expr_operators arith_expr (format "%s %s %s" $1 $2 $3) ; ;; ('<<'|'>>') shift_expr_operators : LTLT | GTGT ; ;; arith_expr: term (('+'|'-') term)* arith_expr : term | arith_expr plus_or_minus term (format "%s %s %s" $1 $2 $3) ; ;; ('+'|'-') plus_or_minus : PLUS | MINUS ; ;; term: factor (('*'|'/'|'%'|'//') factor)* term : factor | term term_operator factor (format "%s %s %s" $1 $2 $3) ; term_operator : MULT | DIV | MOD | DIVDIV ; ;; factor: ('+'|'-'|'~') factor | power factor : prefix_operators factor (format "%s %s" $1 $2) | power ; ;; ('+'|'-'|'~') prefix_operators : PLUS | MINUS | TILDE ; ;; power: atom trailer* ('**' factor)* power : atom trailer_opt exponent_opt (if $3 (if $2 (format "%s $2 %s" $1 $2 $3) (format "%s ** %s" $1 $3)) (if $2 (format "%s %s" $1 $2) (format "%s" $1))) ; trailer_opt : ;;EMPTY | trailer ; exponent_opt : ;;EMPTY | EXPONENT factor (format "** %s" $2) ; ;; atom: '(' [testlist] ')' | '[' [listmaker] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+ atom : POPEN testlist_opt PCLOSE | BOPEN listmaker_opt BCLOSE | BRACEOPEN dictmaker_opt BRACECLOSE | BACKQUOTE testlist BACKQUOTE | NAME | NUMBER_LITERAL | one_or_more_string ; listmaker_opt : ;;EMPTY | listmaker ; dictmaker_opt : ;;EMPTY | dictmaker ; ;; Use (read $1) to peel away the double quotes. one_or_more_string : STRING_LITERAL (read $1) | one_or_more_string STRING_LITERAL (format "%s %s" $1 (read $2)) ; ;; listmaker: test ( list_for | (',' test)* [','] ) listmaker : test listmaker_trailer ; ;; ( list_for | (',' test)* [','] ) listmaker_trailer : list_for | testlist_trailer comma_opt ; ;; (',' test)* testlist_trailer : ;;EMPTY | testlist_trailer COMMA test (format "%s %s %s" $1 $2 $3) ; ;; lambdef: 'lambda' [varargslist] ':' test lambdef : LAMBDA COLON test (format "%s %s %s" $1 $2 $3) ; ;; trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME trailer : POPEN arglist_opt PCLOSE | BOPEN subscriptlist BCLOSE | PERIOD NAME ; ;; [arglist] arglist_opt : ;;EMPTY | arglist ; ;; subscriptlist: subscript (',' subscript)* [','] subscriptlist : comma_sep_subscript_list comma_opt ; ;; subscript (',' subscript)* comma_sep_subscript_list : subscript | comma_sep_subscript_list COMMA subscript ; ;; subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] subscript : PERIOD PERIOD PERIOD | test | zero_or_one_test COLON zero_or_one_test zero_or_one_sliceop ; ;; [sliceop] zero_or_one_sliceop : ;;EMPTY | sliceop ; ;; sliceop: ':' [test] sliceop : COLON zero_or_one_test ; ;; [test] zero_or_one_test : ;;EMPTY | test ; ;; exprlist: expr (',' expr)* [','] exprlist : expr_list comma_opt ; ;; expr (',' expr)* expr_list : expr | expr_list COMMA expr ; ;; testlist: test (',' test)* [','] testlist : comma_sep_test_list comma_opt (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; test (',' test)* comma_sep_test_list : test | comma_sep_test_list COMMA test (format "%s %s %s" $1 $2 $3) ; ;; [','] comma_opt : ;;EMPTY | COMMA ; ;; How is testlist_safe different with testlist??? -ryk 6/13/02 ;; testlist_safe: test [(',' test)+ [',']] testlist_safe : test testlist_safe_trailer_opt (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; [(',' test)+ [',']] testlist_safe_trailer_opt : ;; EMPTY | testlist_safe_term comma_opt (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (',' test)+ testlist_safe_term : COMMA test (format "%s %s" $1 $2) | testlist_safe_term COMMA test (format "%s %s %s" $1 $2 $3) ; ;; dictmaker: test ':' test (',' test ':' test)* [','] dictmaker : test COLON test colon_sep_test comma_opt ; ;; (',' test ':' test)* colon_sep_test : ;;EMPTY | colon_sep_test COMMA test COLON test ; ;; classdef: 'class' NAME ['(' testlist ')'] ':' suite classdef : CLASS NAME paren_testlist_opt COLON suite ; ;; ['(' testlist ')'] paren_testlist_opt : ;;EMPTY | POPEN testlist PCLOSE ; ;; arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) arglist : argument_comma_zom arglist_trailer ; ;; (argument ',')* argument_comma_zom : ;;EMPTY | argument_comma_zom argument COMMA ; ;; (argument [',']| '*' test [',' '**' test] | '**' test) arglist_trailer : argument comma_opt | MULT test comma_mult_mult_test_opt | EXPONENT test ; ;; [',' '**' test] comma_mult_mult_test_opt : ;;EMPTY | COMMA EXPONENT test ; ;; argument: [test '='] test # Really [keyword '='] test argument : test_eq_opt test ; ;; [test '='] test_eq_opt : ;;EMPTY | test ASSIGN ; ;; list_iter: list_for | list_if list_iter : list_for | list_if ; ;; list_for: 'for' exprlist 'in' testlist_safe [list_iter] list_for : FOR exprlist IN testlist_safe list_iter_opt ; ;; list_if: 'if' test [list_iter] list_if : IF test list_iter_opt ; ;; [list_iter] list_iter_opt : ;;EMPTY | list_iter ; ;;; wisent-python.wy ends here |
From: Richard Y. K. <ry...@ds...> - 2002-06-12 15:49:43
|
>>>>> "RK" == Richard Y Kim <ry...@ds...> writes: RK> RK> David and Eric, RK> Here is the updated wisent-python.wy file. RK> I've added all the rules from the official Grammar file as comments, RK> then translated them into wisent syntax. RK> RK> The state of this file is: RK> * M-x wisent-wy-update-outputfile finishes successfully RK> * The resulting table parses simple statements. RK> RK> What needs to be done: RK> * Lots of debugging/verification is needed. RK> * Add proper semantic rules so that the tokens generated by the parser RK> is usable by other tools down the line. RK> * Introduce operator precedence which should simplify the grammar. RK> RK> Any feedback, suggestions, criticism would be appreciated as always. I must have been up too late. Here is the file. ;;; wisent-python.wy -- LALR grammar for Python ;; ;; Copyright (C) 2002 Richard Kim ;; ;; Author: Richard Kim <ry...@ds...> ;; Maintainer: Richard Kim <ry...@ds...> ;; Created: June 2002 ;; Keywords: syntax ;; ;; This file is not part of GNU Emacs. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2, or (at ;; your option) any later version. ;; ;; This software is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ;; General Public License for more details. ;; ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs; see the file COPYING. If not, write to the ;; Free Software Foundation, Inc., 59 Temple Place - Suite 330, ;; Boston, MA 02111-1307, USA. ;;; Commentary: ;; ;; This is still a work in progress. ;; The official Grammar file from the Python source code distribution ;; was the starting point of this wisent version. ;; -------- ;; Settings ;; -------- %{ (setq ;; How `semantic-flex' will setup the lexer input stream. semantic-flex-depth nil ;; python grammar requires BOL tokens to compute indent/dedent semantic-flex-enable-bol t ;; python grammar requires NEWLINE tokens! semantic-flex-enable-newlines t ;; Tell `semantic-flex' to handle Python numbers semantic-number-expression semantic-java-number-regexp ;; Character used to separation a parent/child relationship semantic-type-relation-separator-character '(".") semantic-command-separation-character ";" ;; Init indentation stack wisent-python-lexer-indent-stack nil ) %} %outputfile "wisent-python.el" %parsetable wisent-python-parser-tables %keywordtable wisent-python-keywords %tokentable wisent-python-tokens %languagemode python-mode %setupfunction wisent-python-default-setup %start goal %token <newline> NEWLINE %token <punctuation> LTLTEQ "<<=" %token <punctuation> GTGTEQ ">>=" %token <punctuation> EXPEQ "**=" %token <punctuation> DIVDIVEQ "//=" %token <punctuation> DIVDIV "//" %token <punctuation> LTLT "<<" %token <punctuation> GTGT ">>" %token <punctuation> EXPONENT "**" %token <punctuation> EQ "==" %token <punctuation> GE ">=" %token <punctuation> LE "<=" %token <punctuation> PLUSEQ "+=" %token <punctuation> MINUSEQ "-=" %token <punctuation> MULTEQ "*=" %token <punctuation> DIVEQ "/=" %token <punctuation> MODEQ "%=" %token <punctuation> AMPEQ "&=" %token <punctuation> OREQ "|=" %token <punctuation> HATEQ "^=" %token <punctuation> LTGT "<>" %token <punctuation> NE "!=" %token <punctuation> HAT "^" %token <punctuation> LT "<" %token <punctuation> GT ">" %token <punctuation> AMP "&" %token <punctuation> MULT "*" %token <punctuation> DIV "/" %token <punctuation> MOD "%" %token <punctuation> PLUS "+" %token <punctuation> MINUS "-" %token <punctuation> PERIOD "." %token <punctuation> TILDE "~" %token <punctuation> BAR "|" %token <punctuation> COLON ":" %token <punctuation> SEMICOLON ";" %token <punctuation> COMMA "," %token <punctuation> POPEN "(" %token <punctuation> PCLOSE ")" %token <punctuation> BOPEN "[" %token <punctuation> BCLOSE "]" %token <punctuation> ASSIGN "=" %token <string> STRING_LITERAL %token <number> NUMBER_LITERAL %token <symbol> NAME ;; Customize `wisent-flex' match algorithm ;; - Use string comparison for: ;; - An operator can be made of multiple successive punctuations %put punctuation {string t multiple t} ;; Handle 'bol tokens via function `wisent-python-lex-bol' %token <bol> INDENT DEDENT %put bol handler wisent-python-lex-bol ;; ----------------- ;; Literal terminals ;; ----------------- ;; ----------------- ;; Keyword terminals ;; ----------------- %token AND "and" %put AND summary " ... " %token ASSERT "assert" %put ASSERT summary " ... " %token BREAK "break" %put BREAK summary " ... " %token CLASS "class" %put CLASS summary " ... " %token CONTINUE "continue" %put CONTINUE summary " ... " %token DEF "def" %put DEF summary " ... " %token DEL "del" %put DEL summary " ... " %token ELIF "elif" %put ELIF summary " ... " %token ELSE "else" %put ELSE summary " ... " %token EXCEPT "except" %put EXCEPT summary " ... " %token EXEC "exec" %put EXEC summary " ... " %token FINALLY "finally" %put FINALLY summary " ... " %token FOR "for" %put FOR summary " ... " %token FROM "from" %put FROM summary " ... " %token GLOBAL "global" %put GLOBAL summary " ... " %token IF "if" %put IF summary " ... " %token IMPORT "import" %put IMPORT summary " ... " %token IN "in" %put IN summary " ... " %token IS "is" %put IS summary " ... " %token LAMBDA "lambda" %put LAMDA summary "..." %token NOT "not" %put NOT summary "..." %token OR "or" %put OR summary "..." %token PASS "pass" %put PASS summary "..." %token PRINT "print" %put PRINT summary "..." %token RAISE "raise" %put RAISE summary "..." %token RETURN "return" %put RETURN summary "..." %token TRY "try" %put TRY summary "..." %token WHILE "while" %put WHILE summary "..." %token YIELD "yield" %put YIELD summary "..." %% goal : single_input ;; | file_input ;; | eval_input ; ;; single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE ;; TODO single_input : NEWLINE | simple_stmt (wisent-token $1 'simple_stmt nil nil) | compound_stmt NEWLINE (wisent-token $1 'compound_stmt nil nil) ; ;; file_input: (NEWLINE | stmt)* ENDMARKER file_input : stmt_list ENDMARKER (wisent-token $1 'stmt_list nil nil) ; ;; What is ENDMARKER? ;; Make it empty non-terminal until I figure out what it is. -ryk 6/11/02 10:10pm. ENDMARKER : ;;EMPTY ; ;; (NEWLINE | stmt)* stmt_list : ;;EMPTY | stmt_list newline_or_stmt (wisent-token (concat $1 $2) 'stmt_list nil nil) ; ;; NEWLINE | stmt newline_or_stmt : NEWLINE | stmt (wisent-token $1 'stmt nil nil) ; ;; eval_input: testlist NEWLINE* ENDMARKER eval_input : testlist newline_list ENDMARKER (wisent-token $1 'testlist nil nil) ; ;; NEWLINE* newline_list : ;;EMPTY | newline_list NEWLINE ; ;; funcdef: 'def' NAME parameters ':' suite funcdef : DEF NAME parameters COLON suite ; ;; parameters: '(' [varargslist] ')' parameters : POPEN varargslist_opt PCLOSE ; ;; [varargslist] varargslist_opt : ;;EMPTY | varargslist ; ;; varargslist: (fpdef ['=' test] ',')* ('*' NAME [',' '**' NAME] | '**' NAME) | fpdef ['=' test] (',' fpdef ['=' test])* [','] varargslist : fpdef_opt_test_zom mult_name | fpdef eq_test_opt fpdef_opt_test_trailer_zom comma_opt ; ;; (fpdef ['=' test] ',')* fpdef_opt_test_zom : ;;EMPTY | fpdef_opt_test_zom fpdef eq_test_opt COMMA ; ;; ['=' test] eq_test_opt : ;;EMPTY | ASSIGN test ; ;; ('*' NAME [',' '**' NAME] | '**' NAME) mult_name : MULT NAME multmult_name_opt | EXPONENT NAME ; ;; [',' '**' NAME] multmult_name_opt : ;;EMPTY | COMMA EXPONENT NAME ; ;; (',' fpdef ['=' test])* fpdef_opt_test_trailer_zom : ;;EMPTY | fpdef_opt_test_trailer_zom COMMA fpdef eq_test_opt ; ;; fpdef: NAME | '(' fplist ')' fpdef : NAME | POPEN fplist PCLOSE ; ;; fplist: fpdef (',' fpdef)* [','] fplist : fpdef fpdef_list comma_opt ; ;; (',' fpdef)* fpdef_list : ;;EMPTY | fpdef_list COMMA fpdef ; ;; stmt: simple_stmt | compound_stmt stmt : simple_stmt (wisent-token $1 'simple_stmt nil nil) | compound_stmt (wisent-token $1 'compound_stmt nil nil) ; ;; simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE simple_stmt : small_stmt zero_or_more_small_stmt_list semicolon_opt NEWLINE ; ;; (';' small_stmt)* zero_or_more_small_stmt_list : ;;EMPTY | zero_or_more_small_stmt_list SEMICOLON small_stmt ; ;; [';'] semicolon_opt : ;;EMPTY | SEMICOLON ; ;; small_stmt: expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt | import_stmt | global_stmt | exec_stmt | assert_stmt small_stmt : expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt | import_stmt | global_stmt | exec_stmt | assert_stmt ; ;; expr_stmt: testlist (augassign testlist | ('=' testlist)*) expr_stmt : testlist expr_stmt_trailer ; ;; (augassign testlist | ('=' testlist)*) expr_stmt_trailer : augassign testlist | eq_testlist_zom ; ;; ('=' testlist)* eq_testlist_zom : ;;EMPTY | eq_testlist_zom ASSIGN testlist ; ;; augassign: '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '**=' | '//=' augassign : PLUSEQ | MINUSEQ | MULTEQ | DIVEQ | MODEQ | AMPEQ | OREQ | HATEQ | LTLTEQ | GTGTEQ | EXPEQ | DIVDIVEQ ; ;; # For normal assignments, additional restrictions enforced by the interpreter ;; print_stmt: 'print' ( [ test (',' test)* [','] ] | '>>' test [ (',' test)+ [','] ] ) print_stmt : PRINT print_stmt_trailer ; ;; ( [ test (',' test)* [','] ] | '>>' test [ (',' test)+ [','] ] ) print_stmt_trailer : test_list_with_opt_comma_opt | GTGT test trailing_test_list_with_opt_comma_opt ; ;; [ (',' test)+ [','] ] trailing_test_list_with_opt_comma_opt : ;;EMPTY | trailing_test_list comma_opt ; ;; (',' test)+ trailing_test_list : COMMA test | trailing_test_list COMMA test ; ;; [ test (',' test)* [','] ] test_list_with_opt_comma_opt : ;;EMPTY | test testlist_trailer comma_opt ; ;; del_stmt: 'del' exprlist del_stmt : DEL exprlist ; ;; pass_stmt: 'pass' pass_stmt : PASS ; ;; flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt flow_stmt : break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt ; ;; break_stmt: 'break' break_stmt : BREAK ; ;; continue_stmt: 'continue' continue_stmt : CONTINUE ; ;; return_stmt: 'return' [testlist] return_stmt : RETURN testlist_opt ; ;; [testlist] testlist_opt : ;;EMPTY | testlist ; ;; yield_stmt: 'yield' testlist yield_stmt : YIELD testlist ; ;; raise_stmt: 'raise' [test [',' test [',' test]]] raise_stmt : RAISE zero_one_two_or_three_tests ; ;; [test [',' test [',' test]]] zero_one_two_or_three_tests : ;;EMPTY | test zero_one_or_two_tests ; ;; [',' test [',' test]] zero_one_or_two_tests : ;;EMPTY | COMMA test zero_or_one_comma_test ; ;; [',' test] zero_or_one_comma_test : ;;EMPTY | COMMA test ; ;; import_stmt: 'import' dotted_as_name (',' dotted_as_name)* | 'from' dotted_name 'import' ('*' | import_as_name (',' import_as_name)*) import_stmt : IMPORT dotted_as_name comma_dotted_as_name_zom | FROM dotted_name IMPORT star_or_import_as_name_list ; ;; (',' dotted_as_name)* comma_dotted_as_name_zom : ;;EMPTY | comma_dotted_as_name_zom COMMA dotted_as_name ; ;; ('*' | import_as_name (',' import_as_name)*) star_or_import_as_name_list : MULT | import_as_name comma_import_as_name_zom ; ;; (',' import_as_name)* comma_import_as_name_zom : ;;EMPTY | comma_import_as_name_zom COMMA import_as_name ; ;; import_as_name: NAME [NAME NAME] import_as_name : NAME name_name_opt ; ;; dotted_as_name: dotted_name [NAME NAME] dotted_as_name : dotted_name name_name_opt ; ;; [NAME NAME] name_name_opt : ;;EMPTY | NAME NAME (format "%s %s" $1 $2) ; ;; dotted_name: NAME ('.' NAME)* dotted_name : NAME | dotted_name PERIOD NAME (format "%s %s %s" $1 $2 $3) ; ;; global_stmt: 'global' NAME (',' NAME)* global_stmt : GLOBAL comma_sep_name_list ; ;; NAME (',' NAME)* comma_sep_name_list : NAME | comma_sep_name_list COMMA NAME (format "%s %s %s" $1 $2 $3) ; ;; exec_stmt: 'exec' expr ['in' test [',' test]] exec_stmt : EXEC expr exec_trailer ; ;; ['in' test [',' test]] exec_trailer : ;;EMPTY | IN test comma_test_opt (format "%s %s" $1 $2) ; ;; [',' test] comma_test_opt : ;;EMPTY | COMMA test ; ;; assert_stmt: 'assert' test [',' test] assert_stmt : ASSERT test comma_test_opt (format "%s %s" $1 $2) ; ;; compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | funcdef | classdef compound_stmt : if_stmt | while_stmt | for_stmt | try_stmt | funcdef | classdef ; ;; if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite] if_stmt : IF test COLON suite elif_suite_pair_list else_suite_pair_opt ; ;; ('elif' test ':' suite)* elif_suite_pair_list : ;;EMPTY | elif_suite_pair_list ELIF test COLON suite ; ;; while_stmt: 'while' test ':' suite ['else' ':' suite] while_stmt : WHILE test COLON suite else_suite_pair_opt ; ;; for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite] for_stmt : FOR exprlist IN testlist COLON suite else_suite_pair_opt ; ;; try_stmt: ('try' ':' suite (except_clause ':' suite)+ #diagram:break ;; ['else' ':' suite] | 'try' ':' suite 'finally' ':' suite) try_stmt : TRY COLON suite except_clause_suite_pair_list else_suite_pair_opt | TRY COLON suite FINALLY COLON suite ; ;; (except_clause ':' suite)+ except_clause_suite_pair_list : except_clause COLON suite | except_clause_suite_pair_list except_clause COLON suite ; ;; ['else' ':' suite] else_suite_pair_opt : ;;EMPTY | ELSE COLON suite ; ;; # NB compile.c makes sure that the default except clause is last ;; except_clause: 'except' [test [',' test]] except_clause : EXCEPT zero_one_or_two_test ; ;; [test [',' test]] zero_one_or_two_test : ;;EMPTY | test zero_or_one_comma_test ; ;; suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT suite : simple_stmt | NEWLINE INDENT stmt_oom DEDENT ; ;; stmt+ stmt_oom : stmt | stmt_oom stmt ; ;; test: and_test ('or' and_test)* | lambdef test : and_test test_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) | lambdef ; ;; ('or' and_test)* test_trailer : ;;EMPTY | test_trailer OR and_test (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; and_test: not_test ('and' not_test)* and_test : not_test and_test_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; ('and' not_test)* and_test_trailer : ;;EMPTY | and_test_trailer AND not_test (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; not_test: 'not' not_test | comparison not_test : NOT not_test (format "%s %s" $1 $2) | comparison ; ;; comparison: expr (comp_op expr)* comparison : expr comparison_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (comp_op expr)* comparison_trailer : ;;EMPTY | comparison_trailer comp_op expr (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comp_op : LT | GT | EQ | GE | LE | LTGT | NE | IN | NOT IN | IS | IS NOT ; ;; expr: xor_expr ('|' xor_expr)* expr : xor_expr expr_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; ('|' xor_expr)* expr_trailer : ;;EMPTY | expr_trailer BAR xor_expr (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; xor_expr: and_expr ('^' and_expr)* xor_expr : and_expr xor_expr_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; ('^' and_expr)* xor_expr_trailer : ;;EMPTY | xor_expr_trailer HAT and_expr (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; and_expr: shift_expr ('&' shift_expr)* and_expr : shift_expr and_expr_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; ('&' shift_expr)* and_expr_trailer : ;;EMPTY | and_expr_trailer AMP shift_expr (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; shift_expr: arith_expr (('<<'|'>>') arith_expr)* shift_expr : arith_expr shift_expr_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (('<<'|'>>') arith_expr)* shift_expr_trailer : ;;EMPTH | shift_expr_trailer shift_expr_operators arith_expr (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; ('<<'|'>>') shift_expr_operators : LTLT | GTGT ; ;; arith_expr: term (('+'|'-') term)* arith_expr : term arith_expr_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (('+'|'-') term)* arith_expr_trailer : ;;EMPTY | arith_expr_trailer plus_or_minus term (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; ('+'|'-') plus_or_minus : PLUS | MINUS ; ;; term: factor (('*'|'/'|'%'|'//') factor)* term : factor term_trailer (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (('*'|'/'|'%'|'//') factor)* term_trailer : ;;EMPTY | term_trailer term_operator factor (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; term_operator : MULT | DIV | MOD | DIVDIV ; ;; factor: ('+'|'-'|'~') factor | power factor : prefix_operators factor (format "%s %s" $1 $2) | power ; ;; ('+'|'-'|'~') prefix_operators : PLUS | MINUS | TILDE ; ;; power: atom trailer* ('**' factor)* power : atom trailer_opt exponent_opt (if $3 (if $2 (format "%s $2 %s" $1 $2 $3) (format "%s ** %s" $1 $3)) (if $2 (format "%s %s" $1 $2) (format "%s" $1))) ; trailer_opt : ;;EMPTY | trailer ; exponent_opt : ;;EMPTY | EXPONENT factor (format "** %s" $2) ; ;; atom: '(' [testlist] ')' | '[' [listmaker] ']' | '{' [dictmaker] '}' | '`' testlist '`' | NAME | NUMBER | STRING+ atom : NAME | NUMBER_LITERAL | one_or_more_string ; ;; Use (read $1) to peel away the double quotes. one_or_more_string : STRING_LITERAL (read $1) | one_or_more_string STRING_LITERAL (if $1 (format "%s %s" $1 (read $2)) (format "%s" (read $2))) ; ;; listmaker: test ( list_for | (',' test)* [','] ) listmaker : test listmaker_trailer ; ;; ( list_for | (',' test)* [','] ) listmaker_trailer : list_for | testlist_trailer comma_opt ; ;; lambdef: 'lambda' [varargslist] ':' test lambdef : LAMBDA COLON test (format "%s %s %s" $1 $2 $3) ; ;; trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME trailer : POPEN arglist_opt PCLOSE | BOPEN subscriptlist BCLOSE | PERIOD NAME ; ;; [arglist] arglist_opt : ;;EMPTY | arglist ; ;; subscriptlist: subscript (',' subscript)* [','] subscriptlist : comma_sep_subscript_list comma_opt ; ;; subscript (',' subscript)* comma_sep_subscript_list : subscript | comma_sep_subscript_list COMMA subscript ; ;; subscript: '.' '.' '.' | test | [test] ':' [test] [sliceop] subscript : PERIOD PERIOD PERIOD | test | zero_or_one_test COLON zero_or_one_test zero_or_one_sliceop ; ;; [sliceop] zero_or_one_sliceop : ;;EMPTY | sliceop ; ;; sliceop: ':' [test] sliceop : COLON zero_or_one_test ; ;; [test] zero_or_one_test : ;;EMPTY | test ; ;; exprlist: expr (',' expr)* [','] exprlist : expr_list comma_opt ; ;; expr (',' expr)* expr_list : expr | expr_list COMMA expr ; ;; testlist: test (',' test)* [','] testlist : test testlist_trailer comma_opt (if $3 (if $2 (format "%s $2 %s" $1 $2 $3) (format "%s ** %s" $1 $3)) (if $2 (format "%s %s" $1 $2) (format "%s" $1))) ; ;; (',' test)* testlist_trailer : ;;EMPTY | testlist_trailer COMMA test (if $1 (format "%s %s %s" $1 $2 $3) (format "%s %s" $2 $3)) ; ;; [','] comma_opt : ;;EMPTY | COMMA ; ;; testlist_safe: test [(',' test)+ [',']] testlist_safe : test testlist_safe_trailer_opt (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; [(',' test)+ [',']] testlist_safe_trailer_opt : ;; EMPTY | testlist_safe_term comma_opt (if $2 (format "%s %s" $1 $2) (format "%s" $1)) ; ;; (',' test)+ testlist_safe_term : COMMA test (format "%s %s" $1 $2) | testlist_safe_term COMMA test (format "%s %s %s" $1 $2 $3) ; ;; dictmaker: test ':' test (',' test ':' test)* [','] dictmaker : test COLON test colon_sep_test comma_opt ; ;; (',' test ':' test)* colon_sep_test : ;;EMPTY | colon_sep_test COMMA test COLON test ; ;; classdef: 'class' NAME ['(' testlist ')'] ':' suite classdef : CLASS NAME paren_testlist_opt COLON suite ; ;; ['(' testlist ')'] paren_testlist_opt : ;;EMPTY | POPEN testlist PCLOSE ; ;; arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) arglist : argument_comma_zom arglist_trailer ; ;; (argument ',')* argument_comma_zom : ;;EMPTY | argument_comma_zom argument COMMA ; ;; (argument [',']| '*' test [',' '**' test] | '**' test) arglist_trailer : argument comma_opt | MULT test comma_mult_mult_test_opt | EXPONENT test ; ;; [',' '**' test] comma_mult_mult_test_opt : ;;EMPTY | COMMA EXPONENT test ; ;; argument: [test '='] test # Really [keyword '='] test argument : test_eq_opt test ; ;; [test '='] test_eq_opt : ;;EMPTY | test ASSIGN ; ;; list_iter: list_for | list_if list_iter : list_for | list_if ; ;; list_for: 'for' exprlist 'in' testlist_safe [list_iter] list_for : FOR exprlist IN testlist_safe list_iter_opt ; ;; list_if: 'if' test [list_iter] list_if : IF test list_iter_opt ; ;; [list_iter] list_iter_opt : ;;EMPTY | list_iter ; ;;; wisent-python.wy ends here |
From: Eric M. L. <er...@si...> - 2002-06-12 12:42:56
|
>>> pon...@ne... (David Ponce) seems to think that: [ ... ] >> Lastly, I thought it would be useful to build an EIEIO class to >>handle modes and overrides. This would bring a more traditional OO >>framework to how parsers are specialized. This would make some >>aspects easier to program, and is probably a longer term goal. > >I agree with you on this point even if it surprises me a little bit! >If I correctly remember, sometimes ago you told me that you didn't >want to use EIEIO in semantic's core, because that would probably >prevent a future inclusion of semantic in GNU/Emacs! Have you changed >your mind? > >As I talk of inclusion in GNU/Emacs you might be interested to know >that I signed a global copyright assignment for my work on GNU/Emacs >:-) So there shouldn't be any problem if you still have plan to >include [parts of] semantic in the official distribution. There were two things I was worried about with EIEIO. One was, as you say, adding a dependency that might prevent it being distributed in GNU Emacs. Also, RMS doesn't like core tools in Emacs depending on CL macros. As RMS is also interested in having EIEIO be a part of Emacs, that eliminates one worry. As I already used EIEIO in semanticdb, I think the programatic benefits outweigh (for myself) any style opinion RMS may have. I may also just be that I'm getting ornery. The second reason was that I didn't want EIEIO macros in what needs to be high-performance code (ie, as a way of defining tokens, or executing methods in the parser.) I still think this is true. I think EIEIO would be best used as a way of wrapping up high level APIs where the number of iterations over the EIEIO wrappers is minimal. >[...] >> If it is relevant, could you put this into the 1.4 version of >>semantic.el? Thanks. > >Of course! I will do that as soon as I had tested the new code a >little bit more. I was referring to just the one bug-fix you mentioned in your previous patch. I did eventually look at it yesterday, but it was unclear to me if it translates into the old code. I think the new API should wait till after 1.4. I checked in some of the doc files (like INSTALL and semantic.texi) redirecting help requests to go to the mailing list. Hopefully that will ease up on the amount of email I get. I still have a pile of patches to the c parser to add in. I'll make an extra effort to make that tar ball. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: Eric M. L. <er...@si...> - 2002-06-12 12:29:18
|
Great news! >>> "Richard Y. Kim" <ry...@ds...> seems to think that: >David and Eric, > >Here is the updated wisent-python.wy file. >I've added all the rules from the official Grammar file as comments, >then translated them into wisent syntax. > >The state of this file is: >* M-x wisent-wy-update-outputfile finishes successfully >* The resulting table parses simple statements. > >What needs to be done: >* Lots of debugging/verification is needed. As you develop test cases, please add a file to the "tests" directory which you use for python testing. I tend to include various obscure language features, including a few that don't parse (so I can see the red underlines for unparsable text.) >* Add proper semantic rules so that the tokens generated by the parser > is usable by other tools down the line. >* Introduce operator precedence which should simplify the grammar. > >Any feedback, suggestions, criticism would be appreciated as always. [ ... ] Have fun Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: <pon...@ne...> - 2002-06-12 12:27:36
|
Eric, I did some more changes in semantic related to parser plug-in API. Attached you will find a new tarball with the updated files. Here is a summary: In semantic.el: - Added a new buffer-local variable `semantic-bovinate-parser-name' that can be used to identify the parser currently used. A new function `semantic-bovination-working-message' uses it to produce a suitable working message while parsing. I think it is interesting to know which parser is currently running ;-) - Now `semantic-bovinate-nonterminals' sets the 'reparse-symbol property only if `semantic-raw-to-cooked-token' didn't set it before. Thus, it is possible to set the 'reparse-symbol in the `semantic-expand-nonterminal' function! I used this possibility in `wisent-wy-expand-nonterminal' to correctly setup the re-parse symbol in tokens don't produced by the EXPANDFULL mechanism. Cool :-) In semantic-ctxt.el: - Moved the working- stuff from `semantic-get-local-variables-default' to `semantic-get-local-variables', so overriding functions don't have to worry about that. Also removed unnecessary call to `semantic-deoverlay-list' in `semantic-get-local-variables'. The changes in the other files are related to the above enhancements. What do you think? David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: <pon...@ne...> - 2002-06-12 09:53:19
|
Hi Eric, > In the past we had chatted on several topics which I'll repeat here >without knowing if you took them into account. > > I thought it would be good to put the lexical functions into it's >own file (semantic-lex.el) This could help formalize what is lexical >and what is not. > > I also think it would be good to put the LL parser specific stuff >into it's own file (semantic-bovine.el or some-such.) This would >mimick semantic-wisent more, and make it clearer that parsers are >more of a plug-in. Good ideas! For now I would prefer to stabilize the new plug-in API code. Nevertheless, no problem for me if you want to start creating these new files ;-) > Lastly, I thought it would be useful to build an EIEIO class to >handle modes and overrides. This would bring a more traditional OO >framework to how parsers are specialized. This would make some >aspects easier to program, and is probably a longer term goal. I agree with you on this point even if it surprises me a little bit! If I correctly remember, sometimes ago you told me that you didn't want to use EIEIO in semantic's core, because that would probably prevent a future inclusion of semantic in GNU/Emacs! Have you changed your mind? As I talk of inclusion in GNU/Emacs you might be interested to know that I signed a global copyright assignment for my work on GNU/Emacs :-) So there shouldn't be any problem if you still have plan to include [parts of] semantic in the official distribution. [...] > If it is relevant, could you put this into the 1.4 version of >semantic.el? Thanks. Of course! I will do that as soon as I had tested the new code a little bit more. > I suspect I should really produce a 1.4 distribution so we can move >on to other things more effectively. I think so ;-) Thanks! David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: Richard Y. K. <ry...@ds...> - 2002-06-12 07:49:05
|
David and Eric, Here is the updated wisent-python.wy file. I've added all the rules from the official Grammar file as comments, then translated them into wisent syntax. The state of this file is: * M-x wisent-wy-update-outputfile finishes successfully * The resulting table parses simple statements. What needs to be done: * Lots of debugging/verification is needed. * Add proper semantic rules so that the tokens generated by the parser is usable by other tools down the line. * Introduce operator precedence which should simplify the grammar. Any feedback, suggestions, criticism would be appreciated as always. |
From: Eric M. L. <er...@si...> - 2002-06-11 19:55:04
|
Hi David, I read your message, and it sounds great. I will have to poke around your diff to see how it all works. In the past we had chatted on several topics which I'll repeat here without knowing if you took them into account. I thought it would be good to put the lexical functions into it's own file (semantic-lex.el) This could help formalize what is lexical and what is not. I also think it would be good to put the LL parser specific stuff into it's own file (semantic-bovine.el or some-such.) This would mimick semantic-wisent more, and make it clearer that parsers are more of a plug-in. Lastly, I thought it would be useful to build an EIEIO class to handle modes and overrides. This would bring a more traditional OO framework to how parsers are specialized. This would make some aspects easier to program, and is probably a longer term goal. >- Fixed a little problem in `semantic-bovinate-nonterminals' that > didn't correctly setup the 'reparse-symbol property of expanded > tokens. If it is relevant, could you put this into the 1.4 version of semantic.el? Thanks. I suspect I should really produce a 1.4 distribution so we can move on to other things more effectively. Thanks Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: <pon...@ne...> - 2002-06-11 13:11:33
|
Hi Eric, I finally got time to work on an uniform way to plug the different parsers into Semantic :-) Attached you will find a tarball of all the files I updated (mainly in wisent stuff) to achieve that. Just notice that this work, even if all seems to work well, is probably alpha-release quality and needs more testing ;-) The tarball contains a ChangeLog. Here is an overview of new design. After studying the code in semantic.el I observed that the only function which is actually specific to the parser is `semantic-bovinate-nonterminal'. All other parser related functions rely on it! So I concluded that the more natural way to plug new parsers in Semantic would be to allow overriding of that function! IMO, this design has several advantages: - It is very simple, only one function need to be overrode. - It permits to reuse most of the existing parsing code. - Semantic's behavior is consistent, as it not depends on the parser actually used. The main requirement with a such design is that the new parser must be able to parse one nonterminal at a time! Because `semantic-bovinate-nonterminal' must return one raw semantic token at each call. Fortunately the current LL and LALR parsers can do that :-) In the attached first implementation, I made the following changes in semantic.el: - Removed all references to the default nonterminal symbol 'bovine-toplevel which is specific to the LL parser. Now the default nonterminal is nil! These references were unnecessary because `semantic-bovinate-nonterminal' already used 'bovine-toplevel as the default nonterminal value. - Created a new buffer-local variable `semantic-bovinate-parser' that specifies the override function for `semantic-bovinate-nonterminal'. - Renamed `semantic-bovinate-nonterminal' to `semantic-bovinate-nonterminal-default'. - Implemented `semantic-bovinate-nonterminal' as a wrapper to call the function specified in `semantic-bovinate-parser', or to call `semantic-bovinate-nonterminal-default' by default. - Fixed a little problem in `semantic-bovinate-nonterminals' that didn't correctly setup the 'reparse-symbol property of expanded tokens. The rest of my work consisted in updating Wisent's stuff to obey the new API! As you will notice I removed a lot of code no more necessary, from wisent-bovine.el ;-) Unfortunately the new design has broken the wisent-java.wy/el implementation which was designed to parse a Java source in one pass like traditional bison use _ that is the grammar returns a list of tokens ready for use instead of one raw token as required by the `semantic-bovinate-nonterminal' API :-( IMO, this is not really a big issue. It is far better to use the "semantic orthodox" implementation in wisent-java-tags.wy/el! I am looking forward for you remarks ;-) Thank you for your time! David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: David P. <da...@dp...> - 2002-06-07 18:33:26
|
Richard >I managed to add lots of non-terminals to python grammar. >I am currently blindly moving over the python grammar as is >from the official Grammar file to semantic/wisent. >In particular, I am not yet using operator precedence >capability of wisent which I think will simplify the grammar >a lot. I'll get to it later. > Great news! I just checked the new wisent-flex.el with the handler feature ;-) David |
From: Richard Y. K. <ry...@ds...> - 2002-06-07 16:18:26
|
David, Fine with me. I managed to add lots of non-terminals to python grammar. I am currently blindly moving over the python grammar as is from the official Grammar file to semantic/wisent. In particular, I am not yet using operator precedence capability of wisent which I think will simplify the grammar a lot. I'll get to it later. >>>>> "DP" == David Ponce <pon...@ne...> writes: DP> DP> Hi Richard & Eric, DP> I would like to check the new version of `wisent-flex' in. Any DP> objection? DP> DP> Also Eric did you read the `semantic.texi' patch I submitted to you, DP> following latest updates to `semantic-flex'? Can I check it in? |
From: Eric M. L. <er...@si...> - 2002-06-07 13:43:45
|
>>> pon...@ne... (David Ponce) seems to think that: >Hi Richard & Eric, > >I would like to check the new version of `wisent-flex' in. Any >objection? Do you mean semantic-flex, or did the bol feature move into wisent-flex? I lost track. Either way, that is fine with me. >Also Eric did you read the `semantic.texi' patch I submitted to you, >following latest updates to `semantic-flex'? Can I check it in? [ ... ] Yes, please do. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: <pon...@ne...> - 2002-06-07 11:47:05
|
Hi Richard & Eric, I would like to check the new version of `wisent-flex' in. Any objection? Also Eric did you read the `semantic.texi' patch I submitted to you, following latest updates to `semantic-flex'? Can I check it in? David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: Eric M. L. <er...@si...> - 2002-06-06 17:46:17
|
>>> pon...@ne... (David Ponce) seems to think that: >Hi Eric, > >[...] >> I like the idea of handlers. It is similar to the flex extensions. >> It would be nice if we could get all the different types of extensions >> down into a common set of constructs. This makes a fourth style of >> specialized lexical adaptation. (the first three being the syntax >> table, flags, then the extensions.) >> >> Perhaps when we combine the lexical notions of the LL and LALR >> parsers, we could have one batch of handlers, and enabling whitespace, >> newlines, comments, etc are all different handlers that you explicitly >> add instead of flags. That would be more efficient. Unfortunately, >> it seems like your handlers run at a different level from the >> handlers I'm thinking of. > >I must confess I don't clearly understand what this means: "your >handlers run at a different level from the handlers I'm thinking >of". Could you please elaborate? The current semantic-flex extensions are regexp that look at text. If I understood your handlers properly, they filter already generated tokens. >I already thought about a such architecture for lexical analysis based >on handlers instead of options. It would allow to easily extend the >lexer capabilities by creating new handlers. It could be also a good >mechanism to unify `semantic-flex' and `wisent-flex'. Each parser >could provide a set of "standard" handlers to get the lexical tokens >they need, with an appropriate format. Then each language could >extend the standard set depending on its requirements. This would certainly be the ideal. >Compared to the current implementation of `semantic-flex' and >`wisent-flex' an implementation based on handlers will result in a lot >of function calls. So I am not sure it will be more efficient. If handlers were macros, we could build specialized lexers by constructing our own code out of them, eliminating function calls. Similar to the way we build elisp code from the grammar. >I also thought about a different approach based on filtering of >syntactic tokens, which is close of the current way `semantic-flex' >and `wisent-flex' work. > >Instead of using options to include some category of tokens, >`semantic-flex' could simply convert all input data into a stream of >all the sort of syntactic tokens it found (like if all >`semantic-flex-enable-...' options were enabled). This could miss specialized regexp based tokens as created by semantic-make.el which converts otherwise normal looking text into bigger blocks. >This intermediate generic stream would be very easy to digest by >specific lexers (like 'wisent-flex'). When called, such lexers would >return the next lexical token available or EOI, eliminating unneeded >tokens, and maybe doing more sophisticated processing too. That is a nifty idea. I have read some code in Emacs that goes out of it's way to avoid consing, as if it were a bad thing. This approach would add more consing, but certainly simplify use of it. >A set of default lexers could be provided for each parser. The >simplest one will probably look like this: > >(defun semantic-default-lexer () > (prog1 (car semantic-token-stream) > (setq semantic-token-stream (cdr semantic-token-stream)))) > >A more sophisticated one is `wisent-flex' ;-) > >The issue here would be to adapt the current LL parser so it would be >able to get lexical tokens one at a time. That would be quite the trick since it uses the call stack to unwind errors. ;) The LL parser could just run the filter on the whole input stream. >Globally I think this approach should be faster than the handler's one! > >> Next, I'm a little wary of putting lexical information in the grammar. >> (Yes, I know we already put some of the lexical flags in the grammar >> file too.) If we think it is good to put lexical information in the >> grammar, we should probably design .wy grammar features for other >> parts too the way lex does perhaps. Eeks. > >IMO, having both lexical and syntactic informations in one file is a >good thing. A lot of new parsers adopted a such design. In fact I >think it would nice to also include all the related ELisp code in the >grammar file. So the corresponding .el file could be entirely >generated. And even better only the byte-compiled .elc file could be >directly produced ;-) [ ... ] I tend to agree though I am concerned of bucking the "standard" of flex & bison. I always wanted to have the .bnf (or whatever extension we use) include a full definition of the syntax table as well. Such a definition could have more details in it for our lexical purposes (like numbers,) and then be reduced to the more typical syntax table for the language mode. Such a definition could be used to construct a specialized lexer (as per discussion above) using various handlers. Nifty. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |
From: <pon...@ne...> - 2002-06-06 14:14:02
|
Hi Eric, [...] > I like the idea of handlers. It is similar to the flex extensions. > It would be nice if we could get all the different types of extensions > down into a common set of constructs. This makes a fourth style of > specialized lexical adaptation. (the first three being the syntax > table, flags, then the extensions.) > > Perhaps when we combine the lexical notions of the LL and LALR > parsers, we could have one batch of handlers, and enabling whitespace, > newlines, comments, etc are all different handlers that you explicitly > add instead of flags. That would be more efficient. Unfortunately, > it seems like your handlers run at a different level from the > handlers I'm thinking of. I must confess I don't clearly understand what this means: "your handlers run at a different level from the handlers I'm thinking of". Could you please elaborate? I already thought about a such architecture for lexical analysis based on handlers instead of options. It would allow to easily extend the lexer capabilities by creating new handlers. It could be also a good mechanism to unify `semantic-flex' and `wisent-flex'. Each parser could provide a set of "standard" handlers to get the lexical tokens they need, with an appropriate format. Then each language could extend the standard set depending on its requirements. Compared to the current implementation of `semantic-flex' and `wisent-flex' an implementation based on handlers will result in a lot of function calls. So I am not sure it will be more efficient. I also thought about a different approach based on filtering of syntactic tokens, which is close of the current way `semantic-flex' and `wisent-flex' work. Instead of using options to include some category of tokens, `semantic-flex' could simply convert all input data into a stream of all the sort of syntactic tokens it found (like if all `semantic-flex-enable-...' options were enabled). This intermediate generic stream would be very easy to digest by specific lexers (like 'wisent-flex'). When called, such lexers would return the next lexical token available or EOI, eliminating unneeded tokens, and maybe doing more sophisticated processing too. A set of default lexers could be provided for each parser. The simplest one will probably look like this: (defun semantic-default-lexer () (prog1 (car semantic-token-stream) (setq semantic-token-stream (cdr semantic-token-stream)))) A more sophisticated one is `wisent-flex' ;-) The issue here would be to adapt the current LL parser so it would be able to get lexical tokens one at a time. Globally I think this approach should be faster than the handler's one! > Next, I'm a little wary of putting lexical information in the grammar. > (Yes, I know we already put some of the lexical flags in the grammar > file too.) If we think it is good to put lexical information in the > grammar, we should probably design .wy grammar features for other > parts too the way lex does perhaps. Eeks. IMO, having both lexical and syntactic informations in one file is a good thing. A lot of new parsers adopted a such design. In fact I think it would nice to also include all the related ELisp code in the grammar file. So the corresponding .el file could be entirely generated. And even better only the byte-compiled .elc file could be directly produced ;-) [...] Thank you for your time! David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |
From: Richard Y. K. <ry...@ds...> - 2002-06-06 05:39:40
|
David, Thanks again for your help. So far so good. I plan on building up the grammar bottom up starting with the basic expressions. I'll keep you posted. >>>>> "DP" == David Ponce <da...@dp...> writes: DP> DP> Richard, DP> I worked a little more on your python stuff and submit you a new DP> version based on a new feature I added to `wisent-flex' DP> (in the attached tarball). DP> DP> It is now possible to setup a 'handler property to `semantic-flex' DP> token, which specifies a function to be called by `wisent-flex' to DP> handle such tokens. DP> DP> So with this new feature it is no more necessary to wrap DP> `wisent-flex'. I just wrote a specific 'bol handler called DP> `wisent-python-lex-bol' and setup the grammar accordingly: DP> DP> %token <bol> INDENT DEDENT DP> %put bol handler wisent-python-lex-bol DP> DP> What do you think? Eric, any thoughts? DP> DP> David |
From: Eric M. L. <er...@si...> - 2002-06-05 21:39:37
|
>>> David Ponce <da...@dp...> seems to think that: >Richard, > >I worked a little more on your python stuff and submit you a new >version based on a new feature I added to `wisent-flex' >(in the attached tarball). > >It is now possible to setup a 'handler property to `semantic-flex' >token, which specifies a function to be called by `wisent-flex' to >handle such tokens. > >So with this new feature it is no more necessary to wrap >`wisent-flex'. I just wrote a specific 'bol handler called >`wisent-python-lex-bol' and setup the grammar accordingly: > >%token <bol> INDENT DEDENT >%put bol handler wisent-python-lex-bol > >What do you think? Eric, any thoughts? [ ... ] I like the idea of handlers. It is similar to the flex extensions. It would be nice if we could get all the different types of extensions down into a common set of constructs. This makes a fourth style of specialized lexical adaptation. (the first three being the syntax table, flags, then the extensions.) Perhaps when we combine the lexical notions of the LL and LALR parsers, we could have one batch of handlers, and enabling whitespace, newlines, comments, etc are all different handlers that you explicitly add instead of flags. That would be more efficient. Unfortunately, it seems like your handlers run at a different level from the handlers I'm thinking of. Next, I'm a little wary of putting lexical information in the grammar. (Yes, I know we already put some of the lexical flags in the grammar file too.) If we think it is good to put lexical information in the grammar, we should probably design .wy grammar features for other parts too the way lex does perhaps. Eeks. Anyway, just worried about run-away feature-itis. It would be nice to take the wide range of features and tidy them up. Eric -- Eric Ludlam: za...@gn..., er...@si... Home: www.ultranet.com/~zappo Siege: www.siege-engine.com Emacs: http://cedet.sourceforge.net GNU: www.gnu.org |