Thread: [CEDET-devel] further python grammar tweaks
Brought to you by:
zappo
From: Richard K. <em...@gm...> - 2010-08-21 17:02:20
|
Here is an updated patch that tweaks the grammar to support if-else clause as well as 'with' statements. I have no doubt that there are lots of room for improvement in the python grammar especially for optimization. I was learning semantic/wisent when I first wrote the grammar and I was pretty much blindly porting official grammar into wisent framework. I still have not learned the wisent framework well enough to improve the python grammar. I believe semantic/wisent provides a way to delay parsing low level constructs so that parsing is done to pick out the first or second level objects such as classes and functions, then delay parsing the contents of these later on as needed. I never learned how to do that so if anyone on this list knows how to do that, please go ahead and make that change. As I read wisent-c.wy, I can kind of understand the concept. For example skip-paren-block and skip-brace-block in wisent-c.wy seems to implement this idea of "skip parsing the details for now". Perhaps I'll figure this out at some point, but I bet others on this list can do this much faster than me. === modified file 'cedet-1.0pre7/semantic/wisent/wisent-python.wy' --- cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-04-18 00:22:45 +0000 +++ cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-08-21 16:45:00 +0000 @@ -28,7 +28,9 @@ ;;; Commentary: ;; ;; This is an LALR python parser that follows the official python -;; grammar closely with very few exceptions. +;; grammar at <http://docs.python.org/reference/grammar.html> +;; closely with very few exceptions. + ;; ;;; To do: ;; @@ -258,6 +260,10 @@ %put WHILE summary "Start a 'while' loop" +%keyword WITH "with" +%put WITH summary +"Start a 'with' block" + %keyword YIELD "yield" %put YIELD summary "Create a generator function" @@ -600,6 +606,7 @@ | while_stmt | for_stmt | try_stmt + | with_stmt | funcdef | class_declaration ; @@ -707,6 +714,31 @@ ; ;;;============================================================================ +;;;@@ with_stmt +;;;============================================================================ + +;; with_stmt: 'with' with_item (',' with_item)* ':' suite +with_stmt + : WITH comma_sep_with_item COLON suite + ; + +;; with_item (',' with_item)* +comma_sep_with_item + : with_item + | comma_sep_with_item COMMA with_item + ; + +;; with_item: test ['as' expr] +with_item + : test as_expr_opt + ; + +as_expr_opt + : ;;EMPTY + | AS expr + ; + +;;;============================================================================ ;;;@@ funcdef ;;;============================================================================ @@ -793,16 +825,21 @@ ;;;@ test ;;;**************************************************************************** -;; test: and_test ('or' and_test)* | lambdef +;; test: or_test if_else_opt | lambdef test - : test_test + : or_test if_else_opt | lambdef ; +if_else_opt + : ;;EMPTY + | IF or_test ELSE test + ; + ;; and_test ('or' and_test)* -test_test +or_test : and_test - | test_test OR and_test + | or_test OR and_test () ; |
From: Eric M. L. <er...@si...> - 2010-08-22 15:42:33
|
Hi Richard, Thanks for looking into making more changes. The only test suite is a very brief parser in semantic-utest.el. All known tests run by doing a "make utest" at the top level. What would be better would be to create a python test for the smart completion engine. This has a side effect of testing the parsers on complex constructs, in addition to how it works with the completion engine. This is in semantic-ia-utest.el. All that is needed to write the test is knowledge of how the target language works, and the annotation syntax. Add the new python file to semantic/tests/testwhatever.py, then add that file name into a variable at the top of semantic-ia-utest.el. In python, add the syntax. semantic/tests/testdoublens.cpp shows how to use the -1-, #1# syntax for completions. This would be particularly useful in testing with statements which I assume are like using statements in C++? The basic idea behind the "parse it later" business is that brace or paren blocks can be lexically analyzed and skipped in Emacs very quickly. By using that you can then parse a class very fast, then parse the contents to add the :members. Since it recurses the internal parse is very robust in skipping bogus syntax, and protects the outer class syntax. Thus if you have a rule like this for wisent-java-tags.wy: class_declaration : modifiers_opt CLASS qualified_name superc_opt interfaces_opt class_body (TYPE-TAG $3 $2 $6 (if (or $4 $5) (cons $4 $5)) :typemodifiers $1) ; you see that class_body is where the members will come from. That rule is this: class_body : BRACE_BLOCK (EXPANDFULL $1 class_member_declaration) ; Thus a BRACE_BLOCK, which is quickly skipped, and then the code explicitly recurses with the EXPANDFULL macro, which then does the lexical pass in that block, plus a full robust parse, and the returned tag list becomes the child list for the type. This also allows the partial reparse engine to work. Thus if some huge file takes 5 seconds to parse, and you edit it, it will parse only what you changed, likely in a few millisenconds. Thanks! Eric On 08/21/2010 01:02 PM, Richard Kim wrote: > Here is an updated patch that tweaks the grammar to support if-else > clause as well as 'with' statements. > > I have no doubt that there are lots of room for improvement in the > python grammar especially for optimization. I was learning > semantic/wisent when I first wrote the grammar and I was pretty much > blindly porting official grammar into wisent framework. > I still have not learned the wisent framework well enough to > improve the python grammar. > > I believe semantic/wisent provides a way to delay parsing low level > constructs so that parsing is done to pick out the first or second level > objects such as classes and functions, then delay parsing the contents > of these later on as needed. I never learned how to do that so if > anyone on this list knows how to do that, please go ahead and make that > change. As I read wisent-c.wy, I can kind of understand the concept. > For example skip-paren-block and skip-brace-block in wisent-c.wy > seems to implement this idea of "skip parsing the details for now". > Perhaps I'll figure this out at some point, but I bet others on this > list can do this much faster than me. > > > === modified file 'cedet-1.0pre7/semantic/wisent/wisent-python.wy' > --- cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-04-18 00:22:45 +0000 > +++ cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-08-21 16:45:00 +0000 > @@ -28,7 +28,9 @@ > ;;; Commentary: > ;; > ;; This is an LALR python parser that follows the official python > -;; grammar closely with very few exceptions. > +;; grammar at<http://docs.python.org/reference/grammar.html> > +;; closely with very few exceptions. > + > ;; > ;;; To do: > ;; > @@ -258,6 +260,10 @@ > %put WHILE summary > "Start a 'while' loop" > > +%keyword WITH "with" > +%put WITH summary > +"Start a 'with' block" > + > %keyword YIELD "yield" > %put YIELD summary > "Create a generator function" > @@ -600,6 +606,7 @@ > | while_stmt > | for_stmt > | try_stmt > + | with_stmt > | funcdef > | class_declaration > ; > @@ -707,6 +714,31 @@ > ; > > ;;;============================================================================ > +;;;@@ with_stmt > +;;;============================================================================ > + > +;; with_stmt: 'with' with_item (',' with_item)* ':' suite > +with_stmt > + : WITH comma_sep_with_item COLON suite > + ; > + > +;; with_item (',' with_item)* > +comma_sep_with_item > + : with_item > + | comma_sep_with_item COMMA with_item > + ; > + > +;; with_item: test ['as' expr] > +with_item > + : test as_expr_opt > + ; > + > +as_expr_opt > + : ;;EMPTY > + | AS expr > + ; > + > +;;;============================================================================ > ;;;@@ funcdef > ;;;============================================================================ > > @@ -793,16 +825,21 @@ > ;;;@ test > ;;;**************************************************************************** > > -;; test: and_test ('or' and_test)* | lambdef > +;; test: or_test if_else_opt | lambdef > test > - : test_test > + : or_test if_else_opt > | lambdef > ; > > +if_else_opt > + : ;;EMPTY > + | IF or_test ELSE test > + ; > + > ;; and_test ('or' and_test)* > -test_test > +or_test > : and_test > - | test_test OR and_test > + | or_test OR and_test > () > ; > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by > > Make an app they can't live without > Enter the BlackBerry Developer Challenge > http://p.sf.net/sfu/RIM-dev2dev > _______________________________________________ > Cedet-devel mailing list > Ced...@li... > https://lists.sourceforge.net/lists/listinfo/cedet-devel > |