Hi Richard,
Thanks for looking into making more changes. The only test suite is a
very brief parser in semantic-utest.el. All known tests run by doing a
"make utest" at the top level.
What would be better would be to create a python test for the smart
completion engine. This has a side effect of testing the parsers on
complex constructs, in addition to how it works with the completion
engine. This is in semantic-ia-utest.el. All that is needed to write
the test is knowledge of how the target language works, and the
annotation syntax. Add the new python file to
semantic/tests/testwhatever.py, then add that file name into a variable
at the top of semantic-ia-utest.el.
In python, add the syntax. semantic/tests/testdoublens.cpp shows how to
use the -1-, #1# syntax for completions. This would be particularly
useful in testing with statements which I assume are like using
statements in C++?
The basic idea behind the "parse it later" business is that brace or
paren blocks can be lexically analyzed and skipped in Emacs very
quickly. By using that you can then parse a class very fast, then parse
the contents to add the :members. Since it recurses the internal parse
is very robust in skipping bogus syntax, and protects the outer class
syntax. Thus if you have a rule like this for wisent-java-tags.wy:
class_declaration
: modifiers_opt CLASS qualified_name superc_opt interfaces_opt class_body
(TYPE-TAG $3 $2 $6 (if (or $4 $5) (cons $4 $5)) :typemodifiers $1)
;
you see that class_body is where the members will come from. That rule
is this:
class_body
: BRACE_BLOCK
(EXPANDFULL $1 class_member_declaration)
;
Thus a BRACE_BLOCK, which is quickly skipped, and then the code
explicitly recurses with the EXPANDFULL macro, which then does the
lexical pass in that block, plus a full robust parse, and the returned
tag list becomes the child list for the type.
This also allows the partial reparse engine to work. Thus if some huge
file takes 5 seconds to parse, and you edit it, it will parse only what
you changed, likely in a few millisenconds.
Thanks!
Eric
On 08/21/2010 01:02 PM, Richard Kim wrote:
> Here is an updated patch that tweaks the grammar to support if-else
> clause as well as 'with' statements.
>
> I have no doubt that there are lots of room for improvement in the
> python grammar especially for optimization. I was learning
> semantic/wisent when I first wrote the grammar and I was pretty much
> blindly porting official grammar into wisent framework.
> I still have not learned the wisent framework well enough to
> improve the python grammar.
>
> I believe semantic/wisent provides a way to delay parsing low level
> constructs so that parsing is done to pick out the first or second level
> objects such as classes and functions, then delay parsing the contents
> of these later on as needed. I never learned how to do that so if
> anyone on this list knows how to do that, please go ahead and make that
> change. As I read wisent-c.wy, I can kind of understand the concept.
> For example skip-paren-block and skip-brace-block in wisent-c.wy
> seems to implement this idea of "skip parsing the details for now".
> Perhaps I'll figure this out at some point, but I bet others on this
> list can do this much faster than me.
>
>
> === modified file 'cedet-1.0pre7/semantic/wisent/wisent-python.wy'
> --- cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-04-18 00:22:45 +0000
> +++ cedet-1.0pre7/semantic/wisent/wisent-python.wy 2010-08-21 16:45:00 +0000
> @@ -28,7 +28,9 @@
> ;;; Commentary:
> ;;
> ;; This is an LALR python parser that follows the official python
> -;; grammar closely with very few exceptions.
> +;; grammar at<http://docs.python.org/reference/grammar.html>
> +;; closely with very few exceptions.
> +
> ;;
> ;;; To do:
> ;;
> @@ -258,6 +260,10 @@
> %put WHILE summary
> "Start a 'while' loop"
>
> +%keyword WITH "with"
> +%put WITH summary
> +"Start a 'with' block"
> +
> %keyword YIELD "yield"
> %put YIELD summary
> "Create a generator function"
> @@ -600,6 +606,7 @@
> | while_stmt
> | for_stmt
> | try_stmt
> + | with_stmt
> | funcdef
> | class_declaration
> ;
> @@ -707,6 +714,31 @@
> ;
>
> ;;;============================================================================
> +;;;@@ with_stmt
> +;;;============================================================================
> +
> +;; with_stmt: 'with' with_item (',' with_item)* ':' suite
> +with_stmt
> + : WITH comma_sep_with_item COLON suite
> + ;
> +
> +;; with_item (',' with_item)*
> +comma_sep_with_item
> + : with_item
> + | comma_sep_with_item COMMA with_item
> + ;
> +
> +;; with_item: test ['as' expr]
> +with_item
> + : test as_expr_opt
> + ;
> +
> +as_expr_opt
> + : ;;EMPTY
> + | AS expr
> + ;
> +
> +;;;============================================================================
> ;;;@@ funcdef
> ;;;============================================================================
>
> @@ -793,16 +825,21 @@
> ;;;@ test
> ;;;****************************************************************************
>
> -;; test: and_test ('or' and_test)* | lambdef
> +;; test: or_test if_else_opt | lambdef
> test
> - : test_test
> + : or_test if_else_opt
> | lambdef
> ;
>
> +if_else_opt
> + : ;;EMPTY
> + | IF or_test ELSE test
> + ;
> +
> ;; and_test ('or' and_test)*
> -test_test
> +or_test
> : and_test
> - | test_test OR and_test
> + | or_test OR and_test
> ()
> ;
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by
>
> Make an app they can't live without
> Enter the BlackBerry Developer Challenge
> http://p.sf.net/sfu/RIM-dev2dev
> _______________________________________________
> Cedet-devel mailing list
> Cedet-devel@...
> https://lists.sourceforge.net/lists/listinfo/cedet-devel
>
|