Hi Eric,
I submit you the following patch to semantic.
The main changes are:
* In the idle scheduler:
=20
The `semantic-idle-core-handler' now ignores non file buffers to
speedup things. A more important change is that the
`semantic-idle-scheduler-function' now activates the core handler
even if the idle scheduler is not enabled in current buffer. This
let a chance to process other idle-enabled buffers during idle time.
* In python support:
I fixed some minor bugs in the grammar, and implemented
auto-generation of lexical analyzers when possible. The main
changes are in wisent-python.el. I completely reworked the lexer to
simplify the code, make it more robust, and more compliant with
python 2.3 specs. The new code no more advices the built-in
`scan-lists'. IMO, it is better to use semantic overloads to
implement the missing python-mode overrides, particularly the
"context" API.
I would greatly appreciate your feedback, and if you don't have
objection I would like to commit these changes.
Thank you so much for your time!
David
2004-09-06 David Ponce <david@...>
=09* cedet/semantic/semantic.el
=09(semantic-new-buffer-fcn): Set semantic-new-buffer-fcn-was-run to
=09non-nil before calling semantic functions. Remove unnecessary
=09call to semantic-parse-tree-set-needs-rebuild.
=09(semantic-parse-region-default): Fix messages. Remove obsolete
=09code and comments.
=09* cedet/semantic/semantic-idle.el
=09(semantic-idle-core-handler): Ignore non file buffers. Check here
=09if the idle scheduler is enabled in current buffer. Use dolist
=09and with-current-buffer to simplify the code.
=09(semantic-idle-scheduler-function): Let a chance to process other
=09buffers even if the idle scheduler is not enabled in current
=09buffer. Ensure to always resume the idle scheduler timer. Don't
=09do unnecessary save match data.
=09* cedet/semantic/semantic-lex.el
=09(semantic-lex-init): Code cleanup.
=09(semantic-lex-unterminated-syntax-protection): Don't catch errors
=09when debug-on-error is set.
=09(semantic-lex-catch-errors): Code cleanup. Print the symbol on
=09which lexical analysis failed.
=09
=09* cedet/semantic/wisent/wisent-python.el
=09Remove obsolete code and update comments.
=09Use variable and function names consistent with the library name.
=09Cleanup and improve lexical analysis code.
=09
=09(wisent-python-explicit-line-continuation): Remove.
=09(wisent-python-string-re): Update from python 2.3 specs.
=09(wisent-python-indent-stack): Rename from
=09wisent-python-lexer-indent-stack. Move. All uses changed.
=09(semantic-lex-python-pop-indent-stack): Remove.
=09(wisent-python-implicit-line-joining-p)
=09(wisent-python-forward-string)
=09(wisent-python-forward-line-skip-indented)
=09(wisent-python-end-of-block): New functions.
=09(wisent-python-scan-lists, scan-lists): Remove.
=09(wisent-python-forward-line): Rename from python-next-line.
=09Simplify and make more robust.
=09(wisent-python-lex-beginning-of-line): Rename from
=09semantic-lex-python-beginning-of-line. All uses changed.
=09Simplify with use of above new functions. Don't hard code too
=09much things.
=09(semantic-lex-python-reset-continued-line): Remove.
=09(wisent-python-lex-end-of-line): Rename from
=09semantic-lex-python-newline. All uses changed. Does nothing if
=09implicit line joining. No more handle what remains in the
=09indentation stack.
=09(wisent-python-lex-string): Rename from
=09semantic-lex-python-string. All uses changed. Simplify with use
=09of wisent-python-forward-string.
=09(wisent-python-lex-ignore-backslash): Rename from
=09semantic-lex-python-charquote, simplify. All uses changed.
=09(semantic-lex-python-symbol, semantic-lex-python-number)
=09(semantic-lex-python-blocks): Remove. Auto-generate.
=09(wisent-python-lexer): Rename from semantic-python-lexer. All
=09uses changed. Use updated and auto-generated analyzers.
=09(semantic-parse-region): Remove.
=09(semantic-lex): Override in python-mode to handle the indentation
=09stack.
=09(semantic-get-local-variables): Move.
=09(wisent-python-default-setup): Set parse-sexp-ignore-comments to
=09non-nil. Remove unnecessary code.
=09(wisent-python-lex-buffer): Code cleanup.
=09* cedet/semantic/wisent/wisent-python.wy
=09Reorder and cleanup token declarations to avoid warnings.
=09Auto-generate symbol, keyword, number, block and punctuation
=09analyzers.
=09Use %keyword to declare keywords.
=09(class_declaration): Fix TYPE-TAG production.
=09(paren_class_list): Return a list of class names.
=09(paren_class): Handle dotted name.
Index: semantic/semantic-idle.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/semantic-idle.el,v
retrieving revision 1.28
diff -c -r1.28 semantic-idle.el
*** semantic/semantic-idle.el=0928 Jun 2004 13:58:00 -0000=091.28
--- semantic/semantic-idle.el=096 Sep 2004 12:11:15 -0000
***************
*** 218,278 ****
And also manages services that depend on tag values."
(semantic-exit-on-input 'idle-timer
(let* ((inhibit-quit nil)
! (mode major-mode)
! (buffers (delq (current-buffer) (buffer-list)))
! (others nil)
! (queue semantic-idle-scheduler-queue)
! ;; First, reparse the current buffer.
! (safe (semantic-idle-scheduler-refresh-tags)))
! ;; Now loop over other buffers with same major mode, trying to
! ;; update them as well. Stop on keypress.
! (save-excursion
! (while buffers
! (semantic-throw-on-input 'parsing-mode-buffers)
! (when (buffer-live-p (car buffers))
! (set-buffer (car buffers))
! (when (semantic-idle-scheduler-enabled-p)
! (if (eq major-mode mode)
! (semantic-idle-scheduler-refresh-tags)
! (setq others (cons (current-buffer) others)))))
! (setq buffers (cdr buffers))))
! ;; If re-parse of current buffer completed, evaluate all other
! ;; services. Stop on keypress.
! (when safe
(save-excursion
! (while queue
! (semantic-throw-on-input 'idle-queue)
! (funcall (car queue))
! (setq queue (cdr queue)))))
;; Finally loop over remaining buffers, trying to update them as
;; well. Stop on keypress.
(save-excursion
! (while others
(semantic-throw-on-input 'parsing-other-buffers)
! (set-buffer (car others))
! ;; No need to check more here?
! (semantic-idle-scheduler-refresh-tags)
! (setq others (cdr others))))
)))
=20
(defun semantic-idle-scheduler-function ()
"Function run when after `semantic-idle-scheduler-idle-time'.
This function will reparse the current buffer, and if successful,
call additional functions registered with the timer calls."
! (save-match-data
! (when (and (semantic-idle-scheduler-enabled-p)
! =09 (=3D (recursion-depth) 0))
!=20
! ;; Disable the auto parse timer while re-parsing
! (semantic-idle-scheduler-kill-timer)
!=20
! ;; Handle re-parsing and other scheduled services
! (semantic-safe "idle error: %S"
! =09(semantic-idle-core-handler))
!=20
;; Enable again the auto parse timer
(semantic-idle-scheduler-setup-timer))))
-=20
=0C
;;; REPARSING
;;
--- 218,275 ----
And also manages services that depend on tag values."
(semantic-exit-on-input 'idle-timer
(let* ((inhibit-quit nil)
! (buffers (delq (current-buffer)
! (delq nil
! (mapcar #'(lambda (b)
! (and (buffer-file-name b)
! b))
! (buffer-list)))))
! others safe mode)
! (when (semantic-idle-scheduler-enabled-p)
(save-excursion
! ;; First, reparse the current buffer.
! (setq mode major-mode
! safe (semantic-idle-scheduler-refresh-tags))
! ;; Now loop over other buffers with same major mode, trying to
! ;; update them as well. Stop on keypress.
! (dolist (b buffers)
! (semantic-throw-on-input 'parsing-mode-buffers)
! (with-current-buffer b
! (if (eq major-mode mode)
! (and (semantic-idle-scheduler-enabled-p)
! (semantic-idle-scheduler-refresh-tags))
! (push (current-buffer) others))))
! (setq buffers others))
! ;; If re-parse of current buffer completed, evaluate all other
! ;; services. Stop on keypress.
! (when safe
! (save-excursion
! (dolist (service semantic-idle-scheduler-queue)
! (semantic-throw-on-input 'idle-queue)
! (funcall service)))))
;; Finally loop over remaining buffers, trying to update them as
;; well. Stop on keypress.
(save-excursion
! (dolist (b buffers)
(semantic-throw-on-input 'parsing-other-buffers)
! (with-current-buffer b
! (and (semantic-idle-scheduler-enabled-p)
! (semantic-idle-scheduler-refresh-tags)))))
)))
=20
(defun semantic-idle-scheduler-function ()
"Function run when after `semantic-idle-scheduler-idle-time'.
This function will reparse the current buffer, and if successful,
call additional functions registered with the timer calls."
! (when (zerop (recursion-depth))
! (unwind-protect
! (semantic-safe "idle error: %S"
! ;; Disable the auto parse timer while re-parsing
! (semantic-idle-scheduler-kill-timer)
! ;; Handle re-parsing and other scheduled services
! (save-match-data (semantic-idle-core-handler)))
;; Enable again the auto parse timer
(semantic-idle-scheduler-setup-timer))))
=0C
;;; REPARSING
;;
Index: semantic/semantic-lex.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/semantic-lex.el,v
retrieving revision 1.35
diff -c -r1.35 semantic-lex.el
*** semantic/semantic-lex.el=096 Apr 2004 12:12:19 -0000=091.35
--- semantic/semantic-lex.el=096 Sep 2004 12:11:15 -0000
***************
*** 655,675 ****
;;;###autoload
(defun semantic-lex-init ()
"Initialize any lexical state for this buffer."
! (when (not semantic-lex-comment-regex)
(setq semantic-lex-comment-regex
=09 (if comment-start-skip
=09 (concat "\\(\\s<\\|" comment-start-skip "\\)")
! =09 (concat "\\(\\s<\\)"))
! =09 ))
;; Setup the lexer syntax-table
(setq semantic-lex-syntax-table (copy-syntax-table (syntax-table)))
! (let* ((mods semantic-lex-syntax-modifications)
! =09 mod)
! =09 (while mods
! =09 (setq mod (car mods)
! =09=09 mods (cdr mods))
! =09 (modify-syntax-entry
! =09 (car mod) (nth 1 mod) semantic-lex-syntax-table))))
=20
;;;###autoload
(define-overload semantic-lex (start end &optional depth length)
--- 655,670 ----
;;;###autoload
(defun semantic-lex-init ()
"Initialize any lexical state for this buffer."
! (unless semantic-lex-comment-regex
(setq semantic-lex-comment-regex
=09 (if comment-start-skip
=09 (concat "\\(\\s<\\|" comment-start-skip "\\)")
! =09 "\\(\\s<\\)")))
;; Setup the lexer syntax-table
(setq semantic-lex-syntax-table (copy-syntax-table (syntax-table)))
! (dolist (mod semantic-lex-syntax-modifications)
! (modify-syntax-entry
! (car mod) (nth 1 mod) semantic-lex-syntax-table)))
=20
;;;###autoload
(define-overload semantic-lex (start end &optional depth length)
***************
*** 718,729 ****
If FORMS throws an error, treat this as a syntax problem, and
execute the unterminated syntax code. FORMS should return a position.
Irreguardless of an error, the cursor should be moved to the end of
! the desired syntax, and a position returned."
! `(condition-case nil
(progn ,@forms)
! (error
! (semantic-lex-unterminated-syntax-detected ,syntax)))
! )
(put 'semantic-lex-unterminated-syntax-protection
'lisp-indent-function 1)
=20
--- 713,728 ----
If FORMS throws an error, treat this as a syntax problem, and
execute the unterminated syntax code. FORMS should return a position.
Irreguardless of an error, the cursor should be moved to the end of
! the desired syntax, and a position returned.
! If `debug-on-error' is set, errors are not caught, so that you can
! debug them.
! Avoid using a large FORMS since it is duplicated."
! `(if debug-on-error
(progn ,@forms)
! (condition-case nil
! (progn ,@forms)
! (error
! (semantic-lex-unterminated-syntax-detected ,syntax)))))
(put 'semantic-lex-unterminated-syntax-protection
'lisp-indent-function 1)
=20
***************
*** 1329,1353 ****
out of date.
If there is an error, the syntax that failed is returned.
If there is no error, then the last value of FORMS is returned."
! `(let ((semantic-lex-unterminated-syntax-end-function
! =09 (lambda (syntax start end) (throw ',symbol syntax)))
! =09 ;; Delete the below when semantic-flex is fully retired.
! =09 (semantic-flex-unterminated-syntax-end-function
! =09 (lambda (syntax start end) (throw ',symbol syntax)))
! =09 ret)
! (setq ret (catch ',symbol
! =09=09 (save-excursion
! =09=09 ,@forms
! =09=09 nil)))
! ;; Great Sadness. Assume that FORMS execute within the
! ;; confines of the current buffer only! Mark this thing
! ;; unparseable iff the special symbol was thrown. This
! ;; will prevent future calls from parsing, but will allow
! ;; then to still return the cache.
! (when ret
! (message "Buffer not currently parsable.")
! (semantic-parse-tree-unparseable))
! ret))
(put 'semantic-lex-catch-errors 'lisp-indent-function 1)
=20
=0C
--- 1328,1356 ----
out of date.
If there is an error, the syntax that failed is returned.
If there is no error, then the last value of FORMS is returned."
! (let ((ret (make-symbol "ret"))
! (syntax (make-symbol "syntax"))
! (start (make-symbol "start"))
! (end (make-symbol "end")))
! `(let* ((semantic-lex-unterminated-syntax-end-function
! (lambda (,syntax ,start ,end)
! (throw ',symbol ,syntax)))
! ;; Delete the below when semantic-flex is fully retired.
! (semantic-flex-unterminated-syntax-end-function
! semantic-lex-unterminated-syntax-end-function)
! (,ret (catch ',symbol
! (save-excursion
! ,@forms
! nil))))
! ;; Great Sadness. Assume that FORMS execute within the
! ;; confines of the current buffer only! Mark this thing
! ;; unparseable iff the special symbol was thrown. This
! ;; will prevent future calls from parsing, but will allow
! ;; then to still return the cache.
! (when ,ret
! (message "Buffer not currently parsable (%S)." ,ret)
! (semantic-parse-tree-unparseable))
! ,ret)))
(put 'semantic-lex-catch-errors 'lisp-indent-function 1)
=20
=0C
Index: semantic/semantic.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/semantic.el,v
retrieving revision 1.188
diff -c -r1.188 semantic.el
*** semantic/semantic.el=0920 Jul 2004 17:57:19 -0000=091.188
--- semantic/semantic.el=096 Sep 2004 12:11:15 -0000
***************
*** 292,306 ****
(not (semantic-active-p))
(not (run-hook-with-args-until-success
'semantic-inhibit-functions)))
! ;; Force this buffer to have its cache refreshed.
! (semantic-clear-toplevel-cache)
;; Here are some buffer local variables we can initialize ourselves
;; of a mode does not choose to do so.
(semantic-lex-init)
! ;; Setup for a needed reparse.
! (semantic-parse-tree-set-needs-rebuild)
! ;; Specify that this function has done it's work.
! (setq semantic-new-buffer-fcn-was-run t)
;; Call DB hooks before regular init hooks
(run-hooks 'semantic-init-db-hooks)
;; Lastly, set up semantic modes
--- 292,305 ----
(not (semantic-active-p))
(not (run-hook-with-args-until-success
'semantic-inhibit-functions)))
! ;; Specify that this function has done it's work. At this point
! ;; we can consider that semantic is active in this buffer.
! (setq semantic-new-buffer-fcn-was-run t)
;; Here are some buffer local variables we can initialize ourselves
;; of a mode does not choose to do so.
(semantic-lex-init)
! ;; Force this buffer to have its cache refreshed.
! (semantic-clear-toplevel-cache)
;; Call DB hooks before regular init hooks
(run-hooks 'semantic-init-db-hooks)
;; Lastly, set up semantic modes
***************
*** 410,430 ****
DEPTH specifies the lexical depth to scan.
RETURNONERROR specifies that parsing should end when encountering
unterminated syntax."
! (if (or (eq semantic--parse-table t)
! =09 (eq semantic--parse-table nil))
! ;; If there is no table, or it was set to t, then we are here
! ;; by some other mistake. Do not throw an error deep in the parser=
.
! (error "`semantic-oarse-region-default' called in a buffer that doe=
s not support bovine parsing.")
! (if (or (< end start) (> end (point-max)))
! =09(error "Invalid bounds passed to `semantic-parse-region'"))
! (let ((lexbits (semantic-lex start end depth))
! =09 tags)
! ;; Init a dump
! ;; (if semantic-dump-parse
! ;;=09 (semantic-dump-buffer-init))
! (setq tags (semantic-repeat-parse-whole-stream
! =09=09 lexbits nonterminal returnonerror))
! (nreverse tags))))
=0C
;;; Parsing functions
;;
--- 409,423 ----
DEPTH specifies the lexical depth to scan.
RETURNONERROR specifies that parsing should end when encountering
unterminated syntax."
! (when (or (null semantic--parse-table) (eq semantic--parse-table t))
! ;; If there is no table, or it was set to t, then we are here by
! ;; some other mistake. Do not throw an error deep in the parser.
! (error "No support found to parse buffer %S." (buffer-name)))
! (when (or (< end start) (> end (point-max)))
! (error "Invalid parse region bounds %S, %S." start end))
! (nreverse
! (semantic-repeat-parse-whole-stream
! (semantic-lex start end depth) nonterminal returnonerror)))
=0C
;;; Parsing functions
;;
Index: semantic/wisent/wisent-python.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-python.el,v
retrieving revision 1.45
diff -c -r1.45 wisent-python.el
*** semantic/wisent/wisent-python.el=094 Aug 2004 02:34:23 -0000=091.45
--- semantic/wisent/wisent-python.el=096 Sep 2004 12:11:15 -0000
***************
*** 1,4 ****
! ;;; wisent-python.el --- LALR grammar for Python
;;
;; Copyright (C) 2002, 2004 Richard Kim
;;
--- 1,4 ----
! ;;; wisent-python.el --- Semantic support for Python
;;
;; Copyright (C) 2002, 2004 Richard Kim
;;
***************
*** 27,59 ****
=20
;;; Commentary:
;;
! ;; This file contains the python parser created from the grammar
! ;; specified in wisent-python.wy file. It also has some support code.
;;
;;; Code:
=20
(require 'semantic-wisent)
(require 'wisent-python-wy)
!=20
! ;;;**********************************************************************=
******
! ;;;@ Support Code
! ;;;**********************************************************************=
******
! ;;
! ;; Some of these need to come before `wisent-python-default-setup' so tha=
t
! ;; symbols are defined before their first use.
!=20
! ;; Indentation stack to keep track of INDENT tokens generated without
! ;; matching DEDENT tokens. Generation of each INDENT token results in
! ;; a new integer being added to the beginning of this list where the
! ;; integer represents the indentation of the current line. Each time a
! ;; DEDENT token is generated, the latest entry is popped off
! ;; this list.
! (defvar wisent-python-lexer-indent-stack '(0))
!=20
! ;; Variable set to t only by `semantic-lex-python-charquote' so that
! ;; `semantic-lex-python-beginning-of-line' will not generate any
! ;; INDENT or DEDENT tokens for continued lines.
! (defvar wisent-python-explicit-line-continuation nil)
=20
;; Python strings are delimited by either single quotes or double
;; quotes, e.g., "I'm a string" and 'I too am s string'.
--- 27,49 ----
=20
;;; Commentary:
;;
! ;; This library contains Semantic support code for the Python
! ;; programming language. The LALR grammar used to parse Python
! ;; sources is in the wisent-python.wy file.
! ;;
! ;; The official website for the Python language is at
! ;; <http://python.org/>.
! ;;
! ;; An X/Emacs major mode for editing Python source code is available
! ;; at <http://sourceforge.net/projects/python-mode/>.
;;
;;; Code:
=20
(require 'semantic-wisent)
(require 'wisent-python-wy)
! =0C
! ;;; Lexical analysis
! ;;
=20
;; Python strings are delimited by either single quotes or double
;; quotes, e.g., "I'm a string" and 'I too am s string'.
***************
*** 62,352 ****
;; to be suppressed. For example, r"01\n34" is a string with six
;; characters 0, 1, \, n, 3 and 4. The 'u' prefix means the following
;; string is a unicode.
! (defconst wisent-python-string-re "[rR]?[uU]?['\"]"
"Regexp matching beginning of a python string.")
=20
! ;;;**********************************************************************=
******
! ;;;@ Lexer
! ;;;**********************************************************************=
******
!=20
! ;; Pop all items from the "indent stack" if we are at buffer end.
! (defun semantic-lex-python-pop-indent-stack ()
! (if (eq (point) (cdr semantic-lex-analysis-bounds))
! (while (> (car wisent-python-lexer-indent-stack) 0)
! (semantic-lex-push-token
! (semantic-lex-token 'DEDENT (point) (point)))
! (pop wisent-python-lexer-indent-stack))))
!=20
! (define-lex-analyzer semantic-lex-python-beginning-of-line
! "Handle beginning-of-line case, i.e., possibly generate INDENT or
! DEDENT tokens by comparing current indentation level with the previous
! indentation values stored in `wisent-python-lexer-indent-stack'
! stack."
! (and (and (bolp) (not wisent-python-explicit-line-continuation))
! (let ((last-indent (or (car wisent-python-lexer-indent-stack) 0))
! (last-pos (point))
! curr-indent)
! (skip-chars-forward " \t")
! (setq curr-indent (current-column))
! (cond
! ;; Blank or comment line =3D> no indentation change
! ((looking-at "\\(#\\|$\\)")
! (forward-line 1)
! (setq semantic-lex-end-point (point))
! (semantic-lex-python-pop-indent-stack)
! ;; Since position changed, returning t here won't result in
! ;; infinite loop.
! t)
! ;; No change in indentation.
! ((=3D curr-indent last-indent)
! (setq semantic-lex-end-point (point))
! ;; If pos did not change, then we must return nil so that
! ;; other lexical analyzers can be run.
! nil)
! ;; Indentation increased
! ((> curr-indent last-indent)
! (if (or (not semantic-lex-maximum-depth)
! (< semantic-lex-current-depth semantic-lex-maximum-dep=
th))
! (progn
! ;; Return an INDENT lexical token
! (setq semantic-lex-current-depth (1+ semantic-lex-curren=
t-depth))
! (push curr-indent wisent-python-lexer-indent-stack)
! (semantic-lex-push-token
! (semantic-lex-token 'INDENT last-pos (point)))
! t)
! ;; Add an INDENT_BLOCK token
! (semantic-lex-push-token
! (semantic-lex-token
! 'INDENT_BLOCK
! (progn (beginning-of-line) (point))
! (save-excursion
! (semantic-lex-unterminated-syntax-protection
! 'INDENT_BLOCK
! (let ((starting-indentation (current-indentation)))
! (while (>=3D (current-indentation) starting-indentati=
on)
! (forward-list 1)
! (beginning-of-line)))
! (point)))))
! t)
! )
! ;; Indentation decreased
! (t
! ;; Pop items from indentation stack
! (while (< curr-indent last-indent)
! (setq semantic-lex-current-depth (1- semantic-lex-current-de=
pth))
! (semantic-lex-push-token
! (semantic-lex-token 'DEDENT last-pos (point)))
! (pop wisent-python-lexer-indent-stack)
! (setq last-indent (or (car wisent-python-lexer-indent-stack)=
0)))
! ;; If pos did not change, then we must return nil so that
! ;; other lexical analyzers can be run.
! (not (eq last-pos (point))))
! )))
! nil ;; all the work was done in the previous form
! )
=20
! (define-lex-analyzer semantic-lex-python-reset-continued-line
! "Reset `wisent-python-explicit-line-continuation' back to nil."
! (setq wisent-python-explicit-line-continuation nil)
! ()
)
=20
! (define-lex-analyzer semantic-lex-python-newline
! "Handle NEWLINE syntactic tokens.
! If the following line is an implicit continuation of current line,
! then throw away any immediately following INDENT and DEDENT tokens."
! (looking-at "\\(\n\\|\\s>\\)") ;; newline or end of buffer
! (goto-char (match-end 0))
! (semantic-lex-push-token
! (semantic-lex-token 'NEWLINE (1- (point)) (point)))
! (semantic-lex-python-pop-indent-stack))
!=20
! (define-lex-analyzer semantic-lex-python-string
! "Handle python strings."
! (looking-at wisent-python-string-re)
! (let ((opos (point))
! (e (semantic-lex-unterminated-syntax-protection
! 'STRING_LITERAL
! ;; skip over "r" and/or "u" characters if any
! (goto-char (1- (match-end 0)))
! (cond
! ((looking-at "\"\"\"")
! (forward-char 3)
! (search-forward "\"\"\""))
! (t
! (forward-sexp 1)))
! (point))))
(semantic-lex-push-token
! (semantic-lex-token 'STRING_LITERAL opos e))))
!=20
! (define-lex-analyzer semantic-lex-python-charquote
! "Handle BACKSLASH syntactic tokens."
! (looking-at "\\s\\")
! (forward-char 1)
! (when (looking-at "$")
! (forward-char 1)
! (skip-chars-forward " \t")
! (setq wisent-python-explicit-line-continuation nil))
! (setq semantic-lex-end-point (point)))
=20
! ;; This is same as wisent-java-lex-symbol except for using 'NAME token
! ;; rather than 'IDENTIFIER. -ryk1/05/03.
! (define-lex-regex-analyzer semantic-lex-python-symbol
! "Detect and create identifier or keyword tokens."
! "\\(\\sw\\|\\s_\\)+"
(semantic-lex-push-token
(semantic-lex-token
! (or (semantic-lex-keyword-p (match-string 0))
! 'NAME)
! (match-beginning 0)
! (match-end 0))))
!=20
! ;; Same as wisent-java-lex-number. -ryk1/05/03.
! (define-lex-simple-regex-analyzer semantic-lex-python-number
! "Detect and create number tokens."
! semantic-lex-number-expression 'NUMBER_LITERAL)
!=20
! ;; Same as wisent-java-lex-blocks. -ryk1/05/03.
! (define-lex-block-analyzer semantic-lex-python-blocks
! "Detect and create a open, close or block token."
! (PAREN_BLOCK ("(" LPAREN) (")" RPAREN))
! (BRACE_BLOCK ("{" LBRACE) ("}" RBRACE))
! (BRACK_BLOCK ("[" LBRACK) ("]" RBRACK)))
=20
! (define-lex semantic-python-lexer
"Lexical Analyzer for Python code."
! ;; semantic-lex-python-beginning-of-line needs to be the first so
! ;; that we don't miss any DEDENT tokens at the beginning of lines.
! semantic-lex-python-beginning-of-line
! semantic-lex-python-reset-continued-line
! ;; semantic-lex-python-string needs to come before symbols because
! ;; of the "r" and/or "u" prefix.
! semantic-lex-python-string
semantic-lex-ignore-whitespace
- semantic-lex-python-newline
- semantic-lex-python-number ;; rather than semantic-lex-number
- semantic-lex-python-symbol ;; rather than semantic-lex-symbol-or-key=
word
- semantic-lex-python-charquote
- semantic-lex-python-blocks ;; rather than semantic-lex-paren-or-list=
/semantic-lex-close-paren
semantic-lex-ignore-comments
! semantic-lex-punctuation-type ;; rather than semantic-lex-punctuation
! semantic-lex-default-action
! )
!=20
! (defun python-next-line ()
! "Move the cursor to the next logical line to check for INDENT or DEDENT=
tokens.
! Usually this is simply the next physical line unless strings, lists, expl=
icit
! line continuation, blank lines, or comment lines are encountered.
! This function skips over such items so that the cursor is at the beginnin=
g of
! the next logical line."
! (let (beg)
! (while (not (eolp))
! (setq beg (point))
! (cond
! ;; skip over triple-quote string
! ((looking-at "\"\"\"")
! (forward-char 3)
! ;; TODO: this probably should be protected with
! ;; semantic-lex-unterminated-syntax-protection
! ;; in case the closing triple quote is not found. -ryk2/17/03.
! (search-forward "\"\"\""))
! ;; skip over lists, strings, etc
! ((looking-at "\\(\\s(\\|\\s\"\\|\\s<\\)")
! (forward-sexp 1))
! ;; backslash is the explicit line continuation character
! ((looking-at "\\s\\")
! (forward-line 1))
! ;; skip over white space, word, symbol, punctuation, and paired
! ;; delimiter (backquote) characters.
! (t (skip-syntax-forward "-w_.$")))
! (if (=3D (point) beg)
! (error "You have found a bug in python-next-line")))
! ;; the point now should be at the end of a line
! (forward-line 1)
! (while (looking-at "\\s-*\\(\\s<\\|$\\)") ;; skip blank and comment =
lines
! (forward-line 1)
! (if (eobp) (error "Unterminated List."))
! )))
!=20
! (defun python-scan-lists ( &optional target-column )
! "Without actually changing the position, return the buffer position of
! the next line whose indentation is the same as the current line or less
! than current line."
! (or target-column (setq target-column (current-indentation)))
! (save-excursion
! (python-next-line)
! (while (> (current-indentation) target-column)
! (python-next-line))
! ;; Move the cursor to the original indentation level or first non-whi=
te
! ;; character which ever comes first.
! (skip-chars-forward " \t" (+ (point) target-column))
! (point)))
!=20
! (defadvice scan-lists (around handle-python-mode activate compile)
! "Use python mode specific function, python-scan-lists, if the
! current major mode is python-mode.
! Otherwise simply call the original function."
! (if (and (eq major-mode 'python-mode)
! (not (looking-at "\\s(")))
! (setq ad-return-value (python-scan-lists))
! ad-do-it))
!=20
! ;;;**********************************************************************=
******
! ;;;@ Parser
! ;;;**********************************************************************=
******
!=20
! ;; TODO: Is this really needed? -ryk2/9/03.
! (define-mode-local-override semantic-parse-region python-mode
! (start end &optional nonterminal depth returnonerror)
! "Over-ride in order to initialize some variables."
! (let ((wisent-python-lexer-indent-stack '(0))
! (wisent-python-explicit-line-continuation nil))
! (semantic-parse-region-default
! start end nonterminal depth returnonerror)))
!=20
! ;; Commented this out after learning that there is no need to convert
! ;; tokens to names. See "(semantic)Style Guide". -ryk2/7/03.
! '(define-mode-local-override semantic-parse-region python-mode
! (start end &optional nonterminal depth returnonerror)
! "Over-ride so that 'paren_classes' non-terminal tokens can be intercept=
ed
! then converted to simple names to comply with the semantic token style gu=
ide."
! (let ((tokens (semantic-parse-region-default
! start end nonterminal depth returnonerror)))
! (if (eq nonterminal 'paren_classes)
! (mapcar #'semantic-token-name tokens)
! tokens)))
=20
(define-mode-local-override semantic-get-local-variables python-mode ()
"Get the local variables based on point's context.
To be implemented for python! For now just return nil."
nil)
=20
;;;###autoload
(defun wisent-python-default-setup ()
"Setup buffer for parse."
(wisent-python-wy--install-parser)
(setq
;; Character used to separation a parent/child relationship
semantic-type-relation-separator-character '(".")
semantic-command-separation-character ";"
! wisent-python-lexer-indent-stack '(0)
! semantic-lex-analyzer #'semantic-python-lexer
))
=20
;;;###autoload
! (add-hook 'python-mode-hook #'wisent-python-default-setup)
!=20
;;; Test
;;
(defun wisent-python-lex-buffer ()
! "Run `semantic-python-lexer' on current buffer."
(interactive)
(semantic-lex-init)
! (setq semantic-lex-analyzer 'semantic-python-lexer)
! (let ((token-stream
! (semantic-lex (point-min) (point-max) 0)))
! (with-current-buffer
! (get-buffer-create "*semantic-python-lexer*")
(erase-buffer)
(pp token-stream (current-buffer))
(goto-char (point-min))
--- 52,307 ----
;; to be suppressed. For example, r"01\n34" is a string with six
;; characters 0, 1, \, n, 3 and 4. The 'u' prefix means the following
;; string is a unicode.
! (defconst wisent-python-string-re
! (concat (regexp-opt '("r" "u" "ur" "R" "U" "UR" "Ur" "uR") t)
! "?['\"]")
"Regexp matching beginning of a python string.")
=20
! (defsubst wisent-python-implicit-line-joining-p ()
! "Return non-nil if implicit line joining is active.
! That is, if inside an expressions in parentheses, square brackets or
! curly braces."
! (condition-case nil
! (progn (scan-lists (point) -1 1) t)
! (error nil)))
!=20
! (defsubst wisent-python-forward-string ()
! "Move point at the end of the python string at point."
! (when (looking-at wisent-python-string-re)
! ;; skip the prefix
! (and (match-end 1) (goto-char (match-end 1)))
! ;; skip the quoted part
! (cond
! ((looking-at "\"\"\"[^\"]")
! (search-forward "\"\"\"" nil nil 2))
! ((looking-at "'''[^']")
! (search-forward "'''" nil nil 2))
! ((forward-sexp 1)))))
!=20
! (defun wisent-python-forward-line ()
! "Move point to the beginning of the next logical line.
! Usually this is simply the next physical line unless strings,
! implicit/explicit line continuation, blank lines, or comment lines are
! encountered. This function skips over such items so that the point is
! at the beginning of the next logical line. If the current logical
! line ends at the end of the buffer, leave the point there."
! (while (not (eolp))
! (when (=3D (point)
! (progn
! (cond
! ;; skip over python strings
! ((looking-at wisent-python-string-re)
! (wisent-python-forward-string))
! ;; skip over lists, strings, etc
! ((looking-at "\\(\\s(\\|\\s\"\\|\\s<\\)")
! (forward-sexp 1))
! ;; At the explicit line continuation character
! ;; (backslash) move to next line.
! ((looking-at "\\s\\")
! (forward-line 1))
! ;; skip over white space, word, symbol, punctuation,
! ;; and paired delimiter (backquote) characters.
! ((skip-syntax-forward "-w_.$)")))
! (point)))
! (error "python-forward-line endless loop detected")))
! ;; The point is at eol, skip blank and comment lines.
! (forward-comment (point-max))
! ;; Goto the beginning of the next line.
! (or (eobp) (beginning-of-line)))
!=20
! (defun wisent-python-forward-line-skip-indented ()
! "Move point to the next logical line, skipping indented lines.
! That is the next line whose indentation is less than or equal to the
! identation of the current line."
! (let ((indent (current-indentation)))
! (while (progn (wisent-python-forward-line)
! (and (not (eobp))
! (> (current-indentation) indent))))))
!=20
! (defun wisent-python-end-of-block ()
! "Move point to the end of the current block"
! (let ((indent (current-indentation)))
! (while (and (not (eobp)) (>=3D (current-indentation) indent))
! (wisent-python-forward-line-skip-indented))))
=20
! ;; Indentation stack, what the Python (2.3) language spec. says:
! ;;
! ;; The indentation levels of consecutive lines are used to generate
! ;; INDENT and DEDENT tokens, using a stack, as follows.
! ;;
! ;; Before the first line of the file is read, a single zero is pushed
! ;; on the stack; this will never be popped off again. The numbers
! ;; pushed on the stack will always be strictly increasing from bottom
! ;; to top. At the beginning of each logical line, the line's
! ;; indentation level is compared to the top of the stack. If it is
! ;; equal, nothing happens. If it is larger, it is pushed on the stack,
! ;; and one INDENT token is generated. If it is smaller, it must be one
! ;; of the numbers occurring on the stack; all numbers on the stack
! ;; that are larger are popped off, and for each number popped off a
! ;; DEDENT token is generated. At the end of the file, a DEDENT token
! ;; is generated for each number remaining on the stack that is larger
! ;; than zero.
! (defvar wisent-python-indent-stack)
!=20
! (define-lex-analyzer wisent-python-lex-beginning-of-line
! "Detect and create python indentation tokens at beginning of line."
! (and
! (bolp) (not (wisent-python-implicit-line-joining-p))
! (let ((last-indent (car wisent-python-indent-stack))
! (last-pos (point))
! (curr-indent (current-indentation)))
! (skip-syntax-forward "-")
! (cond
! ;; Skip comments and blank lines. No change in indentation.
! ((or (eolp) (looking-at semantic-lex-comment-regex))
! (forward-comment (point-max))
! (or (eobp) (beginning-of-line))
! (setq semantic-lex-end-point (point))
! ;; Loop lexer to handle the next line.
! t)
! ;; No change in indentation.
! ((=3D curr-indent last-indent)
! (setq semantic-lex-end-point (point))
! ;; Try next analyzers.
! nil)
! ;; Indentation increased
! ((> curr-indent last-indent)
! (if (or (not semantic-lex-maximum-depth)
! (< semantic-lex-current-depth semantic-lex-maximum-depth))
! (progn
! ;; Return an INDENT lexical token
! (setq semantic-lex-current-depth (1+ semantic-lex-current-de=
pth))
! (push curr-indent wisent-python-indent-stack)
! (semantic-lex-push-token
! (semantic-lex-token 'INDENT last-pos (point))))
! ;; Add an INDENT_BLOCK token
! (semantic-lex-push-token
! (semantic-lex-token
! 'INDENT_BLOCK
! (progn (beginning-of-line) (point))
! (semantic-lex-unterminated-syntax-protection 'INDENT_BLOCK
! (wisent-python-end-of-block)
! (point)))))
! ;; Loop lexer to handle tokens in current line.
! t)
! ;; Indentation decreased
! (t
! ;; Pop items from indentation stack
! (while (< curr-indent last-indent)
! (setq semantic-lex-current-depth (1- semantic-lex-current-depth)=
)
! (semantic-lex-push-token
! (semantic-lex-token 'DEDENT last-pos (point)))
! (setq last-indent (pop wisent-python-indent-stack)))
! ;; If pos did not change, then we must return nil so that
! ;; other lexical analyzers can be run.
! (/=3D last-pos (point)))
! )))
! ;; All the work was done in the above analyzer matching condition.
)
=20
! (define-lex-regex-analyzer wisent-python-lex-end-of-line
! "Detect and create python newline tokens.
! Just skip the newline character if the following line is an implicit
! continuation of current line."
! "\\(\n\\|\\s>\\)"
! (if (wisent-python-implicit-line-joining-p)
! (setq semantic-lex-end-point (match-end 0))
(semantic-lex-push-token
! (semantic-lex-token 'NEWLINE (point) (match-end 0)))))
=20
! (define-lex-regex-analyzer wisent-python-lex-string
! "Detect and create python string tokens."
! wisent-python-string-re
(semantic-lex-push-token
(semantic-lex-token
! 'STRING_LITERAL
! (point)
! (semantic-lex-unterminated-syntax-protection 'STRING_LITERAL
! (wisent-python-forward-string)
! (point)))))
!=20
! (define-lex-regex-analyzer wisent-python-lex-ignore-backslash
! "Detect and skip over backslash (explicit line joining) tokens.
! A backslash must be the last token of a physical line, it is illegal
! elsewhere on a line outside a string literal."
! "\\s\\\\s-*$"
! ;; Skip over the detected backslash and go to the first
! ;; non-whitespace character in the next physical line.
! (forward-line)
! (skip-syntax-forward "-")
! (setq semantic-lex-end-point (point)))
=20
! (define-lex wisent-python-lexer
"Lexical Analyzer for Python code."
! ;; Must analyze beginning of line first to handle indentation.
! wisent-python-lex-beginning-of-line
! wisent-python-lex-end-of-line
! ;; Must analyze string before symbol to handle string prefix.
! wisent-python-lex-string
! ;; Analyzers auto-generated from grammar.
! wisent-python-wy--<number>-regexp-analyzer
! wisent-python-wy--<keyword>-keyword-analyzer
! wisent-python-wy--<symbol>-regexp-analyzer
! wisent-python-wy--<block>-block-analyzer
! wisent-python-wy--<punctuation>-string-analyzer
! ;; Ignored things.
! wisent-python-lex-ignore-backslash
semantic-lex-ignore-whitespace
semantic-lex-ignore-comments
! ;; Signal error on unhandled syntax.
! semantic-lex-default-action)
! =0C
! ;;; Overridden Semantic API.
! ;;
! (define-mode-local-override semantic-lex python-mode
! (start end &optional depth length)
! "Lexically analyze python code in current buffer.
! See the function `semantic-lex' for the meaning of the START, END,
! DEPTH and LENGTH arguments.
! This function calls `wisent-python-lexer' to actually perform the
! lexical analysis, then emits the necessary python DEDENT tokens from
! what remains in the `wisent-python-indent-stack'."
! (let* ((wisent-python-indent-stack (list 0))
! (stream (wisent-python-lexer start end depth length))
! (semantic-lex-token-stream nil))
! ;; Emit DEDENT tokens if something remains in the INDENT stack.
! (while (> (pop wisent-python-indent-stack) 0)
! (semantic-lex-push-token (semantic-lex-token 'DEDENT end end)))
! (nconc stream (nreverse semantic-lex-token-stream))))
=20
(define-mode-local-override semantic-get-local-variables python-mode ()
"Get the local variables based on point's context.
To be implemented for python! For now just return nil."
nil)
=20
+ ;;; Enable Semantic in `python-mode'.
+ ;;
+=20
;;;###autoload
(defun wisent-python-default-setup ()
"Setup buffer for parse."
(wisent-python-wy--install-parser)
+ (set (make-local-variable 'parse-sexp-ignore-comments) t)
(setq
;; Character used to separation a parent/child relationship
semantic-type-relation-separator-character '(".")
semantic-command-separation-character ";"
! ;; The following is no more necessary as semantic-lex is overriden
! ;; in python-mode.
! ;; semantic-lex-analyzer 'wisent-python-lexer
))
=20
;;;###autoload
! (add-hook 'python-mode-hook 'wisent-python-default-setup)
! =0C
;;; Test
;;
(defun wisent-python-lex-buffer ()
! "Run `wisent-python-lexer' on current buffer."
(interactive)
(semantic-lex-init)
! (let ((token-stream (semantic-lex (point-min) (point-max) 0)))
! (with-current-buffer (get-buffer-create "*wisent-python-lexer*")
(erase-buffer)
(pp token-stream (current-buffer))
(goto-char (point-min))
Index: semantic/wisent/wisent-python.wy
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-python.wy,v
retrieving revision 1.32
diff -c -r1.32 wisent-python.wy
*** semantic/wisent/wisent-python.wy=0926 May 2004 08:50:24 -0000=091.32
--- semantic/wisent/wisent-python.wy=096 Sep 2004 12:11:15 -0000
***************
*** 32,43 ****
;;
;;; To do:
;;
- ;; * Modify the grammar to take advantage of scan-lists advice, i.e.,
- ;; generate INDENT_BLOCK tokens such as PAREN_BLOCK tokens rather than
- ;; generating whole bunch of lexical tokens.
- ;;
- ;; * Verify that \ character is handled properly by the lexer and parser.
- ;;
;; * Verify that semantic-lex-python-number regexp is correct.
=20
;; --------
--- 32,37 ----
***************
*** 60,88 ****
%start paren_classes
%start indented_block_body
=20
! %token <newline> NEWLINE
!=20
! ;; ---------------------
! ;; Parenthesis terminals
! ;; ---------------------
! %token <open-paren> LPAREN =09"("
! %token <close-paren> RPAREN =09")"
! %token <open-paren> LBRACE =09"{"
! %token <close-paren> RBRACE "}"
! %token <open-paren> LBRACK =09"["
! %token <close-paren> RBRACK =09"]"
!=20
! ;; ---------------
! ;; Block terminals
! ;; ---------------
! %token <semantic-list> PAREN_BLOCK "^("
! %token <semantic-list> BRACE_BLOCK "^{"
! %token <semantic-list> BRACK_BLOCK "^\\["
! %token <semantic-list> INDENT_BLOCK "^\\s-+"
=20
;; ------------------
;; Operator terminals
;; ------------------
%token <punctuation> LTLTEQ =09"<<=3D"
%token <punctuation> GTGTEQ =09">>=3D"
%token <punctuation> EXPEQ=09"**=3D"
--- 54,91 ----
%start paren_classes
%start indented_block_body
=20
! ;; -------------------------------
! ;; Misc. Python specific terminals
! ;; -------------------------------
! ;; The value of these tokens are for documentation only, they are not
! ;; used by the lexer.
! %token <charquote> BACKSLASH =09 "\\"
! %token <newline> NEWLINE "\n"
! %token <indentation> INDENT "^\\s-+"
! %token <indentation> DEDENT "[^:INDENT:]"=20
! %token <indentation> INDENT_BLOCK "(INDENT DEDENT)"
!=20
! ;; -----------------------------
! ;; Block & Parenthesis terminals
! ;; -----------------------------
! %type <block> ;;syntax "\\s(\\|\\s)" matchdatatype block
!=20
! %token <block> PAREN_BLOCK "(LPAREN RPAREN)"
! %token <block> BRACE_BLOCK "(LBRACE RBRACE)"
! %token <block> BRACK_BLOCK "(LBRACK RBRACK)"
!=20
! %token <open-paren> LPAREN "("
! %token <close-paren> RPAREN ")"
! %token <open-paren> LBRACE "{"
! %token <close-paren> RBRACE "}"
! %token <open-paren> LBRACK "["
! %token <close-paren> RBRACK "]"
=20
;; ------------------
;; Operator terminals
;; ------------------
+ %type <punctuation> ;;syntax "\\(\\s.\\|\\s$\\|\\s'\\)+" matchdatatype s=
tring
+=20
%token <punctuation> LTLTEQ =09"<<=3D"
%token <punctuation> GTGTEQ =09">>=3D"
%token <punctuation> EXPEQ=09"**=3D"
***************
*** 117,259 ****
%token <punctuation> TILDE =09"~"
%token <punctuation> BAR =09"|"
%token <punctuation> COLON =09":"
! %token <punctuation> SEMICOLON =09";"
%token <punctuation> COMMA =09","
%token <punctuation> ASSIGN =09"=3D"
! %token <punctuation> BACKQUOTE =09"`"
=20
- %token <charquote> BACKSLASH =09"\\"
- %put charquote string t
=20
;; -----------------
;; Literal terminals
;; -----------------
%token <string> STRING_LITERAL
%token <number> NUMBER_LITERAL
=20
%token <symbol> NAME
=20
- %token=09 INDENT
- %token=09 DEDENT
-=20
;; -----------------
;; Keyword terminals
;; -----------------
=20
! %token AND=09 "and"
! %put AND summary
"Logical AND binary operator ... "
=20
! %token ASSERT=09 "assert"
! %put ASSERT summary
"Raise AssertionError exception if <expr> is false"
=20
! %token BREAK=09 "break"
! %put BREAK summary
"Terminate 'for' or 'while loop"
=20
! %token CLASS=09 "class"
! %put CLASS summary
"Define a new class"
=20
! %token CONTINUE=09 "continue"
! %put CONTINUE summary
"Skip to the next interation of enclosing for or whilte loop"
=20
! %token DEF=09 "def"
! %put DEF summary
"Define a new function"
=20
! %token DEL=09 "del"
! %put DEL summary
"Delete specified objects, i.e., undo what assignment did"
=20
! %token ELIF=09 "elif"
! %put ELIF summary
"Shorthand for 'else if' following an 'if' statement"
=20
! %token ELSE=09 "else"
! %put ELSE summary
"Start the 'else' clause following an 'if' statement"
=20
! %token EXCEPT=09 "except"
! %put EXCEPT summary
"Specify exception handlers along with 'try' keyword"
=20
! %token EXEC=09 "exec"
! %put EXEC summary
"Dynamically execute python code"
=20
! %token FINALLY=09 "finally"
! %put FINALLY summary
"Specify code to be executed after 'try' statements whether or not an exc=
eption occured"
=20
! %token FOR=09 "for"
! %put FOR summary
"Start a 'for' loop"
=20
! %token FROM=09 "from"
! %put FROM summary
"Modify behavior of 'import' statement"
=20
! %token GLOBAL=09 "global"
! %put GLOBAL summary
"Declare one or more symbols as global symbols"
=20
! %token IF=09 "if"
! %put IF summary
"Start 'if' conditional statement"
=20
! %token IMPORT=09 "import"
! %put IMPORT summary
"Load specified modules"
=20
! %token IN=09 "in"
! %put IN summary
"Part of 'for' statement "
=20
! %token IS=09 "is"
! %put IS summary
"Binary operator that tests for object equality"
=20
! %token LAMBDA=09 "lambda"
! %put LAMDA summary
"Create anonymous function"
=20
! %token NOT=09 "not"
! %put NOT summary
"Unary boolean negation operator"
=20
! %token OR=09 "or"
! %put OR summary
"Binary logical 'or' operator"
=20
! %token PASS=09 "pass"
! %put PASS summary
"Statement that does nothing"
=20
! %token PRINT=09 "print"
! %put PRINT summary
"Print each argument to standard output"
=20
! %token RAISE=09 "raise"
! %put RAISE summary
"Raise an exception"
=20
! %token RETURN=09 "return"
! %put RETURN summary
"Return from a function"
=20
! %token TRY=09 "try"
! %put TRY summary
"Start of statements protected by exception handlers"
=20
! %token WHILE=09 "while"
! %put WHILE summary
"Start a 'while' loop"
=20
! %token YIELD=09 "yield"
! %put YIELD summary
"Create a generator function"
=20
%%
--- 120,261 ----
%token <punctuation> TILDE =09"~"
%token <punctuation> BAR =09"|"
%token <punctuation> COLON =09":"
! %token <punctuation> SEMICOLON=09";"
%token <punctuation> COMMA =09","
%token <punctuation> ASSIGN =09"=3D"
! %token <punctuation> BACKQUOTE=09"`"
=20
=20
;; -----------------
;; Literal terminals
;; -----------------
%token <string> STRING_LITERAL
+=20
+ %type <number> ;;syntax semantic-lex-number-expression
%token <number> NUMBER_LITERAL
=20
+ %type <symbol> ;;syntax "\\(\\sw\\|\\s_\\)+"
%token <symbol> NAME
=20
;; -----------------
;; Keyword terminals
;; -----------------
+ %type <keyword> ;;syntax "\\(\\sw\\|\\s_\\)+" matchdatatype keyword
=20
! %keyword AND=09 "and"
! %put AND summary
"Logical AND binary operator ... "
=20
! %keyword ASSERT=09 "assert"
! %put ASSERT summary
"Raise AssertionError exception if <expr> is false"
=20
! %keyword BREAK=09 "break"
! %put BREAK summary
"Terminate 'for' or 'while loop"
=20
! %keyword CLASS=09 "class"
! %put CLASS summary
"Define a new class"
=20
! %keyword CONTINUE=09 "continue"
! %put CONTINUE summary
"Skip to the next interation of enclosing for or whilte loop"
=20
! %keyword DEF=09 "def"
! %put DEF summary
"Define a new function"
=20
! %keyword DEL=09 "del"
! %put DEL summary
"Delete specified objects, i.e., undo what assignment did"
=20
! %keyword ELIF=09 "elif"
! %put ELIF summary
"Shorthand for 'else if' following an 'if' statement"
=20
! %keyword ELSE=09 "else"
! %put ELSE summary
"Start the 'else' clause following an 'if' statement"
=20
! %keyword EXCEPT=09 "except"
! %put EXCEPT summary
"Specify exception handlers along with 'try' keyword"
=20
! %keyword EXEC=09 "exec"
! %put EXEC summary
"Dynamically execute python code"
=20
! %keyword FINALLY=09 "finally"
! %put FINALLY summary
"Specify code to be executed after 'try' statements whether or not an exc=
eption occured"
=20
! %keyword FOR=09 "for"
! %put FOR summary
"Start a 'for' loop"
=20
! %keyword FROM=09 "from"
! %put FROM summary
"Modify behavior of 'import' statement"
=20
! %keyword GLOBAL=09 "global"
! %put GLOBAL summary
"Declare one or more symbols as global symbols"
=20
! %keyword IF=09 "if"
! %put IF summary
"Start 'if' conditional statement"
=20
! %keyword IMPORT=09 "import"
! %put IMPORT summary
"Load specified modules"
=20
! %keyword IN=09 "in"
! %put IN summary
"Part of 'for' statement "
=20
! %keyword IS=09 "is"
! %put IS summary
"Binary operator that tests for object equality"
=20
! %keyword LAMBDA=09 "lambda"
! %put LAMDA summary
"Create anonymous function"
=20
! %keyword NOT=09 "not"
! %put NOT summary
"Unary boolean negation operator"
=20
! %keyword OR=09 "or"
! %put OR summary
"Binary logical 'or' operator"
=20
! %keyword PASS=09 "pass"
! %put PASS summary
"Statement that does nothing"
=20
! %keyword PRINT=09 "print"
! %put PRINT summary
"Print each argument to standard output"
=20
! %keyword RAISE=09 "raise"
! %put RAISE summary
"Raise an exception"
=20
! %keyword RETURN=09 "return"
! %put RETURN summary
"Return from a function"
=20
! %keyword TRY=09 "try"
! %put TRY summary
"Start of statements protected by exception handlers"
=20
! %keyword WHILE=09 "while"
! %put WHILE summary
"Start a 'while' loop"
=20
! %keyword YIELD=09 "yield"
! %put YIELD summary
"Create a generator function"
=20
%%
***************
*** 740,746 ****
;; classdef: 'class' NAME ['(' testlist ')'] ':' suite
class_declaration
: CLASS NAME paren_class_list_opt COLON suite
! (TYPE-TAG $2 $1 $5 $3)
;
=20
;; ['(' testlist ')']
--- 742,751 ----
;; classdef: 'class' NAME ['(' testlist ')'] ':' suite
class_declaration
: CLASS NAME paren_class_list_opt COLON suite
! (TYPE-TAG $2 $1 ;; Name "class"
! $5 ;; Members
! (cons $3 nil) ;; (SUPERCLASSES . INTERFACES)
! )
;
=20
;; ['(' testlist ')']
***************
*** 751,757 ****
=20
paren_class_list
: PAREN_BLOCK
! (EXPANDFULL $1 paren_classes)
;
=20
;; parameters: '(' [varargslist] ')'
--- 756,762 ----
=20
paren_class_list
: PAREN_BLOCK
! (mapcar 'semantic-tag-name (EXPANDFULL $1 paren_classes))
;
=20
;; parameters: '(' [varargslist] ')'
***************
*** 773,779 ****
;; parser can parse general expressions, I don't see much benefit in
;; generating a string of expression as base class "name".
paren_class
! : NAME
;
=20
;;;**********************************************************************=
******
--- 778,784 ----
;; parser can parse general expressions, I don't see much benefit in
;; generating a string of expression as base class "name".
paren_class
! : dotted_name
;
=20
;;;**********************************************************************=
******
|