CEDET

>>> David PONCE <david.ponce@...> seems to think that:
>>   | id COLON type_or_nonempty_type_or_type_plus from_typeexpr_opt
>>     (let ((variables (cons $1 my-theoryformal-idops))
>>           (result (VARIABLE-TAG variables $3 nil)))
>>       (message "%S:%S %S" $1 $3 $4)
>>       (message "variables = %S" variables)
>>       (message "result = %S" result)
>>       (setq my-theoryformal-idops nil)
>>       result)
>>   | id COLON typeexpr
>>     (let ((variables (cons $1 my-theoryformal-idops))
>>           (result (VARIABLE-TAG variables $3 nil)))
>>       (message "%S:%S" $1 $3)
>>       (message "variables = %S" variables)
>>       (message "result = %S" result)
>>       (setq my-theoryformal-idops nil)
>>       result)
>>   | opsym COLON typeexpr
>>     (let ((variables (cons $1 my-theoryformal-idops))
>>           (result (VARIABLE-TAG variables $3 nil)))
>>       (message "%S:%S" $1 $3)
>>       (message "variables = %S" variables)
>>       (message "result = %S" result)
>>       (setq my-theoryformal-idops nil)
>>       result)
>
>It looks that the `let' statements above have to be `let*' because
>the value of `variables' is used to compute the value of `result'.
>
>IMO the `let' clauses above fail with a "(void-variable variables)"
>error which is probably masked by a `condition-case' at a upper level
>in semantic code.
  [ ... ]

I have long wondered how to handle situations like this.  I went this
route because it allowed a user to throw complete garbage at a
parser, and the parser would live through it.  This lets you allow
parsing to be pervasive in -hooks, -functions, and utilities that
would suffer if the parser threw an error because some esoteric file
the user doesn't care about got on some list.

On the other hand, it makes development tricky if you don't know what
to look for.

I went for option 1 as I knew the tricks to find problems.

In Emacs, you cannot hook into the debug system yourself.

Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
or a derivation, it could create it in some sort of instrumented
fashion that allows errors to propagate out of the system more easily.

For example, when an action is executed, instrumentation would make
note of it's location, and errors would be caught jumping the
developer to the correct buffer location.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

> I have long wondered how to handle situations like this.  I went this
 > route because it allowed a user to throw complete garbage at a
 > parser, and the parser would live through it.  This lets you allow
 > parsing to be pervasive in -hooks, -functions, and utilities that
 > would suffer if the parser threw an error because some esoteric file
 > the user doesn't care about got on some list.
 >
 > On the other hand, it makes development tricky if you don't know what
 > to look for.
 >
 > I went for option 1 as I knew the tricks to find problems.
 >
 > In Emacs, you cannot hook into the debug system yourself.
 >
 > Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
 > or a derivation, it could create it in some sort of instrumented
 > fashion that allows errors to propagate out of the system more easily.
 >
 > For example, when an action is executed, instrumentation would make
 > note of it's location, and errors would be caught jumping the
 > developer to the correct buffer location.

Perhaps another possibility could be to use this sort of macro
(inspired by the `report-errors' macro included in devel. version of
GNU Emacs 21) at the relevant places, instead of a crude
`condition-case':

(defmacro semantic-safe (format &rest body)
   "Turn into a FORMAT message any error caught during eval of BODY.
Return the value of last BODY form or nil if an error occurred.
FORMAT can have a %s escape which will be replaced with the actual
error message.
If `debug-on-error' is set, errors are not caught, so that you can
debug them.
Avoid using a large BODY since it is duplicated."
   ;;(declare (debug t) (indent 1))
   `(if debug-on-error
        (progn @,body)
      (condition-case err
	 (progn @,body)
        (error
         (message ,format (error-message-string err))
         nil))))
(put 'semantic-safe 'lisp-indent-function 1)

Without `debug-on-error' set, you will at least get an error
message.  And when `debug-on-error' is set the debugger will be
entered on error.

For example in `wisent-parse-stream' (at line 220 in wisent-bovine.el)
we could replace:

     (setq wisent-lex-istream stream
           cache (condition-case nil
                     (wisent-parse semantic-toplevel-bovine-table
                                   wisent-lexer-function
                                   wisent-error-function
                                   goal)
                   (error nil)))

with:

     (setq wisent-lex-istream stream
           cache (semantic-safe "wisent-parse-stream: %s"
                     (wisent-parse semantic-toplevel-bovine-table
                                   wisent-lexer-function
                                   wisent-error-function
                                   goal)))

For the parser, the global performance cost should be negligible (one
extra (if debug-on-error ...) test).  But for the developer the
benefit could be important ;-)

`semantic-safe' could be put in semantic-fw.el.  Or perhaps it should
go into CEDET/common as `cedet-safe'?

What do you think?

David

>>> David Ponce <david.ponce@...> seems to think that:
> > I have long wondered how to handle situations like this.  I went this
> > route because it allowed a user to throw complete garbage at a
> > parser, and the parser would live through it.  This lets you allow
> > parsing to be pervasive in -hooks, -functions, and utilities that
> > would suffer if the parser threw an error because some esoteric file
> > the user doesn't care about got on some list.
> >
> > On the other hand, it makes development tricky if you don't know what
> > to look for.
> >
> > I went for option 1 as I knew the tricks to find problems.
> >
> > In Emacs, you cannot hook into the debug system yourself.
> >
> > Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
> > or a derivation, it could create it in some sort of instrumented
> > fashion that allows errors to propagate out of the system more easily.
> >
> > For example, when an action is executed, instrumentation would make
> > note of it's location, and errors would be caught jumping the
> > developer to the correct buffer location.
>
>Perhaps another possibility could be to use this sort of macro
>(inspired by the `report-errors' macro included in devel. version of
>GNU Emacs 21) at the relevant places, instead of a crude
>`condition-case':
>
>(defmacro semantic-safe (format &rest body)
>   "Turn into a FORMAT message any error caught during eval of BODY.
>Return the value of last BODY form or nil if an error occurred.
>FORMAT can have a %s escape which will be replaced with the actual
>error message.
>If `debug-on-error' is set, errors are not caught, so that you can
>debug them.
>Avoid using a large BODY since it is duplicated."
>   ;;(declare (debug t) (indent 1))
>   `(if debug-on-error
>        (progn @,body)
>      (condition-case err
>	 (progn @,body)
>        (error
>         (message ,format (error-message-string err))
>         nil))))
>(put 'semantic-safe 'lisp-indent-function 1)
>
>Without `debug-on-error' set, you will at least get an error
>message.  And when `debug-on-error' is set the debugger will be
>entered on error.
>
>For example in `wisent-parse-stream' (at line 220 in wisent-bovine.el)
>we could replace:
>
>     (setq wisent-lex-istream stream
>           cache (condition-case nil
>                     (wisent-parse semantic-toplevel-bovine-table
>                                   wisent-lexer-function
>                                   wisent-error-function
>                                   goal)
>                   (error nil)))
>
>with:
>
>     (setq wisent-lex-istream stream
>           cache (semantic-safe "wisent-parse-stream: %s"
>                     (wisent-parse semantic-toplevel-bovine-table
>                                   wisent-lexer-function
>                                   wisent-error-function
>                                   goal)))
>
>For the parser, the global performance cost should be negligible (one
>extra (if debug-on-error ...) test).  But for the developer the
>benefit could be important ;-)
>
>`semantic-safe' could be put in semantic-fw.el.  Or perhaps it should
>go into CEDET/common as `cedet-safe'?
  [ ... ]

This looks like a good idea.  semantic-fw.el seems like a good place
for it as well.  If the support for this sort of thing got much larger
with many other debugging forms, it would make sense to go in
cedet/common.

It would be nice if it could provide a clue for which rule and
expression was being run when the error was thrown in the message
statement.  I've inserted such code into the parser directly before,
and still been lost not knowing which chunk of code was broken in my
parser.  Doing so, however, would make it much more parser dependent,
probably going into semantic.el directly.

I'm also curious as to how noisy the above function would be.  I
don't know how many errors are secretly thrown and caught.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hi Eric,

[...]
> This looks like a good idea.  semantic-fw.el seems like a good place
> for it as well.  If the support for this sort of thing got much larger
> with many other debugging forms, it would make sense to go in
> cedet/common.

A patch follows providing a first implementation.  It mainly concerns
Wisent for now, but the bovinator has source debugging ;-)

> It would be nice if it could provide a clue for which rule and
> expression was being run when the error was thrown in the message
> statement.  I've inserted such code into the parser directly before,
> and still been lost not knowing which chunk of code was broken in my
> parser.  Doing so, however, would make it much more parser dependent,
> probably going into semantic.el directly.
[...]

I artificially introduced this subtle nasty error in
semantic-grammar.wy:

rules:
    lifo_rules
    (apply 'nconc (nreverse $$1))
  ;

where `$$1' is of course a "void variable".

When `debug-on-error' is not set and I open a grammar file I now get
this message, so I can suspect there is problem while parsing!

"wisent-parse-stream: Symbol's value as variable is void: $$1"

I retry with `debug-on-error' set to get the following nice back trace
which clearly tells me that the "void variable" error occurred in
"rules:0" semantic action, that is the first rule of nonterminal
`rules'.

Debugger entered--Lisp error: (void-variable $$1)
  (nreverse $$1)
  (apply (quote nconc) (nreverse $$1))
  (cons (apply (quote nconc) (nreverse $$1)) $region)
  (aset stack (1- sp) (cons (apply ... ...) $region))
  (let* (($region1 ...) ($region ...) ($nterm ...) ($action "rules:0")
  state) (setq sp (- sp 2) gotos (aref gotos ...) state (cdr ...) sp
  (+ sp 2)) (aset stack sp state) (aset stack (1- sp) (cons ...
  $region)) sp)
  rules:0([0 ("$$grammar") 9 ("input" 1513 . 1518) 37 ...
  [...]

While in the debugger, looking at `semantic-debug-parser-source'
value gives me what grammar file I have to fix. Very nice ;-)

e semantic-debug-parser-source RET
=3D=3D> "semantic-grammar.wy"

I can also use

M-x wisent-debug-on-entry RET semantic-grammar-wy--parse-table RET
rules:0 RET

to debug the suspect semantic action :-)

If you haven't objection, I will check the patch in.

Thoughts?

David

2004-02-18  David Ponce  <david@...>

=09* cedet/semantic/semantic-fw.el=20

=09(semantic-safe): New macro.

=09* cedet/semantic/semantic-idle.el

=09(semantic-idle-core-handler): New function.
=09(semantic-idle-scheduler-function): Call it through
=09`semantic-safe' to help debugging.

=09* cedet/semantic/wisent/wisent-bovine.el

=09(wisent-parse-stream): Use `semantic-safe' to help debugging.
=09(wisent-parse-region): Likewise.

Index: semantic/semantic-fw.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/semantic-fw.el,v
retrieving revision 1.27
diff -c -r1.27 semantic-fw.el
*** semantic/semantic-fw.el=0912 Feb 2004 02:03:25 -0000=091.27
--- semantic/semantic-fw.el=0918 Feb 2004 08:44:22 -0000
***************
*** 106,113 ****
  Mark OLDFNALIAS as obsolete, such that the byte compiler
  will throw a warning when it encounters this symbol."
    (defalias oldfnalias newfn)
!   (make-obsolete oldfnalias newfn)
!   )
 =20
  (defun semantic-varalias-obsolete (oldvaralias newvar)
    "Make OLDVARALIAS an alias for variable NEWVAR.
--- 106,112 ----
  Mark OLDFNALIAS as obsolete, such that the byte compiler
  will throw a warning when it encounters this symbol."
    (defalias oldfnalias newfn)
!   (make-obsolete oldfnalias newfn))
 =20
  (defun semantic-varalias-obsolete (oldvaralias newvar)
    "Make OLDVARALIAS an alias for variable NEWVAR.
***************
*** 124,139 ****
      can't make obsolete variable `%s'\n\
      alias of `%s'." (error-message-string err) oldvaralias newvar)
         )))
-=20
    (make-obsolete-variable oldvaralias newvar))
 =20
-=20
  ;;; Semantic autoloads
  ;;
! ;; Load semantic-al after compatibility code, to allow to use it in
! ;; autoloads without infinite recursive load problems.
  (load "semantic-loaddefs" nil t)
  =0C
  ;;; Misc utilities
  ;;
  (defun semantic-map-buffers (fun)
--- 123,156 ----
      can't make obsolete variable `%s'\n\
      alias of `%s'." (error-message-string err) oldvaralias newvar)
         )))
    (make-obsolete-variable oldvaralias newvar))
 =20
  ;;; Semantic autoloads
  ;;
! ;; Load semantic-loaddefs after compatibility code, to allow to use it
! ;; in autoloads without infinite recursive load problems.
  (load "semantic-loaddefs" nil t)
  =0C
+ ;;; Help debugging
+ ;;
+ (defmacro semantic-safe (format &rest body)
+   "Turn into a FORMAT message any error caught during eval of BODY.
+ Return the value of last BODY form or nil if an error occurred.
+ FORMAT can have a %s escape which will be replaced with the actual
+ error message.
+ If `debug-on-error' is set, errors are not caught, so that you can
+ debug them.
+ Avoid using a large BODY since it is duplicated."
+   ;;(declare (debug t) (indent 1))
+   `(if debug-on-error
+        (progn ,@body)
+      (condition-case err
+ =09 (progn ,@body)
+        (error
+         (message ,format (error-message-string err))
+         nil))))
+ (put 'semantic-safe 'lisp-indent-function 1)
+=20
  ;;; Misc utilities
  ;;
  (defun semantic-map-buffers (fun)
***************
*** 165,171 ****
            (with-current-buffer b
              (if (memq major-mode modes)
                  (funcall fun)))))))
-=20
  =0C
  ;;; Behavioral APIs
  ;;
--- 182,187 ----
***************
*** 546,552 ****
  =09(semantic-augment-function-help (ad-get-arg 0)))))
 =20
  =0C
! ;;; User Interrupe handling
  ;;
  (defvar semantic-current-input-throw-symbol nil
    "The current throw symbol for `semantic-exit-on-input'.")
--- 562,568 ----
  =09(semantic-augment-function-help (ad-get-arg 0)))))
 =20
  =0C
! ;;; User Interrupt handling
  ;;
  (defvar semantic-current-input-throw-symbol nil
    "The current throw symbol for `semantic-exit-on-input'.")
Index: semantic/semantic-idle.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/semantic-idle.el,v
retrieving revision 1.16
diff -c -r1.16 semantic-idle.el
*** semantic/semantic-idle.el=0912 Feb 2004 02:04:33 -0000=091.16
--- semantic/semantic-idle.el=0918 Feb 2004 08:44:22 -0000
***************
*** 202,209 ****
 =20
  ;;; IDLE Function
  ;;
! ;; This is the idle function which handles reparsing, and also
! ;; manages services that depend on tag values.
  (defun semantic-idle-scheduler-function ()
    "Function run when after `semantc-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
--- 202,236 ----
 =20
  ;;; IDLE Function
  ;;
! (defun semantic-idle-core-handler ()
!   "Core idle function that handles reparsing.
! And also manages services that depend on tag values."
!   (semantic-exit-on-input 'idle-timer
!     ;; First, reparse the current buffer.
!     (let ((inhibit-quit nil)
! =09  (lexok (semantic-idle-scheduler-refresh-tags))
! =09  (buffers (buffer-list))
! =09  (queue semantic-idle-scheduler-queue)
! =09  )
!=20
!       ;; Now loop over other buffers, trying to update them as well.
!       (save-excursion
! =09(while buffers
! =09  (semantic-throw-on-input 'parsing-all-buffers)
! =09  (set-buffer (car buffers))
! =09  (when semantic-idle-scheduler-mode
! =09    (semantic-idle-scheduler-refresh-tags)
! =09    )
! =09  (setq buffers (cdr buffers))))
!=20
!       ;; Evaluate all other services.  Stop on keypress.
!       (save-excursion
! =09(while queue
! =09  (semantic-throw-on-input 'idle-queue)
! =09  (funcall (car queue))
! =09  (setq queue (cdr queue))))
!       )))
!=20
  (defun semantic-idle-scheduler-function ()
    "Function run when after `semantc-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
***************
*** 213,246 ****
      ;; Disable the auto parse timer while re-parsing
      (semantic-idle-scheduler-kill-timer)
 =20
!     (condition-case err
! =09(semantic-exit-on-input 'idle-timer
! =09  ;; First, reparse the current buffer.
! =09  (let ((inhibit-quit nil)
! =09=09(lexok (semantic-idle-scheduler-refresh-tags))
! =09=09(buffers (buffer-list))
! =09=09(queue semantic-idle-scheduler-queue)
! =09=09)
! =09   =20
! =09    ;; Now loop over other buffers, trying to update them as well.
! =09    (save-excursion
! =09      (while buffers
! =09=09(semantic-throw-on-input 'parsing-all-buffers)
! =09=09(set-buffer (car buffers))
! =09=09(when semantic-idle-scheduler-mode
! =09=09  (semantic-idle-scheduler-refresh-tags)
! =09=09  )
! =09=09(setq buffers (cdr buffers))))
! =09   =20
! =09    ;; Evaluate all other services.  Stop on keypress.
! =09    (save-excursion
! =09      (while queue
! =09=09(semantic-throw-on-input 'idle-queue)
! =09=09(funcall (car queue))
! =09=09(setq queue (cdr queue))))
! =09    ))
!       (error (message "idle error: %S" err)))
!    =20
      ;; Enable again the auto parse timer
      (semantic-idle-scheduler-setup-timer)))
  =0C
--- 240,249 ----
      ;; Disable the auto parse timer while re-parsing
      (semantic-idle-scheduler-kill-timer)
 =20
!     ;; Handle re-parsing and other scheduled services
!     (semantic-safe "idle error: %S"
!       (semantic-idle-core-handler))
!=20
      ;; Enable again the auto parse timer
      (semantic-idle-scheduler-setup-timer)))
  =0C
Index: semantic/wisent/wisent-bovine.el
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-bovine.el,v
retrieving revision 1.31
diff -c -r1.31 wisent-bovine.el
*** semantic/wisent/wisent-bovine.el=0923 Jan 2004 08:34:57 -0000=091.31
--- semantic/wisent/wisent-bovine.el=0918 Feb 2004 08:44:23 -0000
***************
*** 218,229 ****
        (setq la-elt nil))
      ;; Parse
      (setq wisent-lex-istream stream
!           cache (condition-case nil
                      (wisent-parse semantic-toplevel-bovine-table
                                    wisent-lexer-function
                                    wisent-error-function
!                                   goal)
!                   (error nil)))
      ;; Manage returned lookahead token
      (if wisent-lookahead
          (if (eq (caar la-elt) wisent-lookahead)
--- 218,228 ----
        (setq la-elt nil))
      ;; Parse
      (setq wisent-lex-istream stream
!           cache (semantic-safe "wisent-parse-stream: %s"
                      (wisent-parse semantic-toplevel-bovine-table
                                    wisent-lexer-function
                                    wisent-error-function
!                                   goal)))
      ;; Manage returned lookahead token
      (if wisent-lookahead
          (if (eq (caar la-elt) wisent-lookahead)
***************
*** 272,283 ****
      (while wisent-lex-istream
        ;; Parse
        (setq wisent-lex-lookahead (car lstack)
!             tag (condition-case nil
                      (wisent-parse semantic-toplevel-bovine-table
                                    wisent-lexer-function
                                    wisent-error-function
!                                   goal)
!                   (error nil)))
        ;; Manage returned lookahead token
        (if wisent-lookahead
            (if (eq (car lstack) wisent-lookahead)
--- 271,281 ----
      (while wisent-lex-istream
        ;; Parse
        (setq wisent-lex-lookahead (car lstack)
!             tag (semantic-safe "wisent-parse-region: %s"
                      (wisent-parse semantic-toplevel-bovine-table
                                    wisent-lexer-function
                                    wisent-error-function
!                                   goal)))
        ;; Manage returned lookahead token
        (if wisent-lookahead
            (if (eq (car lstack) wisent-lookahead)

Eric,

Here is another patch for semantic-idle.el that follows my previous one.

Implementation now take into account
`semantic-idle-scheduler-max-buffer-size'.

Also, as it looked that scheduled services were evaluated only in
current buffer, I re-ordered the process to evaluate them just after
re-parsing the current buffer completed, before re-parsing other
buffers.

That gives a better chance to refresh more things in the current
buffer during idle time.  After all the current buffer is the one
you're working on ;-)

What do you think?

David

2004-02-18  David Ponce  <david@...>  NOT YET COMMITTED

	* hacking/semantic-idle.el
	
	(semantic-idle-scheduler-enabled-p): Take into account
	`semantic-idle-scheduler-max-buffer-size'.
	(semantic-idle-core-handler): If re-parse of current buffer
	completed, process scheduled services on current buffer before to
	try to re-parse other buffers.
	(semantic-idle-scheduler-function)
	(semantic-idle-scheduler-refresh-tags): Doc fix.

*** /home/ponce/emacs21/cedet/semantic/semantic-idle.el	2004-02-18 09:19:47.000000000 +0100
--- /home/ponce/emacs21/hacking/semantic-idle.el	2004-02-18 13:40:37.000000000 +0100
***************
*** 118,127 ****
  
  (defsubst semantic-idle-scheduler-enabled-p ()
    "Return non-nil if idle-scheduler is enabled for this buffer.
! See also the variable `semantic-idle-scheduler-max-buffer-size'."
!   (if semantic-idle-scheduler-mode
!       (and (not semantic-debug-enabled)
! 	   (not semantic-lex-debug))))
  
  (defun semantic-idle-scheduler-mode-setup ()
    "Setup option `semantic-idle-scheduler-mode'.
--- 118,130 ----
  
  (defsubst semantic-idle-scheduler-enabled-p ()
    "Return non-nil if idle-scheduler is enabled for this buffer.
! idle-scheduler is disabled when debugging or if the buffer size
! exceeds the `semantic-idle-scheduler-max-buffer-size' threshold."
!   (and semantic-idle-scheduler-mode
!        (not semantic-debug-enabled)
!        (not semantic-lex-debug)
!        (or (<= semantic-idle-scheduler-max-buffer-size 0)
! 	   (< (buffer-size) semantic-idle-scheduler-max-buffer-size))))
  
  (defun semantic-idle-scheduler-mode-setup ()
    "Setup option `semantic-idle-scheduler-mode'.
***************
*** 206,238 ****
    "Core idle function that handles reparsing.
  And also manages services that depend on tag values."
    (semantic-exit-on-input 'idle-timer
-     ;; First, reparse the current buffer.
      (let ((inhibit-quit nil)
! 	  (lexok (semantic-idle-scheduler-refresh-tags))
! 	  (buffers (buffer-list))
! 	  (queue semantic-idle-scheduler-queue)
! 	  )
! 
!       ;; Now loop over other buffers, trying to update them as well.
!       (save-excursion
! 	(while buffers
! 	  (semantic-throw-on-input 'parsing-all-buffers)
! 	  (set-buffer (car buffers))
! 	  (when semantic-idle-scheduler-mode
! 	    (semantic-idle-scheduler-refresh-tags)
! 	    )
! 	  (setq buffers (cdr buffers))))
! 
!       ;; Evaluate all other services.  Stop on keypress.
        (save-excursion
! 	(while queue
! 	  (semantic-throw-on-input 'idle-queue)
! 	  (funcall (car queue))
! 	  (setq queue (cdr queue))))
        )))
  
  (defun semantic-idle-scheduler-function ()
!   "Function run when after `semantc-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
  call additional functions registered with the timer calls."
    (when (semantic-idle-scheduler-enabled-p)

--- 209,240 ----
    "Core idle function that handles reparsing.
  And also manages services that depend on tag values."
    (semantic-exit-on-input 'idle-timer
      (let ((inhibit-quit nil)
! 	  (buffers (delq (current-buffer) (buffer-list)))
! 	  (queue semantic-idle-scheduler-queue))
        (save-excursion
! 	;; First, reparse the current buffer.
! 	(when (semantic-idle-scheduler-refresh-tags)
!           ;; If completed, evaluate all other services.
!           ;; Stop on keypress.
!           (while queue
!             (semantic-throw-on-input 'idle-queue)
!             (funcall (car queue))
!             (setq queue (cdr queue)))))
!        (save-excursion
! 	 ;; Now loop over other buffers, trying to update them as
! 	 ;; well.  Stop on keypress.
! 	 (while buffers
! 	   (semantic-throw-on-input 'parsing-all-buffers)
! 	   (when (buffer-live-p (car buffers))
! 	     (set-buffer (car buffers))
! 	     (and (semantic-idle-scheduler-enabled-p)
! 		  (semantic-idle-scheduler-refresh-tags)))
! 	   (setq buffers (cdr buffers))))
        )))
  
  (defun semantic-idle-scheduler-function ()
!   "Function run when after `semantic-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
  call additional functions registered with the timer calls."
    (when (semantic-idle-scheduler-enabled-p)
***************
*** 291,298 ****
  Return nil if there is some sort of syntax error preventing a full
  reparse.
  
! Does nothing if the current buffer doesn't need reparsing or if its
! size exceeds the `semantic-idle-scheduler-max-buffer-size' threshold."
  
    (let* ((semantic-bovination-working-type nil)
  	 (inhibit-quit nil)
--- 293,299 ----
  Return nil if there is some sort of syntax error preventing a full
  reparse.
  
! Does nothing if the current buffer doesn't need reparsing."
  
    (let* ((semantic-bovination-working-type nil)
  	 (inhibit-quit nil)

Oops!

It looks that I made a mistake in my previous patch proposal, as
scheduled services might need refreshed data from other buffers
(for completion for instance).  So services must be run after re-parsing
other buffers.

Here is a new version of `semantic-idle-core-handler':

(defun semantic-idle-core-handler ()
  "Core idle function that handles reparsing.
And also manages services that depend on tag values."
  (semantic-exit-on-input 'idle-timer
    (let* ((inhibit-quit nil)
           (buffers (delq (current-buffer) (buffer-list)))
           (queue semantic-idle-scheduler-queue)
           ;; First, reparse the current buffer.
           (safe (semantic-idle-scheduler-refresh-tags)))
      ;; Now loop over other buffers, trying to update them as
      ;; well.  Stop on keypress.
      (save-excursion
        (while buffers
          (semantic-throw-on-input 'parsing-all-buffers)
          (when (buffer-live-p (car buffers))
            (set-buffer (car buffers))
            (and (semantic-idle-scheduler-enabled-p)
                 (semantic-idle-scheduler-refresh-tags)))
          (setq buffers (cdr buffers))))
      ;; If re-parse of current buffer completed, evaluate all other
      ;; services.  Stop on keypress.
      (when safe
        (save-excursion
          (while queue
            (semantic-throw-on-input 'idle-queue)
            (funcall (car queue))
            (setq queue (cdr queue)))))
      )))

Perhaps things could be improved to better use the idle time, like
this:

1. Re-parse the current buffer.

2. Re-parse other buffers having the same major mode as the current
   buffer.

3. Eval scheduled services (only if re-parse of the current buffer
   succeeded?).

4. Re-parse other remaining buffers.

Does that look correct?  If so, I will try an implementation.

David

David,

  That sounds like a great solution now that I understand it better.
Please do check it in.

Thanks 
Eric

>>> David PONCE <david.ponce@...> seems to think that:
>Hi Eric,
>
>[...]
>> This looks like a good idea.  semantic-fw.el seems like a good place
>> for it as well.  If the support for this sort of thing got much larger
>> with many other debugging forms, it would make sense to go in
>> cedet/common.
>
>A patch follows providing a first implementation.  It mainly concerns
>Wisent for now, but the bovinator has source debugging ;-)
>
>> It would be nice if it could provide a clue for which rule and
>> expression was being run when the error was thrown in the message
>> statement.  I've inserted such code into the parser directly before,
>> and still been lost not knowing which chunk of code was broken in my
>> parser.  Doing so, however, would make it much more parser dependent,
>> probably going into semantic.el directly.
>[...]
>
>I artificially introduced this subtle nasty error in
>semantic-grammar.wy:
>
>rules:
>    lifo_rules
>    (apply 'nconc (nreverse $$1))
>  ;
>
>where `$$1' is of course a "void variable".
>
>When `debug-on-error' is not set and I open a grammar file I now get
>this message, so I can suspect there is problem while parsing!
>
>"wisent-parse-stream: Symbol's value as variable is void: $$1"
>
>I retry with `debug-on-error' set to get the following nice back trace
>which clearly tells me that the "void variable" error occurred in
>"rules:0" semantic action, that is the first rule of nonterminal
>`rules'.
>
>Debugger entered--Lisp error: (void-variable $$1)
>  (nreverse $$1)
>  (apply (quote nconc) (nreverse $$1))
>  (cons (apply (quote nconc) (nreverse $$1)) $region)
>  (aset stack (1- sp) (cons (apply ... ...) $region))
>  (let* (($region1 ...) ($region ...) ($nterm ...) ($action "rules:0")
>  state) (setq sp (- sp 2) gotos (aref gotos ...) state (cdr ...) sp
>  (+ sp 2)) (aset stack sp state) (aset stack (1- sp) (cons ...
>  $region)) sp)
>  rules:0([0 ("$$grammar") 9 ("input" 1513 . 1518) 37 ...
>  [...]
>
>While in the debugger, looking at `semantic-debug-parser-source'
>value gives me what grammar file I have to fix. Very nice ;-)
>
>e semantic-debug-parser-source RET
>==> "semantic-grammar.wy"
>
>I can also use
>
>M-x wisent-debug-on-entry RET semantic-grammar-wy--parse-table RET
>rules:0 RET
>
>to debug the suspect semantic action :-)
>
>If you haven't objection, I will check the patch in.
>
>Thoughts?
>
>David
>
>2004-02-18  David Ponce  <david@...>
>
>	* cedet/semantic/semantic-fw.el 
>
>	(semantic-safe): New macro.
>
>	* cedet/semantic/semantic-idle.el
>
>	(semantic-idle-core-handler): New function.
>	(semantic-idle-scheduler-function): Call it through
>	`semantic-safe' to help debugging.
>
>	* cedet/semantic/wisent/wisent-bovine.el
>
>	(wisent-parse-stream): Use `semantic-safe' to help debugging.
>	(wisent-parse-region): Likewise.
>
>Index: semantic/semantic-fw.el
>===================================================================
>RCS file: /cvsroot/cedet/cedet/semantic/semantic-fw.el,v
>retrieving revision 1.27
>diff -c -r1.27 semantic-fw.el
>*** semantic/semantic-fw.el	12 Feb 2004 02:03:25 -0000	1.27
>--- semantic/semantic-fw.el	18 Feb 2004 08:44:22 -0000
>***************
>*** 106,113 ****
>  Mark OLDFNALIAS as obsolete, such that the byte compiler
>  will throw a warning when it encounters this symbol."
>    (defalias oldfnalias newfn)
>!   (make-obsolete oldfnalias newfn)
>!   )
>  
>  (defun semantic-varalias-obsolete (oldvaralias newvar)
>    "Make OLDVARALIAS an alias for variable NEWVAR.
>--- 106,112 ----
>  Mark OLDFNALIAS as obsolete, such that the byte compiler
>  will throw a warning when it encounters this symbol."
>    (defalias oldfnalias newfn)
>!   (make-obsolete oldfnalias newfn))
>  
>  (defun semantic-varalias-obsolete (oldvaralias newvar)
>    "Make OLDVARALIAS an alias for variable NEWVAR.
>***************
>*** 124,139 ****
>      can't make obsolete variable `%s'\n\
>      alias of `%s'." (error-message-string err) oldvaralias newvar)
>         )))
>- 
>    (make-obsolete-variable oldvaralias newvar))
>  
>- 
>  ;;; Semantic autoloads
>  ;;
>! ;; Load semantic-al after compatibility code, to allow to use it in
>! ;; autoloads without infinite recursive load problems.
>  (load "semantic-loaddefs" nil t)
>  
>  ;;; Misc utilities
>  ;;
>  (defun semantic-map-buffers (fun)
>--- 123,156 ----
>      can't make obsolete variable `%s'\n\
>      alias of `%s'." (error-message-string err) oldvaralias newvar)
>         )))
>    (make-obsolete-variable oldvaralias newvar))
>  
>  ;;; Semantic autoloads
>  ;;
>! ;; Load semantic-loaddefs after compatibility code, to allow to use it
>! ;; in autoloads without infinite recursive load problems.
>  (load "semantic-loaddefs" nil t)
>  
>+ ;;; Help debugging
>+ ;;
>+ (defmacro semantic-safe (format &rest body)
>+   "Turn into a FORMAT message any error caught during eval of BODY.
>+ Return the value of last BODY form or nil if an error occurred.
>+ FORMAT can have a %s escape which will be replaced with the actual
>+ error message.
>+ If `debug-on-error' is set, errors are not caught, so that you can
>+ debug them.
>+ Avoid using a large BODY since it is duplicated."
>+   ;;(declare (debug t) (indent 1))
>+   `(if debug-on-error
>+        (progn ,@body)
>+      (condition-case err
>+ 	 (progn ,@body)
>+        (error
>+         (message ,format (error-message-string err))
>+         nil))))
>+ (put 'semantic-safe 'lisp-indent-function 1)
>+ 
>  ;;; Misc utilities
>  ;;
>  (defun semantic-map-buffers (fun)
>***************
>*** 165,171 ****
>            (with-current-buffer b
>              (if (memq major-mode modes)
>                  (funcall fun)))))))
>- 
>  
>  ;;; Behavioral APIs
>  ;;
>--- 182,187 ----
>***************
>*** 546,552 ****
>  	(semantic-augment-function-help (ad-get-arg 0)))))
>  
>  
>! ;;; User Interrupe handling
>  ;;
>  (defvar semantic-current-input-throw-symbol nil
>    "The current throw symbol for `semantic-exit-on-input'.")
>--- 562,568 ----
>  	(semantic-augment-function-help (ad-get-arg 0)))))
>  
>  
>! ;;; User Interrupt handling
>  ;;
>  (defvar semantic-current-input-throw-symbol nil
>    "The current throw symbol for `semantic-exit-on-input'.")
>Index: semantic/semantic-idle.el
>===================================================================
>RCS file: /cvsroot/cedet/cedet/semantic/semantic-idle.el,v
>retrieving revision 1.16
>diff -c -r1.16 semantic-idle.el
>*** semantic/semantic-idle.el	12 Feb 2004 02:04:33 -0000	1.16
>--- semantic/semantic-idle.el	18 Feb 2004 08:44:22 -0000
>***************
>*** 202,209 ****
>  
>  ;;; IDLE Function
>  ;;
>! ;; This is the idle function which handles reparsing, and also
>! ;; manages services that depend on tag values.
>  (defun semantic-idle-scheduler-function ()
>    "Function run when after `semantc-idle-scheduler-idle-time'.
>  This function will reparse the current buffer, and if successful,
>--- 202,236 ----
>  
>  ;;; IDLE Function
>  ;;
>! (defun semantic-idle-core-handler ()
>!   "Core idle function that handles reparsing.
>! And also manages services that depend on tag values."
>!   (semantic-exit-on-input 'idle-timer
>!     ;; First, reparse the current buffer.
>!     (let ((inhibit-quit nil)
>! 	  (lexok (semantic-idle-scheduler-refresh-tags))
>! 	  (buffers (buffer-list))
>! 	  (queue semantic-idle-scheduler-queue)
>! 	  )
>! 
>!       ;; Now loop over other buffers, trying to update them as well.
>!       (save-excursion
>! 	(while buffers
>! 	  (semantic-throw-on-input 'parsing-all-buffers)
>! 	  (set-buffer (car buffers))
>! 	  (when semantic-idle-scheduler-mode
>! 	    (semantic-idle-scheduler-refresh-tags)
>! 	    )
>! 	  (setq buffers (cdr buffers))))
>! 
>!       ;; Evaluate all other services.  Stop on keypress.
>!       (save-excursion
>! 	(while queue
>! 	  (semantic-throw-on-input 'idle-queue)
>! 	  (funcall (car queue))
>! 	  (setq queue (cdr queue))))
>!       )))
>! 
>  (defun semantic-idle-scheduler-function ()
>    "Function run when after `semantc-idle-scheduler-idle-time'.
>  This function will reparse the current buffer, and if successful,
>***************
>*** 213,246 ****
>      ;; Disable the auto parse timer while re-parsing
>      (semantic-idle-scheduler-kill-timer)
>  
>!     (condition-case err
>! 	(semantic-exit-on-input 'idle-timer
>! 	  ;; First, reparse the current buffer.
>! 	  (let ((inhibit-quit nil)
>! 		(lexok (semantic-idle-scheduler-refresh-tags))
>! 		(buffers (buffer-list))
>! 		(queue semantic-idle-scheduler-queue)
>! 		)
>! 	    
>! 	    ;; Now loop over other buffers, trying to update them as well.
>! 	    (save-excursion
>! 	      (while buffers
>! 		(semantic-throw-on-input 'parsing-all-buffers)
>! 		(set-buffer (car buffers))
>! 		(when semantic-idle-scheduler-mode
>! 		  (semantic-idle-scheduler-refresh-tags)
>! 		  )
>! 		(setq buffers (cdr buffers))))
>! 	    
>! 	    ;; Evaluate all other services.  Stop on keypress.
>! 	    (save-excursion
>! 	      (while queue
>! 		(semantic-throw-on-input 'idle-queue)
>! 		(funcall (car queue))
>! 		(setq queue (cdr queue))))
>! 	    ))
>!       (error (message "idle error: %S" err)))
>!     
>      ;; Enable again the auto parse timer
>      (semantic-idle-scheduler-setup-timer)))
>  
>--- 240,249 ----
>      ;; Disable the auto parse timer while re-parsing
>      (semantic-idle-scheduler-kill-timer)
>  
>!     ;; Handle re-parsing and other scheduled services
>!     (semantic-safe "idle error: %S"
>!       (semantic-idle-core-handler))
>! 
>      ;; Enable again the auto parse timer
>      (semantic-idle-scheduler-setup-timer)))
>  
>Index: semantic/wisent/wisent-bovine.el
>===================================================================
>RCS file: /cvsroot/cedet/cedet/semantic/wisent/wisent-bovine.el,v
>retrieving revision 1.31
>diff -c -r1.31 wisent-bovine.el
>*** semantic/wisent/wisent-bovine.el	23 Jan 2004 08:34:57 -0000	1.31
>--- semantic/wisent/wisent-bovine.el	18 Feb 2004 08:44:23 -0000
>***************
>*** 218,229 ****
>        (setq la-elt nil))
>      ;; Parse
>      (setq wisent-lex-istream stream
>!           cache (condition-case nil
>                      (wisent-parse semantic-toplevel-bovine-table
>                                    wisent-lexer-function
>                                    wisent-error-function
>!                                   goal)
>!                   (error nil)))
>      ;; Manage returned lookahead token
>      (if wisent-lookahead
>          (if (eq (caar la-elt) wisent-lookahead)
>--- 218,228 ----
>        (setq la-elt nil))
>      ;; Parse
>      (setq wisent-lex-istream stream
>!           cache (semantic-safe "wisent-parse-stream: %s"
>                      (wisent-parse semantic-toplevel-bovine-table
>                                    wisent-lexer-function
>                                    wisent-error-function
>!                                   goal)))
>      ;; Manage returned lookahead token
>      (if wisent-lookahead
>          (if (eq (caar la-elt) wisent-lookahead)
>***************
>*** 272,283 ****
>      (while wisent-lex-istream
>        ;; Parse
>        (setq wisent-lex-lookahead (car lstack)
>!             tag (condition-case nil
>                      (wisent-parse semantic-toplevel-bovine-table
>                                    wisent-lexer-function
>                                    wisent-error-function
>!                                   goal)
>!                   (error nil)))
>        ;; Manage returned lookahead token
>        (if wisent-lookahead
>            (if (eq (car lstack) wisent-lookahead)
>--- 271,281 ----
>      (while wisent-lex-istream
>        ;; Parse
>        (setq wisent-lex-lookahead (car lstack)
>!             tag (semantic-safe "wisent-parse-region: %s"
>                      (wisent-parse semantic-toplevel-bovine-table
>                                    wisent-lexer-function
>                                    wisent-error-function
>!                                   goal)))
>        ;; Manage returned lookahead token
>        (if wisent-lookahead
>            (if (eq (car lstack) wisent-lookahead)
>
>
>

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David PONCE <david.ponce@...> seems to think that:
>Eric,
>
>Here is another patch for semantic-idle.el that follows my previous one.
>
>Implementation now take into account
>`semantic-idle-scheduler-max-buffer-size'.

Heh, good idea. ;)

>Also, as it looked that scheduled services were evaluated only in
>current buffer, I re-ordered the process to evaluate them just after
>re-parsing the current buffer completed, before re-parsing other
>buffers.
>
>That gives a better chance to refresh more things in the current
>buffer during idle time.  After all the current buffer is the one
>you're working on ;-)

What happens when the reparsing is ordered in this way is that it
reparses the current buffer, then executes the summary mode, which
calls into semanticdb, which reparses those buffers (if they are in
the same project area) anyway.

The list of reparsing and idle functions is supposed to exit on user
input (letting Emacs act snappier).  Putting the parsing before the
idle functions lets the idle functions take less time, cutting back
on the user visible delay.  I'm not really sure which is the best
solution of the two.

That all being said, the user-input level exiting doesn't seem to
work when byte compiled.  I haven't quite figured it out yet why that
is.  If the macro is used in `semantic-test-throw-on-input' it seems
to work, but the idle timer does not. :(  If it did, the delays may
become more obvious.

Thanks
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David PONCE <david.ponce@...> seems to think that:
>Oops!
>
>It looks that I made a mistake in my previous patch proposal, as
>scheduled services might need refreshed data from other buffers
>(for completion for instance).  So services must be run after re-parsing
>other buffers.
  [ ... ]
>Perhaps things could be improved to better use the idle time, like
>this:
>
>1. Re-parse the current buffer.
>
>2. Re-parse other buffers having the same major mode as the current
>   buffer.
>
>3. Eval scheduled services (only if re-parse of the current buffer
>   succeeded?).
>
>4. Re-parse other remaining buffers.
>
>Does that look correct?  If so, I will try an implementation.
  [ ... ]

That makes good sense.  If you have an idea on implementing it, that
would be great.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Eric,

I committed the `semantic-safe' changes and the
`semantic-idle-scheduler-max-buffer-size' fix.

[...]
>>Perhaps things could be improved to better use the idle time, like
>>this:
>>
>>1. Re-parse the current buffer.
>>
>>2. Re-parse other buffers having the same major mode as the current
>>  buffer.
>>
>>3. Eval scheduled services (only if re-parse of the current buffer
>>  succeeded?).
>>
>>4. Re-parse other remaining buffers.
>>
>>Does that look correct?  If so, I will try an implementation.
> 
>   [ ... ]
> 
> That makes good sense.  If you have an idea on implementing it, that
> would be great.

Here it is ;-)

David

Index: semantic-idle.el
===================================================================

RCS file: /cvsroot/cedet/cedet/semantic/semantic-idle.el,v
retrieving revision 1.17
diff -c -r1.17 semantic-idle.el
*** semantic-idle.el	18 Feb 2004 14:38:59 -0000	1.17
--- semantic-idle.el	18 Feb 2004 14:59:17 -0000
***************
*** 210,237 ****
  And also manages services that depend on tag values."
    (semantic-exit-on-input 'idle-timer
      (let* ((inhibit-quit nil)
! 	   (buffers (delq (current-buffer) (buffer-list)))
! 	   (queue semantic-idle-scheduler-queue)
! 	   ;; First, reparse the current buffer.
! 	   (lexok (semantic-idle-scheduler-refresh-tags))
! 	   )
! 
!       ;; Now loop over other buffers, trying to update them as well.
        (save-excursion
! 	(while buffers
! 	  (semantic-throw-on-input 'parsing-all-buffers)
! 	  (when (buffer-live-p (car buffers))
! 	    (set-buffer (car buffers))
! 	    (and (semantic-idle-scheduler-enabled-p)
! 		 (semantic-idle-scheduler-refresh-tags)))
! 	  (setq buffers (cdr buffers))))
! 
!       ;; Evaluate all other services.  Stop on keypress.
        (save-excursion
! 	(while queue
! 	  (semantic-throw-on-input 'idle-queue)
! 	  (funcall (car queue))
! 	  (setq queue (cdr queue))))
        )))
  
  (defun semantic-idle-scheduler-function ()
--- 210,250 ----
  And also manages services that depend on tag values."
    (semantic-exit-on-input 'idle-timer
      (let* ((inhibit-quit nil)
!            (mode    major-mode)
!            (buffers (delq (current-buffer) (buffer-list)))
!            (others  nil)
!            (queue   semantic-idle-scheduler-queue)
!            ;; First, reparse the current buffer.
!            (safe    (semantic-idle-scheduler-refresh-tags)))
!       ;; Now loop over other buffers with same major mode, trying to
!       ;; update them as well.  Stop on keypress.
        (save-excursion
!         (while buffers
!           (semantic-throw-on-input 'parsing-mode-buffers)
!           (when (buffer-live-p (car buffers))
!             (set-buffer (car buffers))
!             (when (semantic-idle-scheduler-enabled-p)
!               (if (eq major-mode mode)
!                   (semantic-idle-scheduler-refresh-tags)
!                 (setq others (cons (current-buffer) others)))))
!           (setq buffers (cdr buffers))))
!       ;; If re-parse of current buffer completed, evaluate all other
!       ;; services.  Stop on keypress.
!       (when safe
!         (save-excursion
!           (while queue
!             (semantic-throw-on-input 'idle-queue)
!             (funcall (car queue))
!             (setq queue (cdr queue)))))
!       ;; Finally loop over remaining buffers, trying to update them as
!       ;; well.  Stop on keypress.
        (save-excursion
!         (while others
!           (semantic-throw-on-input 'parsing-other-buffers)
!           (set-buffer (car others))
!           ;; No need to check more here?
!           (semantic-idle-scheduler-refresh-tags)
!           (setq others (cdr others))))
        )))
  
  (defun semantic-idle-scheduler-function ()

Hello David and Eric,

"Eric M. Ludlam" <eric@...> writes:

>>>> David PONCE <david.ponce@...> seems to think that:
>>>   | id COLON type_or_nonempty_type_or_type_plus from_typeexpr_opt
>>>     (let ((variables (cons $1 my-theoryformal-idops))
>>>           (result (VARIABLE-TAG variables $3 nil)))
>>>       (message "%S:%S %S" $1 $3 $4)
>>>       (message "variables = %S" variables)
>>>       (message "result = %S" result)
>>>       (setq my-theoryformal-idops nil)
>>>       result)
>>>   | id COLON typeexpr
>>>     (let ((variables (cons $1 my-theoryformal-idops))
>>>           (result (VARIABLE-TAG variables $3 nil)))
>>>       (message "%S:%S" $1 $3)
>>>       (message "variables = %S" variables)
>>>       (message "result = %S" result)
>>>       (setq my-theoryformal-idops nil)
>>>       result)
>>>   | opsym COLON typeexpr
>>>     (let ((variables (cons $1 my-theoryformal-idops))
>>>           (result (VARIABLE-TAG variables $3 nil)))
>>>       (message "%S:%S" $1 $3)
>>>       (message "variables = %S" variables)
>>>       (message "result = %S" result)
>>>       (setq my-theoryformal-idops nil)
>>>       result)
>>
>>It looks that the `let' statements above have to be `let*' because
>>the value of `variables' is used to compute the value of `result'.

Thanks for the sharp eye!  That was indeed the problem with this rule.

>>IMO the `let' clauses above fail with a "(void-variable variables)"
>>error which is probably masked by a `condition-case' at a upper level
>>in semantic code.
>   [ ... ]
>
> I have long wondered how to handle situations like this.  I went
> this route because it allowed a user to throw complete garbage at a
> parser, and the parser would live through it.  This lets you allow
> parsing to be pervasive in -hooks, -functions, and utilities that
> would suffer if the parser threw an error because some esoteric file
> the user doesn't care about got on some list.
>
> On the other hand, it makes development tricky if you don't know what
> to look for.

You're telling me! :)

> I went for option 1 as I knew the tricks to find problems.
>
> In Emacs, you cannot hook into the debug system yourself.
>
> Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
> or a derivation, it could create it in some sort of instrumented
> fashion that allows errors to propagate out of the system more easily.
>
> For example, when an action is executed, instrumentation would make
> note of it's location, and errors would be caught jumping the
> developer to the correct buffer location.
>
> Eric

The new debug framework that you two dropped in over the past couple
of days is working very well for me.  Now, errors in Lisp are not
completely hidden from me and I can debug more effectively.

As I march forward though, I'm having problems with grammar
compilation.  I make changes to my grammar and compile it using
semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
force regeneration, the generated lisp code relating to the rule that
I have just updated remains unchanged.

If I quit and restart emacs then compilation succeeds.  I did not
touch semantic.cache.

Any idea what's going on?

Joe

>>> Joseph Kiniry <kiniry@...> seems to think that:
  [ ... ]
>> Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
>> or a derivation, it could create it in some sort of instrumented
>> fashion that allows errors to propagate out of the system more easily.
>>
>> For example, when an action is executed, instrumentation would make
>> note of it's location, and errors would be caught jumping the
>> developer to the correct buffer location.
>>
>> Eric
>
>The new debug framework that you two dropped in over the past couple
>of days is working very well for me.  Now, errors in Lisp are not
>completely hidden from me and I can debug more effectively.

Huzzah!

>As I march forward though, I'm having problems with grammar
>compilation.  I make changes to my grammar and compile it using
>semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
>force regeneration, the generated lisp code relating to the rule that
>I have just updated remains unchanged.
>
>If I quit and restart emacs then compilation succeeds.  I did not
>touch semantic.cache.
  [ ... ]

That's curious.  I saw something similar while fiddling with idle time
completion where it looked like the cache was not accepting
incremental edits.  I hit "C-u M-x bovinate", and everything was
better.  As I was fiddling in the idle time reparse and errors were
being thrown, I thought I had merely broken something in that buffer.
I only saw it the one time.

Does this happen once or every time?  I suspect David would be better
equipped to answer this question.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

>>>> Joseph Kiniry <kiniry@...> seems to think that:
>   [ ... ]
>>> Perhaps when a compiler is made from a .wy or .by buffer with C-c C-c
>>> or a derivation, it could create it in some sort of instrumented
>>> fashion that allows errors to propagate out of the system more easily.
>>>
>>> For example, when an action is executed, instrumentation would make
>>> note of it's location, and errors would be caught jumping the
>>> developer to the correct buffer location.
>>>
>>> Eric
>>
>>The new debug framework that you two dropped in over the past couple
>>of days is working very well for me.  Now, errors in Lisp are not
>>completely hidden from me and I can debug more effectively.
>
> Huzzah!

Indeed!  Now I'm finally writing my expand-tag function, and its a
beast!  E.g., a single relativly small datatype declaration like this:

  stack[T: TYPE, U:TYPE+]: DATATYPE 
     WITH SUBTYPES empty-stack, non-empty-stack
   BEGIN
    empty: empty?: empty-stack
    push(top:T, pop:stack): nonempty?: non-empty-stack
   END stack

ends up implicitly defining one top-level type, four subtypes, two
constructors, four boolean functions (recognizers), two accessor
functions, a datatype function, and seven judgements. <sigh>

>>As I march forward though, I'm having problems with grammar
>>compilation.  I make changes to my grammar and compile it using
>>semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
>>force regeneration, the generated lisp code relating to the rule that
>>I have just updated remains unchanged.
>>
>>If I quit and restart emacs then compilation succeeds.  I did not
>>touch semantic.cache.
>   [ ... ]
>
> That's curious.  I saw something similar while fiddling with idle time
> completion where it looked like the cache was not accepting
> incremental edits.  I hit "C-u M-x bovinate", and everything was
> better.  As I was fiddling in the idle time reparse and errors were
> being thrown, I thought I had merely broken something in that buffer.
> I only saw it the one time.

I tried exactly that as well, but it does not/did not help.

> Does this happen once or every time?  I suspect David would be better
> equipped to answer this question.

Once it begins to happen, it will continue to happen in that instance
of Emacs.  I find that I must quit and restart to get it to stop.

Are there some internal variables I can check the values of for you
to try to debug if/when it happens again?

Joe
-- 
Joseph R. Kiniry                    ID 78860581      ICQ 4344804 
SOS Group, University of Nijmegen  http://www.cs.kun.nl/~kiniry/
KindSoftware, LLC                   http://www.kindsoftware.com/
Board Chair: NICE                    http://www.eiffel-nice.org/

>>> Joseph Kiniry <kiniry@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>>>>> Joseph Kiniry <kiniry@...> seems to think that:
  [ ... ]
>>>The new debug framework that you two dropped in over the past couple
>>>of days is working very well for me.  Now, errors in Lisp are not
>>>completely hidden from me and I can debug more effectively.
>>
>> Huzzah!
>
>Indeed!  Now I'm finally writing my expand-tag function, and its a
>beast!  E.g., a single relativly small datatype declaration like this:
>
>  stack[T: TYPE, U:TYPE+]: DATATYPE 
>     WITH SUBTYPES empty-stack, non-empty-stack
>   BEGIN
>    empty: empty?: empty-stack
>    push(top:T, pop:stack): nonempty?: non-empty-stack
>   END stack
>
>ends up implicitly defining one top-level type, four subtypes, two
>constructors, four boolean functions (recognizers), two accessor
>functions, a datatype function, and seven judgements. <sigh>

Eh?  Humans use this language?  It will be interesting to see how
other semantic tools handle this language when your parser is ready.

>>>As I march forward though, I'm having problems with grammar
>>>compilation.  I make changes to my grammar and compile it using
>>>semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
>>>force regeneration, the generated lisp code relating to the rule that
>>>I have just updated remains unchanged.
>>>
>>>If I quit and restart emacs then compilation succeeds.  I did not
>>>touch semantic.cache.
>>   [ ... ]
>>
>> That's curious.  I saw something similar while fiddling with idle time
>> completion where it looked like the cache was not accepting
>> incremental edits.  I hit "C-u M-x bovinate", and everything was
>> better.  As I was fiddling in the idle time reparse and errors were
>> being thrown, I thought I had merely broken something in that buffer.
>> I only saw it the one time.
>
>I tried exactly that as well, but it does not/did not help.
>
>> Does this happen once or every time?  I suspect David would be better
>> equipped to answer this question.
>
>Once it begins to happen, it will continue to happen in that instance
>of Emacs.  I find that I must quit and restart to get it to stop.
>
>Are there some internal variables I can check the values of for you
>to try to debug if/when it happens again?
  [ ... ]

One quick thing to try is "M-x global-semanticdb-minor-mode" to
disable yet another layer of cache.  I can't think of any other
cached information.

When it does happen, you could put the cursor on the offending bit of
code, and execute `semantic-test-all-format-tag-functions'.  The very
last entry in the list is a dump of the lisp representation.  You
could verify if it is, or is not the same.

If it is the same, there is some cache elsewhere we need to find.  If
it is different, the "C-u M-x bovinate" should fix it.

If it does not, then you could kill the buffer, and find it again.
If that doesn't work, turn off semanticdb, kill the buffer, and find
it again.

That's all I can think of at the moment.
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hello Joseph,

[...]
 > The new debug framework that you two dropped in over the past couple
 > of days is working very well for me.  Now, errors in Lisp are not
 > completely hidden from me and I can debug more effectively.

Nice!

 > As I march forward though, I'm having problems with grammar
 > compilation.  I make changes to my grammar and compile it using
 > semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
 > force regeneration, the generated lisp code relating to the rule that
 > I have just updated remains unchanged.

I have observed that in rare cases, when tags are no more synchronized
with the source.

I didn't find why, but at some point, the idle timer becomes nil, and
auto parse no more occur.  As it is a global timer probably this can
affect the whole Emacs session.

Generally I do this in the current buffer, and it cure the problem for
me:

  M-x global-semantic-idle-scheduler-mode  (turn off idle mode)
  M-x semantic-clear-toplevel-cache        (clear tags in current buffer)
  M-x global-semantic-idle-scheduler-mode  (turn on idle mode again to
                                            recreate the idle timer)

Hope it will help.

David

>>> David Ponce <david.ponce@...> seems to think that:
>Hello Joseph,
>
>[...]
> > The new debug framework that you two dropped in over the past couple
> > of days is working very well for me.  Now, errors in Lisp are not
> > completely hidden from me and I can debug more effectively.
>
>Nice!
>
> > As I march forward though, I'm having problems with grammar
> > compilation.  I make changes to my grammar and compile it using
> > semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
> > force regeneration, the generated lisp code relating to the rule that
> > I have just updated remains unchanged.
>
>I have observed that in rare cases, when tags are no more synchronized
>with the source.
>
>I didn't find why, but at some point, the idle timer becomes nil, and
>auto parse no more occur.  As it is a global timer probably this can
>affect the whole Emacs session.
>
>Generally I do this in the current buffer, and it cure the problem for
>me:
>
>  M-x global-semantic-idle-scheduler-mode  (turn off idle mode)
>  M-x semantic-clear-toplevel-cache        (clear tags in current buffer)
>  M-x global-semantic-idle-scheduler-mode  (turn on idle mode again to
>                                            recreate the idle timer)
  [ ... ]

Does that still happen?  I've recombobulated that idle scheduler
function a few times since you first reported that in hopes of
catching the issue.

I was having problems when there was incomplete syntax, such as an
unterminated list.  It would throw things into a tizzy.  I had fixed
that one issue, however.  (It was a problem with quoting in a symbol
to a macro.)

I have not needed to turn the idle scheduler on/off in a about a
month.  Is that still happening for you?

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

[...]
 > Does that still happen?  I've recombobulated that idle scheduler
 > function a few times since you first reported that in hopes of
 > catching the issue.
 >
 > I was having problems when there was incomplete syntax, such as an
 > unterminated list.  It would throw things into a tizzy.  I had fixed
 > that one issue, however.  (It was a problem with quoting in a symbol
 > to a macro.)
 >
 > I have not needed to turn the idle scheduler on/off in a about a
 > month.  Is that still happening for you?

Yes that still happens sometimes (rarely).  In such cases I noticed
that the idle timer became nil even if the idle mode was enabled.

I must confess that this problem often occurs after I entered the
debugger.  So probably it is due to a bad interaction between `debug'
and the various hooks and timers semantic uses.

Perhaps it is the problem Joseph has had too, as it seems he has often
used the Lisp debugger recently ;-)

David

>>> David Ponce <david.ponce@...> seems to think that:
>[...]
> > Does that still happen?  I've recombobulated that idle scheduler
> > function a few times since you first reported that in hopes of
> > catching the issue.
> >
> > I was having problems when there was incomplete syntax, such as an
> > unterminated list.  It would throw things into a tizzy.  I had fixed
> > that one issue, however.  (It was a problem with quoting in a symbol
> > to a macro.)
> >
> > I have not needed to turn the idle scheduler on/off in a about a
> > month.  Is that still happening for you?
>
>Yes that still happens sometimes (rarely).  In such cases I noticed
>that the idle timer became nil even if the idle mode was enabled.
>
>I must confess that this problem often occurs after I entered the
>debugger.  So probably it is due to a bad interaction between `debug'
>and the various hooks and timers semantic uses.
>
>Perhaps it is the problem Joseph has had too, as it seems he has often
>used the Lisp debugger recently ;-)
  [ ... ]

Do you think a trick like this might be a quick fix?!

Eric

*** semantic-idle.el.~1.17.~	2004-02-18 10:20:07.000000000 -0500
--- semantic-idle.el	2004-02-19 22:03:15.000000000 -0500
***************
*** 238,244 ****
    "Function run when after `semantic-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
  call additional functions registered with the timer calls."
!   (when (semantic-idle-scheduler-enabled-p)
  
      ;; Disable the auto parse timer while re-parsing
      (semantic-idle-scheduler-kill-timer)
--- 238,245 ----
    "Function run when after `semantic-idle-scheduler-idle-time'.
  This function will reparse the current buffer, and if successful,
  call additional functions registered with the timer calls."
!   (when (and (semantic-idle-scheduler-enabled-p)
! 	     (= (recursion-depth) 0))
  
      ;; Disable the auto parse timer while re-parsing
      (semantic-idle-scheduler-kill-timer)

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

Hello David,

David Ponce <david.ponce@...> writes:

> Hello Joseph,
>
> [...]
>  > The new debug framework that you two dropped in over the past couple
>  > of days is working very well for me.  Now, errors in Lisp are not
>  > completely hidden from me and I can debug more effectively.
>
> Nice!
>
>  > As I march forward though, I'm having problems with grammar
>  > compilation.  I make changes to my grammar and compile it using
>  > semantic-grammar-create-package (C-c C-c).  Unfortunately, even if I
>  > force regeneration, the generated lisp code relating to the rule that
>  > I have just updated remains unchanged.
>
> I have observed that in rare cases, when tags are no more synchronized
> with the source.
>
> I didn't find why, but at some point, the idle timer becomes nil, and
> auto parse no more occur.  As it is a global timer probably this can
> affect the whole Emacs session.
>
> Generally I do this in the current buffer, and it cure the problem for
> me:
>
>   M-x global-semantic-idle-scheduler-mode  (turn off idle mode)
>   M-x semantic-clear-toplevel-cache        (clear tags in current buffer)
>   M-x global-semantic-idle-scheduler-mode  (turn on idle mode again to
>                                             recreate the idle timer)
>
> Hope it will help.
>
> David

This situation raised its head again a few minutes ago.  I followed
this sequence of operations and it indeed fixed the problem, this time
without having to restart Emacs.

The next time it happens I will follow Eric's suggested debugging
course of action.

Joe
-- 
Joseph R. Kiniry                    ID 78860581      ICQ 4344804 
SOS Group, University of Nijmegen  http://www.cs.kun.nl/~kiniry/
KindSoftware, LLC                   http://www.kindsoftware.com/
Board Chair: NICE                    http://www.eiffel-nice.org/