CEDET

Michael Reiher <redm@...> writes:

> Hi!
>
> After some time I decided to give it another try to get ecb+semantic in xemacs 
> going. Because I don't want to have to believe, that coding in xemacs is 
> inherently inconvenient:) And ecb+semantic would be a big step forward! If 
> only I would get it into a usable state...
>
> The thing is, it's usable but pretty annoying. The prolems seem all to be 
> semantic related.
>
> First it eats lots of CPU while actually beeing idle. Usually around
> 20% of my Athlon 1600, sometimes even up to 60%! I have a file open,
> and there is just the cursor blinking i.e. I'm not editing or
> anything. So what the hell is it doing?? The number of opened files
> doesn't seem to matter. And the hour glass cursor is constantly
> flashing up.

I've found the same with CVS cedet (and CVS Emacs).  What it seems to
be doing is trying to find header files over and over again.  strace
shows it doing stat64 of header files in each of the directories in
semantic-default-c-path, but apparently not doing anything with those
it finds, and then it loops.  (All this seems to be done while emacs
is idle, since it's interruptible just by interacting with
emacs---very cool.  If only it were cunningly doing useful work!)

I don't know whether that's what 1.0 beta3 is doing.  I'd assumed it
was a bug in the recent semanticdb-find-default-throttle stuff, but
that's not in 1.0-beta3.

[...]

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>Michael Reiher <redm@...> writes:
>
>> Hi!
>>
>> After some time I decided to give it another try to get ecb+semantic in xemacs 
>> going. Because I don't want to have to believe, that coding in xemacs is 
>> inherently inconvenient:) And ecb+semantic would be a big step forward! If 
>> only I would get it into a usable state...
>>
>> The thing is, it's usable but pretty annoying. The prolems seem all to be 
>> semantic related.
>>
>> First it eats lots of CPU while actually beeing idle. Usually around
>> 20% of my Athlon 1600, sometimes even up to 60%! I have a file open,
>> and there is just the cursor blinking i.e. I'm not editing or
>> anything. So what the hell is it doing?? The number of opened files
>> doesn't seem to matter. And the hour glass cursor is constantly
>> flashing up.
>
>I've found the same with CVS cedet (and CVS Emacs).  What it seems to
>be doing is trying to find header files over and over again.  strace
>shows it doing stat64 of header files in each of the directories in
>semantic-default-c-path, but apparently not doing anything with those
>it finds, and then it loops.  (All this seems to be done while emacs
>is idle, since it's interruptible just by interacting with
>emacs---very cool.  If only it were cunningly doing useful work!)
>
>I don't know whether that's what 1.0 beta3 is doing.  I'd assumed it
>was a bug in the recent semanticdb-find-default-throttle stuff, but
>that's not in 1.0-beta3.
  [ ... ]

The addition of the throttle stuff certainly allows semantic to
rummage around in the header file directories more.  beta3 does not do
that.

The idle timer that manages other stuff is part of
`global-semantic-idle-scheduler-mode', so if you turn that off, all
the idle time features are disabled and you should get your cpu back.

The features the scheduler has are:

1) parsing files in background
   - this is useful so that if you request data later, there is no
   waiting.
2) summary mode
   - tries to show you what the symbol you are on is all about
3) completion mode
   - Give you completions in a tooltip.

The parsing in the background feature has a bunch of side-effects,
such as:

* decoration mode
* highlight unmatched syntax
* stickyfunc mode

all those can be enabled/disabled to the point you like.

If semantic is making xemacs busy, it is most likely the idle
completion mode.  Disabling it does not make that feature unavailable,
it just turns off the auto-popup part.  You can always request the
list later with `semantic-complete-analyze-inline' (just bind it to a
key.)  Summary mode is also similarly a bit of busy work.

Why Emacs would call stat a lot, I'm not sure.  It may be that some
combination of system databases and tweaking the include list can
solve that problem.  Those features are pretty new and need some study
to figure out:

  1) What is useful
  2) what is not useful
  3) do useful stuff only.

Basically, it needs lots of folks to not only use it, but spend time
tweaking it to the point where performance and accuracy is good.  We
can then encode those values, and write a good guide on how to get it
set up right.  Ideally it would be fully automated.  That, of course,
is gated by good documentation on the topic which I need to work on.

The missing tags (alluded to earlier) could be an issue with the
parser.  The best tactic there is to enable
`semantic-show-unmatched-syntax-mode' and reparse the file with

 C-u M-x bovinate RET

and see what it can't figure out.  It may be some obscure syntax or
template code is making it drop some tags.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> The addition of the throttle stuff certainly allows semantic to
> rummage around in the header file directories more.  beta3 does not
> do that.
>
> The idle timer that manages other stuff is part of
> `global-semantic-idle-scheduler-mode', so if you turn that off, all
> the idle time features are disabled and you should get your cpu back.
>
> The features the scheduler has are:
>
> 1) parsing files in background
>    - this is useful so that if you request data later, there is no
>    waiting.
> 2) summary mode
>    - tries to show you what the symbol you are on is all about
> 3) completion mode
>    - Give you completions in a tooltip.

That's what I was expecting, yes.  I'm using the default
semanticdb-find-default-throttle (project system recursive), so what I
suspect is happening is that it's trying to find all of these headers
to find out what their full pathnames are, so it can see if they're
already present in a table that it's already loaded.

I guess it might do that each time the idle scheduler kicks in, and
probably it doesn't remember the results from the last time it did it,
so to the naive observer it looks like a spin.  Does that seem
plausible?

[...]

> If semantic is making xemacs busy, it is most likely the idle
> completion mode.  Disabling it does not make that feature
> unavailable, it just turns off the auto-popup part.  You can always
> request the list later with `semantic-complete-analyze-inline' (just
> bind it to a key.)  Summary mode is also similarly a bit of busy
> work.

Cool, I'll try that.  I've never noticed that feature, so I probably
won't notice if it's disabled.

> Why Emacs would call stat a lot, I'm not sure.  It may be that some
> combination of system databases and tweaking the include list can
> solve that problem.  Those features are pretty new and need some
> study to figure out:
>
>   1) What is useful
>   2) what is not useful
>   3) do useful stuff only.
>
> Basically, it needs lots of folks to not only use it, but spend time
> tweaking it to the point where performance and accuracy is good.  We
> can then encode those values, and write a good guide on how to get
> it set up right.  Ideally it would be fully automated.  That, of
> course, is gated by good documentation on the topic which I need to
> work on.

That sounds right.  I strongly suspect most of the problems are simply
because I don't understand the interactions of all these settings, and
some simple example setups would have got me going very quickly.

> The missing tags (alluded to earlier) could be an issue with the
> parser.  The best tactic there is to enable
> `semantic-show-unmatched-syntax-mode' and reparse the file with
>
>  C-u M-x bovinate RET
>
> and see what it can't figure out.  It may be some obscure syntax or
> template code is making it drop some tags.

I'm sure that happens, but I'm not seeing that.  I'm just seeing that
tags in other files (files I haven't visited, but I know have
semantic.cache files in the default save directory) aren't being
found.  I mostly understand why that is, I just don't see how to get
things to work as I'd like.

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>[...]
>
>> The addition of the throttle stuff certainly allows semantic to
>> rummage around in the header file directories more.  beta3 does not
>> do that.
>>
>> The idle timer that manages other stuff is part of
>> `global-semantic-idle-scheduler-mode', so if you turn that off, all
>> the idle time features are disabled and you should get your cpu back.
>>
>> The features the scheduler has are:
>>
>> 1) parsing files in background
>>    - this is useful so that if you request data later, there is no
>>    waiting.
>> 2) summary mode
>>    - tries to show you what the symbol you are on is all about
>> 3) completion mode
>>    - Give you completions in a tooltip.
>
>That's what I was expecting, yes.  I'm using the default
>semanticdb-find-default-throttle (project system recursive), so what I
>suspect is happening is that it's trying to find all of these headers
>to find out what their full pathnames are, so it can see if they're
>already present in a table that it's already loaded.
>
>I guess it might do that each time the idle scheduler kicks in, and
>probably it doesn't remember the results from the last time it did it,
>so to the naive observer it looks like a spin.  Does that seem
>plausible?

That does seem right.  If the stat is really slow, an empty proxy
could be generated to remember that state.  Loading caches, but not
source files might be a better tactic.

Of course, I checked the code, and there should be no stat for
unloaded files.  The logic looks for a dirname in the list of open
databases, and then for a specific table in the list of tables in a
found DB.  Neither pings the disk.

I think the problem occurs when the tag is converted into a file name
on the path in `semantic-dependency-tag-file'.

Perhaps this patch will help:

*** semantic-tag-file.el	29 Apr 2004 22:08:45 -0400	1.8
--- semantic-tag-file.el	08 Feb 2005 23:19:30 -0500	
***************
*** 111,117 ****
         (when (semantic-tag-buffer tag)
           (set-buffer (semantic-tag-buffer tag)))
         (let ((name (semantic-tag-name tag)))
!          (cond ((file-exists-p name)
                  (expand-file-name name))
                 ((and (symbolp semantic-dependency-include-path)
                       (fboundp semantic-dependency-include-path))
--- 111,119 ----
         (when (semantic-tag-buffer tag)
           (set-buffer (semantic-tag-buffer tag)))
         (let ((name (semantic-tag-name tag)))
!          (cond ((semantic--tag-get-property tag 'dependency-file)
! 		(semantic--tag-get-property tag 'dependency-file))
! 	       ((file-exists-p name)
                  (expand-file-name name))
                 ((and (symbolp semantic-dependency-include-path)
                       (fboundp semantic-dependency-include-path))
***************
*** 123,128 ****
--- 125,131 ----
                      (if (file-exists-p (concat (car p) "/" name))
                          (setq found (concat (car p) "/" name)))
                      (setq p (cdr p)))
+ 		  (semantic--tag-put-property tag 'dependency-file found)
                    found))))))))
  
  (make-obsolete-overload 'semantic-find-dependency


As you can see, it will cache the discovered name, which should cut
down on the stats.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> I think the problem occurs when the tag is converted into a file
> name on the path in `semantic-dependency-tag-file'.

Yes, I think that's exactly it.  I have a file which includes
"openssl/ssl.h" or something, and semantic is trying to resolve that
to a full pathname.  Your patch helps, but I fear there are other
places.  For example semanticdb-find-table-for-include-default.  It
may also be necessary to record "not found" information, for those
cases where a header file isn't in any of the paths (otherwise I
suspect it'll do the same set of failed stats next time).

I'm guessing caching this information (using a property as you've
done) is better than multiple stats.  I guess it would depend on
disk/filesystem performance and the length of semantic-default-c-path.
I don't think any of the stats that it's doing for me take very long;
it's just doing lots of them.

Does the idle scheduler (or its tasks) ever decide that nothing's
changed, and return without doing too much work?  I don't notice that
happening, but perhaps I haven't waited long enough.  That strikes me
as something that would make use on single-processor machines more
pleasant (on my dual Xeon machine at work I don't care much if Emacs
is wasting 100% of one hyperthread, but on my Athlon at home that's
annoying).

[...]

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>[...]
>
>> I think the problem occurs when the tag is converted into a file
>> name on the path in `semantic-dependency-tag-file'.
>
>Yes, I think that's exactly it.  I have a file which includes
>"openssl/ssl.h" or something, and semantic is trying to resolve that
>to a full pathname.  Your patch helps, but I fear there are other
>places.  For example semanticdb-find-table-for-include-default.  It
>may also be necessary to record "not found" information, for those
>cases where a header file isn't in any of the paths (otherwise I
>suspect it'll do the same set of failed stats next time).
>
>I'm guessing caching this information (using a property as you've
>done) is better than multiple stats.  I guess it would depend on
>disk/filesystem performance and the length of semantic-default-c-path.
>I don't think any of the stats that it's doing for me take very long;
>it's just doing lots of them.

I've updated the patch (below) for storing the unknown dependency file
also.  It would probably be important to time it.  I suspect that it
is irrelevant for short include paths, and important for long ones.

>Does the idle scheduler (or its tasks) ever decide that nothing's
>changed, and return without doing too much work?  I don't notice that
>happening, but perhaps I haven't waited long enough.  That strikes me
>as something that would make use on single-processor machines more
>pleasant (on my dual Xeon machine at work I don't care much if Emacs
>is wasting 100% of one hyperthread, but on my Athlon at home that's
>annoying).
  [ ... ]

The analyzer (semantic-analyzer.el) will take its analysis result, and
cache it over the calculated region.  See
`semantic-analyze-current-context' help, and
`semantic-cache-data-to-buffer' help for the mechanism.  As these
results are large, they are discarded when the cursor moves out of the
specified symbol region.  Since that is what you spend most of your
time doing while editing, I am not surprised it is not caching much.

Some aspects of what the analyzer calculates could probably be cached
between runs, such as the semanticdb search path, a functions local
variables, and what not.  The key would be finding the most time
consuming tasks, and solving those first.  The data cache mechanism
could be expanded to handle other situations and regions over time.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> "Eric M. Ludlam" <eric@...> seems to think that:
>>>> Bruce Stephens <bruce+cedet@...> seems to think that:
  [ ... ]
>>I'm guessing caching this information (using a property as you've
>>done) is better than multiple stats.  I guess it would depend on
>>disk/filesystem performance and the length of semantic-default-c-path.
>>I don't think any of the stats that it's doing for me take very long;
>>it's just doing lots of them.
>
>I've updated the patch (below) for storing the unknown dependency file
  [ ... ]

I forgot the patch.  Here it is.

Eric

*** semantic-tag-file.el	29 Apr 2004 22:08:45 -0400	1.8
--- semantic-tag-file.el	09 Feb 2005 13:47:12 -0500	
***************
*** 110,129 ****
         ;; TODO: Allow TAG to travel with originating file info.
         (when (semantic-tag-buffer tag)
           (set-buffer (semantic-tag-buffer tag)))
!        (let ((name (semantic-tag-name tag)))
!          (cond ((file-exists-p name)
!                 (expand-file-name name))
!                ((and (symbolp semantic-dependency-include-path)
!                      (fboundp semantic-dependency-include-path))
!                 (funcall semantic-dependency-include-path name))
!                (t
!                 (let ((p semantic-dependency-include-path)
!                       (found nil))
!                   (while (and p (not found))
!                     (if (file-exists-p (concat (car p) "/" name))
!                         (setq found (concat (car p) "/" name)))
!                     (setq p (cdr p)))
!                   found))))))))
  
  (make-obsolete-overload 'semantic-find-dependency
                          'semantic-dependency-tag-file)
--- 110,139 ----
         ;; TODO: Allow TAG to travel with originating file info.
         (when (semantic-tag-buffer tag)
           (set-buffer (semantic-tag-buffer tag)))
!        (let* ((name (semantic-tag-name tag))
! 	      (result
! 	       (cond ((semantic--tag-get-property tag 'dependency-file)
! 		      (semantic--tag-get-property tag 'dependency-file))
! 		     ((file-exists-p name)
! 		      (expand-file-name name))
! 		     ((and (symbolp semantic-dependency-include-path)
! 			   (fboundp semantic-dependency-include-path))
! 		      (funcall semantic-dependency-include-path name))
! 		     (t
! 		      (let ((p semantic-dependency-include-path)
! 			    (found nil))
! 			(while (and p (not found))
! 			  (if (file-exists-p (concat (car p) "/" name))
! 			      (setq found (concat (car p) "/" name)))
! 			  (setq p (cdr p)))
! 			found)))))
! 	 (if (stringp result)
! 	     (progn
! 	       (semantic--tag-put-property tag 'dependency-file result)
! 	       result)
! 	   (semantic--tag-put-property tag 'dependency-file 'none)
! 	   nil))))
!     ))
  
  (make-obsolete-overload 'semantic-find-dependency
                          'semantic-dependency-tag-file)

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> I've updated the patch (below) for storing the unknown dependency
> file also.  It would probably be important to time it.  I suspect
> that it is irrelevant for short include paths, and important for
> long ones.

Yeah, now I come to think of it an unfound file probably indicates a
problem with the include path, so a better response might be to
indicate an error somehow.  (I've no reason to think that that
happened in my configuration: I think all the files were found.)

[...]

> The analyzer (semantic-analyzer.el) will take its analysis result, and
> cache it over the calculated region.  See
> `semantic-analyze-current-context' help, and
> `semantic-cache-data-to-buffer' help for the mechanism.  As these
> results are large, they are discarded when the cursor moves out of the
> specified symbol region.  Since that is what you spend most of your
> time doing while editing, I am not surprised it is not caching much.

That all seems like the right thing to do.  I was thinking of
unselected buffers mostly (if they're being decorated or parsed or
whatever), an idle service could (once it has finished) set a
buffer-local variable containing the value of buffer-modified-tick and
point.  But maybe that's just not where significant work is being
done.  (Someone would need to profile and find out.)

[...]

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
  [ ... ]
>> The analyzer (semantic-analyzer.el) will take its analysis result, and
>> cache it over the calculated region.  See
>> `semantic-analyze-current-context' help, and
>> `semantic-cache-data-to-buffer' help for the mechanism.  As these
>> results are large, they are discarded when the cursor moves out of the
>> specified symbol region.  Since that is what you spend most of your
>> time doing while editing, I am not surprised it is not caching much.
>
>That all seems like the right thing to do.  I was thinking of
>unselected buffers mostly (if they're being decorated or parsed or
>whatever), an idle service could (once it has finished) set a
>buffer-local variable containing the value of buffer-modified-tick and
>point.  But maybe that's just not where significant work is being
>done.  (Someone would need to profile and find out.)
  [ ... ]

The part of the idle scheduler that reparses is a rather clever bit,
if I do say so myself.  First, as you type, semantic tracks the
specific characters that are inserted or deleted.  At idle time, those
changes are collected, and a list of tags that are dirty are
generated, or perhaps a crucial change was made and the whole buffer
needs to be refreshed.  Use `global-semantic-show-parser-state-mode'
to see what happens to buffer state.  Use
`global-semantic-highlight-edits-mode' to watch your edits get
tracked.  Use the option `semantic-edits-verbose-flag' to watch the
incremental parser do it's thing.

Once there is a list of dirty tags, each one is reparsed individually,
saving oodles of time.  For all modes like decoration mode, the subset
of tags is passed in and are redecorated individually via the
link/unlink hooks.  A tag is "linked" into a buffer when the tag data
has an overlay that maps directly to a region in a buffer.

It is my opinion that the modification accounting part has been
optimized to nearly the best extent possible in Lisp code.  One area
that may need some more assistance is dealing with bogus syntax (such
as an unterminated list or string) during an explicit reparse
request. 

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> It is my opinion that the modification accounting part has been
> optimized to nearly the best extent possible in Lisp code.  One area
> that may need some more assistance is dealing with bogus syntax
> (such as an unterminated list or string) during an explicit reparse
> request.

That's matched by my experience: things are much faster now I've
disabled idle-summary and idle-completion.  (So I still have
idle-decoration, which I think is pretty.)  

I've bound completion to a key, but it doesn't seem to work: probably
this is just because (with the default settings) it's doing recursive
searching through header files, and there are just too many of them
for that to be practical.  I'll have to experiment and see what's
going on.

I don't see a suitable function to bind to get the equivalent of the
idle-summary information, which I'm missing.  That looks easy to copy
from semantic-idle.el.

Maybe I could deactivate the idle stuff if Emacs seems idle for too
long (10 seconds or something), only reactivating it when I do
something?  That feels like it might be easy to do (I haven't looked
at the code closely enough to see, of course).

On Wednesday 09 February 2005 00:52, Eric M. Ludlam wrote:
> The features the scheduler has are:
>
> 1) parsing files in background
>    - this is useful so that if you request data later, there is no
>    waiting.
> 2) summary mode
>    - tries to show you what the symbol you are on is all about
> 3) completion mode
>    - Give you completions in a tooltip.
>
I did some experiments today an it looks like that my CPU comsumption prolem 
is connected to the summary mode. Once it starts comsuming heavily CPU, I can 
stop that by disabling semantic-idle-summary-mode. If I reenable it, it also 
starts consuming again. I also can remember seeing some function declarations 
flashing in the minibuffer, permanantly alternating between syntax 
highlighted and plain. What I wonder is: I'm not typing, so shouldn't 
semantic, after updating caches or showing completions, settle down and wait 
for further input? Instead its busy although there is actually nothing to do. 

Can this be connected to the matter you are discussing with Bruce? At least I 
don't see excessive stating. 

>
> The missing tags (alluded to earlier) could be an issue with the
> parser.  The best tactic there is to enable
> `semantic-show-unmatched-syntax-mode' and reparse the file with
>
>  C-u M-x bovinate RET
>
> and see what it can't figure out.  It may be some obscure syntax or
> template code is making it drop some tags.
>
Hmm, if I do that there is nothing happening (It just says "Retrieving tags 
took xxx seconds"). Even if I insert obvious syntax errors. OTOH, if I do 
just M-x bovinate, then I get just:

---------
(("implementedfoo" function
	(:type "int")
	nil #<extent [48, 95)
 semantic 0x8eaed28 in buffer ClassWithExternals.h>)
("foospace" type
(:type "namespace")
nil #<extent [97, 135)
semantic 0x8eaed0c in buffer ClassWithExternals.h>))
----------

Should there be more tags recognized? The file is pretty simple:

----------
#include "ParentClass.h"

void anotherfoo();

int implementedfoo() {
     int d;
     e();
 }

namespace foospace { class fooclass; };

class ClassWithExternals : public ParentClass
{
private:
  int i;

  void blah( int i );

public:
  ClassWithExternals();
  ~ClassWithExternals();
};
---------

Greets 

Michael

Michael Reiher <michael.reiher@...> writes:

[...]

> I did some experiments today an it looks like that my CPU
> comsumption prolem is connected to the summary mode. Once it starts
> comsuming heavily CPU, I can stop that by disabling
> semantic-idle-summary-mode. If I reenable it, it also starts
> consuming again. I also can remember seeing some function
> declarations flashing in the minibuffer, permanantly alternating
> between syntax highlighted and plain.

I don't see that; I see proper eldoc style summaries.

> What I wonder is: I'm not typing, so shouldn't semantic, after
> updating caches or showing completions, settle down and wait for
> further input? Instead its busy although there is actually nothing
> to do.

That's what bothers me.

> Can this be connected to the matter you are discussing with Bruce?

It's the same general area, but I think the code has changed
significantly since 1.0 beta3, so it may be that the details are
different.

> At least I don't see excessive stating.

I'm not sure that that's actually a sign of the excessive CPU usage.
It happened to be easy to spot (whenever I see odd behaviour, I tend
to try running strace to see whether that shows anything odd; not so
much because it often does, but because it's so easy to do), and I
feel it shouldn't be happening (once it's found a file it shouldn't
have to look for it again), but it may well turn out to be
insignificant.

[...]

>>> Michael Reiher <michael.reiher@...> seems to think that:
>On Wednesday 09 February 2005 00:52, Eric M. Ludlam wrote:
>> The features the scheduler has are:
>>
>> 1) parsing files in background
>>    - this is useful so that if you request data later, there is no
>>    waiting.
>> 2) summary mode
>>    - tries to show you what the symbol you are on is all about
>> 3) completion mode
>>    - Give you completions in a tooltip.
>>
>I did some experiments today an it looks like that my CPU comsumption prolem 
>is connected to the summary mode. Once it starts comsuming heavily CPU, I can 
>stop that by disabling semantic-idle-summary-mode. If I reenable it, it also 
>starts consuming again. I also can remember seeing some function declarations 
>flashing in the minibuffer, permanantly alternating between syntax 
>highlighted and plain. What I wonder is: I'm not typing, so shouldn't 
>semantic, after updating caches or showing completions, settle down and wait 
>for further input? Instead its busy although there is actually nothing to do. 
>
>Can this be connected to the matter you are discussing with Bruce? At least I 
>don't see excessive stating. 

The idle summary mode will certainly consume time attempting to lookup
a symbol.  The idle timer used is a repeating timer.  The doc says:

.-------
| If REPEAT is non-nil, do the action each time Emacs has been idle for
| exactly SECS seconds (that is, only once for each time Emacs becomes idle).
.-------

so it should eventually stop doing stuff.

It seems likely that a repeating XEmacs idle timer is different and needs
some alternate arguments.  I'm assuming you are using XEmacs due to
the output you listed below.

One way to verify that it is only called once is something like this:

(add-hook 'semantic-before-idle-scheduler-reparse-hooks
	  (lambda () (message "Idle reparse...")))

(add-hook 'semantic-after-idle-scheduler-reparse-hooks
	  (lambda () (message "Idle reparse...done")))

thus, if you move the cursor, then wait, it should say "..done", and
then not repeat.  The summary mode happens outside that band, so it
won't be measured here.  It seems like some sort of metric gathering
would be useful here.


>> The missing tags (alluded to earlier) could be an issue with the
>> parser.  The best tactic there is to enable
>> `semantic-show-unmatched-syntax-mode' and reparse the file with
>>
>>  C-u M-x bovinate RET
>>
>> and see what it can't figure out.  It may be some obscure syntax or
>> template code is making it drop some tags.
>>
>Hmm, if I do that there is nothing happening (It just says "Retrieving tags 
>took xxx seconds"). Even if I insert obvious syntax errors. OTOH, if I do 
>just M-x bovinate, then I get just:
>
>---------
>(("implementedfoo" function
>	(:type "int")
>	nil #<extent [48, 95)
> semantic 0x8eaed28 in buffer ClassWithExternals.h>)
>("foospace" type
>(:type "namespace")
>nil #<extent [97, 135)
>semantic 0x8eaed0c in buffer ClassWithExternals.h>))
>----------
>
>Should there be more tags recognized? The file is pretty simple:
  [ ... ]

I tried your example, and got a much longer list of tags using GNU
Emacs.  I think that problem may be fixed in CVS.  See this message
from the devel archives.

http://sourceforge.net/mailarchive/message.php?msg_id=9592636

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

On Thursday 10 February 2005 05:13, Eric M. Ludlam wrote:
>
> The idle summary mode will certainly consume time attempting to lookup
> a symbol.  The idle timer used is a repeating timer.  The doc says:
>
> .-------
>
> | If REPEAT is non-nil, do the action each time Emacs has been idle for
> | exactly SECS seconds (that is, only once for each time Emacs becomes
> | idle).
>
> .-------
>
> so it should eventually stop doing stuff.
>
> It seems likely that a repeating XEmacs idle timer is different and needs
> some alternate arguments.  I'm assuming you are using XEmacs due to
> the output you listed below.

Yep.

If that is what you mean, it says here:

--------
(run-with-idle-timer SECS REPEAT FUNCTION &rest ARGS)

Documentation:
Perform an action the next time Emacs is idle for SECS seconds.
If REPEAT is non-nil, do this each time Emacs is idle for SECS seconds.
SECS may be an integer or a floating point number.
The action is to call FUNCTION with arguments ARGS.

This function returns a timer object which you can use in `cancel-timer'.
--------

Seems not so much different after all...

>
> One way to verify that it is only called once is something like this:
>
> (add-hook 'semantic-before-idle-scheduler-reparse-hooks
> 	  (lambda () (message "Idle reparse...")))
>
> (add-hook 'semantic-after-idle-scheduler-reparse-hooks
> 	  (lambda () (message "Idle reparse...done")))
>
> thus, if you move the cursor, then wait, it should say "..done", and
> then not repeat.  The summary mode happens outside that band, so it
> won't be measured here.  It seems like some sort of metric gathering
> would be useful here.
>
Not sure if I understand what you mean here, but the "Idle reparse..." and 
then "done" only appears just one or two times, as it probably should.

> >
> >Should there be more tags recognized? The file is pretty simple:
>
>   [ ... ]
>
> I tried your example, and got a much longer list of tags using GNU
> Emacs.  I think that problem may be fixed in CVS.  See this message
> from the devel archives.
>
> http://sourceforge.net/mailarchive/message.php?msg_id=9592636
>
Ah, cool, I'll try to compile current CVS later today.

Greets 

Michael

On Thursday 10 February 2005 14:14, Michael Reiher wrote:
> >
> > I tried your example, and got a much longer list of tags using GNU
> > Emacs.  I think that problem may be fixed in CVS.  See this message
> > from the devel archives.
> >
> > http://sourceforge.net/mailarchive/message.php?msg_id=9592636
>
> Ah, cool, I'll try to compile current CVS later today.
>
Ok, just tried. But it fails to compile.

First it complains in common at:
xemacs -batch --no-site-file -l autoloads-compile-script -f 
cedet-batch-update-autoloads cedet-loaddefs.el .

It's probably not the right thing to do, but replacing cedet-loaddefs.el with 
cedet-autogen.el made it compile.

But then it break again in semantic:

xemacs -batch --no-site-file -l semantic-compile-script -f batch-byte-compile 
semantic.el semantic-util.el semantic-ctxt.el semantic-lex.el 
semantic-edit.el semantic-fw.el semantic-grammar.el semantic-tag.el 
semantic-find.el semantic-format.el semantic-complete.el semantic-sort.el 
semantic-decorate.el semantic-tag-file.el semantic-doc.el semantic-tag-ls.el 
semantic-ast.el
Loading /usr/share/emacs/site-lisp/debian-startup...
Loading 00debian...
Loading /usr/share/xemacs-21.4.15/lisp/auto-autoloads...
Loading /usr/share/xemacs-21.4.15/lisp/mule/auto-autoloads...
Loading site-start...
Loading 00debian-vars...
Loading 50autoconf...
Loading 50cedet-common...
Loading 50dictionaries-common...
Error while loading 50dictionaries-common
Loading 50eieio...
Loading 51ede...
Loading 51speedbar...
Loading 52semantic...
Loading 52semantic...

[...]

Compiling /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el...
While compiling semantic-lex-debug-break in 
file /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el:
  ** semantic-read-event called with 1 argument, but accepts only 0
While compiling toplevel forms:
  !! Symbol's function definition is void ((semantic-lex-expand-block-specs))
>>Error occurred processing semantic-lex.el:
Symbol's function definition is void: semantic-lex-expand-block-specs

Any idea?

Greets 

Michael

>>> Michael Reiher <redm@...> seems to think that:
>On Thursday 10 February 2005 14:14, Michael Reiher wrote:
  [ ... ]
>It's probably not the right thing to do, but replacing cedet-loaddefs.el with 
>cedet-autogen.el made it compile.
>
>But then it break again in semantic:
>
>xemacs -batch --no-site-file -l semantic-compile-script -f batch-byte-compile 
>semantic.el semantic-util.el semantic-ctxt.el semantic-lex.el 
>semantic-edit.el semantic-fw.el semantic-grammar.el semantic-tag.el 
>semantic-find.el semantic-format.el semantic-complete.el semantic-sort.el 
>semantic-decorate.el semantic-tag-file.el semantic-doc.el semantic-tag-ls.el 
>semantic-ast.el
>Loading /usr/share/emacs/site-lisp/debian-startup...
>Loading 00debian...
>Loading /usr/share/xemacs-21.4.15/lisp/auto-autoloads...
>Loading /usr/share/xemacs-21.4.15/lisp/mule/auto-autoloads...
>Loading site-start...
>Loading 00debian-vars...
>Loading 50autoconf...
>Loading 50cedet-common...
>Loading 50dictionaries-common...
>Error while loading 50dictionaries-common
>Loading 50eieio...
>Loading 51ede...
>Loading 51speedbar...
>Loading 52semantic...
>Loading 52semantic...
>
>[...]
>
>Compiling /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el...
>While compiling semantic-lex-debug-break in 
>file /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el:
>  ** semantic-read-event called with 1 argument, but accepts only 0
>While compiling toplevel forms:
>  !! Symbol's function definition is void ((semantic-lex-expand-block-specs))
>>>Error occurred processing semantic-lex.el:
>Symbol's function definition is void: semantic-lex-expand-block-specs
  [ ... ]

You may have a mix of files built with Emacs and XEmacs.  Try (from
cedet directory):

make bootstrap

and see if that cleans things up.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>[...]
>
>> It is my opinion that the modification accounting part has been
>> optimized to nearly the best extent possible in Lisp code.  One area
>> that may need some more assistance is dealing with bogus syntax
>> (such as an unterminated list or string) during an explicit reparse
>> request.
>
>That's matched by my experience: things are much faster now I've
>disabled idle-summary and idle-completion.  (So I still have
>idle-decoration, which I think is pretty.)  
>
>I've bound completion to a key, but it doesn't seem to work: probably
>this is just because (with the default settings) it's doing recursive
>searching through header files, and there are just too many of them
>for that to be practical.  I'll have to experiment and see what's
>going on.
>
>I don't see a suitable function to bind to get the equivalent of the
>idle-summary information, which I'm missing.  That looks easy to copy
>from semantic-idle.el.

You could try `semantic-ia-show-doc' which is new in CVS.  It is
rougly the same, and shows the prototype and any handy commentary that
may be available.  The semantic-ia functions are all prototypes of
things that eventually get expanded elsewhere, so I'd be interested in
your opinion.

>Maybe I could deactivate the idle stuff if Emacs seems idle for too
>long (10 seconds or something), only reactivating it when I do
>something?  That feels like it might be easy to do (I haven't looked
>at the code closely enough to see, of course).
  [ ... ]

Attached is a patch to semantic-idle.el which adds a verbosity flag.
Set the flag to t, and you can determine what is keeping things so
busy.

There is one new optimization I added while adding this, which is a
new cut-off to not reparse buffers if they don't need it.  The comment
near that addition explains why.

Eric

*** semantic-idle.el	21 Sep 2004 07:52:13 -0400	1.29
--- semantic-idle.el	12 Feb 2005 11:08:51 -0500	
***************
*** 1,6 ****
  ;;; semantic-idle.el --- Schedule parsing tasks in idle time
  
! ;;; Copyright (C) 2003, 2004 Eric M. Ludlam
  
  ;; Author: Eric M. Ludlam <zappo@...>
  ;; Keywords: syntax
--- 1,6 ----
  ;;; semantic-idle.el --- Schedule parsing tasks in idle time
  
! ;;; Copyright (C) 2003, 2004, 2005 Eric M. Ludlam
  
  ;; Author: Eric M. Ludlam <zappo@...>
  ;; Keywords: syntax
***************
*** 49,54 ****
--- 49,60 ----
  (defvar semantic-idle-scheduler-timer nil
    "Timer used to schedule tasks in idle time.")
  
+ (defcustom semantic-idle-scheduler-verbose-flag nil
+   "*Non-nil means that the idle scheduler should provide debug messages.
+ Use this setting to debug idle activities."
+   :group 'semantic
+   :type 'boolean)
+ 
  (defcustom semantic-idle-scheduler-idle-time 2
    "*Time in seconds of idle before scheduling events.
  This time should be short enough to ensure that idle-scheduler will be
***************
*** 216,221 ****
--- 222,229 ----
  (defun semantic-idle-core-handler ()
    "Core idle function that handles reparsing.
  And also manages services that depend on tag values."
+   (when semantic-idle-scheduler-verbose-flag
+     (working-temp-message "IDLE: Core handler..."))
    (semantic-exit-on-input 'idle-timer
      (let* ((inhibit-quit nil)
             (buffers (delq (current-buffer)
***************
*** 246,252 ****
            (save-excursion
              (dolist (service semantic-idle-scheduler-queue)
                (semantic-throw-on-input 'idle-queue)
!               (funcall service)))))
        ;; Finally loop over remaining buffers, trying to update them as
        ;; well.  Stop on keypress.
        (save-excursion
--- 254,265 ----
            (save-excursion
              (dolist (service semantic-idle-scheduler-queue)
                (semantic-throw-on-input 'idle-queue)
! 	      (when semantic-idle-scheduler-verbose-flag
! 		(working-temp-message "IDLE: execture service %s..." service))
!               (funcall service)
! 	      (when semantic-idle-scheduler-verbose-flag
! 		(working-temp-message "IDLE: execture service %s...done" service))
! 	      ))))
        ;; Finally loop over remaining buffers, trying to update them as
        ;; well.  Stop on keypress.
        (save-excursion
***************
*** 255,261 ****
            (with-current-buffer b
              (and (semantic-idle-scheduler-enabled-p)
                   (semantic-idle-scheduler-refresh-tags)))))
!       )))
  
  (defun semantic-idle-scheduler-function ()
    "Function run when after `semantic-idle-scheduler-idle-time'.
--- 268,276 ----
            (with-current-buffer b
              (and (semantic-idle-scheduler-enabled-p)
                   (semantic-idle-scheduler-refresh-tags)))))
!       ))
!   (when semantic-idle-scheduler-verbose-flag
!     (working-temp-message "IDLE: Core handler...done")))
  
  (defun semantic-idle-scheduler-function ()
    "Function run when after `semantic-idle-scheduler-idle-time'.
***************
*** 310,354 ****
  
  Does nothing if the current buffer doesn't need reparsing."
  
!   (let* ((semantic-working-type nil)
! 	 (inhibit-quit nil)
! 	 (working-use-echo-area-p
! 	  (not semantic-idle-scheduler-working-in-modeline-flag))
! 	 (working-status-dynamic-type
! 	  (if semantic-idle-scheduler-no-working-message
! 	      nil
! 	    working-status-dynamic-type))
! 	 (working-status-percentage-type
! 	  (if semantic-idle-scheduler-no-working-message
! 	      nil
! 	    working-status-percentage-type))
! 	 (lexically-safe t)
! 	 )
!     ;; Let people hook into this, but don't let them hose
!     ;; us over!
!     (condition-case nil
! 	(run-hooks 'semantic-before-idle-scheduler-reparse-hooks)
!       (error (setq semantic-before-idle-scheduler-reparse-hooks nil)))
! 
!     (unwind-protect
! 	;; Perform the parsing.
! 	(when (semantic-lex-catch-errors idle-scheduler
! 		(save-excursion (semantic-fetch-tags))
! 		  nil)
! 	  ;; If we are here, it is because the lexical step failed,
! 	  ;; proably due to unterminated lists or something like that.
! 
! 	  ;; We do nothing, and just wait for the next idle timer
! 	  ;; to go off.  In the meantime, remember this, and make sure
! 	  ;; no other idle services can get executed.
! 	  (setq lexically-safe nil))
        ;; Let people hook into this, but don't let them hose
        ;; us over!
        (condition-case nil
! 	  (run-hooks 'semantic-after-idle-scheduler-reparse-hooks)
! 	(error (setq semantic-after-idle-scheduler-reparse-hooks nil))))
!     ;; Return if we are lexically safe
!     lexically-safe))
  
  
  ;;; IDLE SERVICES
--- 325,387 ----
  
  Does nothing if the current buffer doesn't need reparsing."
  
!   ;; These checks actually occur in `semantic-fetch-tags', but if we
!   ;; do them here, then all the bovination hooks are not run, and
!   ;; we save lots of time.
!   (cond 
!    ;; If the buffer was previously marked unparseable,
!    ;; then don't waste our time.
!    ((semantic-parse-tree-unparseable-p)
!     nil)
!    ;; The parse tree is already ok.
!    ((semantic-parse-tree-up-to-date-p)
!     t)
!    (t
!     ;; If these cases are true, then don't do any additional work.
!     (let* ((semantic-working-type nil)
! 	   (inhibit-quit nil)
! 	   (working-use-echo-area-p
! 	    (not semantic-idle-scheduler-working-in-modeline-flag))
! 	   (working-status-dynamic-type
! 	    (if semantic-idle-scheduler-no-working-message
! 		nil
! 	      working-status-dynamic-type))
! 	   (working-status-percentage-type
! 	    (if semantic-idle-scheduler-no-working-message
! 		nil
! 	      working-status-percentage-type))
! 	   (lexically-safe t)
! 	   )
        ;; Let people hook into this, but don't let them hose
        ;; us over!
        (condition-case nil
! 	  (run-hooks 'semantic-before-idle-scheduler-reparse-hooks)
! 	(error (setq semantic-before-idle-scheduler-reparse-hooks nil)))
! 
!       (unwind-protect
! 	  ;; Perform the parsing.
! 	  (progn
! 	    (when semantic-idle-scheduler-verbose-flag
! 	      (working-temp-message "IDLE: reparse %s..." (buffer-name)))
! 	    (when (semantic-lex-catch-errors idle-scheduler
! 		    (save-excursion (semantic-fetch-tags))
! 		    nil)
! 	      ;; If we are here, it is because the lexical step failed,
! 	      ;; proably due to unterminated lists or something like that.
! 	    
! 	      ;; We do nothing, and just wait for the next idle timer
! 	      ;; to go off.  In the meantime, remember this, and make sure
! 	      ;; no other idle services can get executed.
! 	      (setq lexically-safe nil))
! 	    (when semantic-idle-scheduler-verbose-flag
! 	      (working-temp-message "IDLE: reparse %s...done" (buffer-name))))
! 	;; Let people hook into this, but don't let them hose
! 	;; us over!
! 	(condition-case nil
! 	    (run-hooks 'semantic-after-idle-scheduler-reparse-hooks)
! 	  (error (setq semantic-after-idle-scheduler-reparse-hooks nil))))
!       ;; Return if we are lexically safe
!       lexically-safe))))
  
  
  ;;; IDLE SERVICES

On Saturday 12 February 2005 16:23, Eric M. Ludlam wrote:
> >Compiling
> > /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el...
> > While compiling semantic-lex-debug-break in
> >file /home/michael/Scripts/site-lisp/cedet-cvs/semantic/semantic-lex.el:
> >  ** semantic-read-event called with 1 argument, but accepts only 0
> >While compiling toplevel forms:
> >  !! Symbol's function definition is void
> > ((semantic-lex-expand-block-specs))
> >
> >>>Error occurred processing semantic-lex.el:
> >
> >Symbol's function definition is void: semantic-lex-expand-block-specs
>
>   [ ... ]
>
> You may have a mix of files built with Emacs and XEmacs.  Try (from
> cedet directory):
>
> make bootstrap
>
> and see if that cleans things up.
>

Hmm, no, that didn't make any difference. I also tried fresh cvs checkout, but 
same error. Also note, that I don't have Emacs installed here, just XEmacs 
21.4.15 (Debian Sarge).

My elisp knowledge is somewhat limited, but what I wonder is that the function 
semantic-lex-expand-block-specs seems actually defined in this file?

Anything else I could try?

Greets
Michael