CEDET

Barry OReilly writes:
> Hi, if I open a .h file and go to a function declaration and execute
> semantic-analyze-proto-impl-toggle , I find that it is not able to find the
> implementation if Emacs has not visited it yet.  For example, if I open only
> Util.h then it won't find the function implementation in Util.cc.  But if I
> open Util.h and Util.cc in Emacs, then it does.
>
> I imagine it's because I haven't indicated to EDE a way to find .cc files.
>
> The directory structure is like this:
>    (common-sub-root)/inc/Util.h
>    (common-sub-root)/src/Util.cc
> How may I customize my EDE project to know where to find .cc files?  I've
> observed semantic-analyze-proto-impl-toggle does not lead to a call to
> ede-expand-filename-impl which calls locate-fcn .

The easiest way is to use GNU Global. It will tell Semantic where it can
find symbols of this name, and it will then visit those symbols and
check if it is the correct definition to your declaration.

For using GNU Global, create the tag database in the root of your
project. Then use

(semanticdb-enable-gnu-global-databases 'c-mode)
(semanticdb-enable-gnu-global-databases 'c++-mode)

(setq ede-locate-setup-options
        '(ede-locate-global
	  ede-locate-base))     

-David

> The easiest way is to use GNU Global. It will tell Semantic where it can
> find symbols of this name, and it will then visit those symbols and
> check if it is the correct definition to your declaration.

I downloaded latest GNU Global (6.2.8), ran gtags at the root, opened my
Util.h only, issued semantic-analyze-proto-impl-toggle for myUtilFunc,
waited for 26 minutes, and then Semantic successfully opened Util.cc and
placed me at myUtilFunc's definition.  To keep whatever caches there are
warm, I kept the same Emacs session open, including the Util.h and Util.cc
buffers, went to the same declaration of myUtilFunc in Util.h, and issued
semantic-analyze-proto-impl-toggle again.  This caused another 26 minute
wait.  During the command, Emacs visits many .cc files, which are in same
the source tree but have nothing to do with my Util class.

I executed 'global myUtilFunc' at the command line and that returns very
quickly with two .cc files, one of which is the correct one.  Another class
has a function of the same name.

Here's the top lines of an elp profile of the
semantic-analyze-proto-impl-toggle command:

semantic-analyze-proto-impl-toggle                          1
1591.447626   1591.447626
semantic-analyze-tag-references                             1
1591.444366   1591.444366
semantic-analyze-tag-references-default                     1
1591.444355   1591.444355
semantic--analyze-refs-full-lookup                          1
1591.078446   1591.078446
semantic--analyze-refs-full-lookup-with-parents             1
1591.078438   1591.078438
semanticdb-find-tags-collector                              3
1590.857997   530.28599900
semanticdb-find-tags-by-name-method                         61
1590.700132   26.077051344
semanticdb-deep-find-tags-by-name-method                    119
1589.770119   13.359412764
semantic-symref-result-get-tags                             2
1589.478069   794.7390345
semantic-find-file-noselect                                 1238
1542.5541070  1.2460049329
semantic-new-buffer-fcn                                     1240
121.49295800  0.0979781919
semanticdb-semantic-init-hook-fcn                           1240
118.70507000  0.0957298951
semantic--set-buffer-cache                                  1240
88.824982000  0.0716330500
semanticdb-synchronize-table                                2480
88.813512000  0.0358119000
semanticdb-refresh-references                               1241
88.650469     0.0714347050
semanticdb-add-reference                                    8515
88.556084999  0.0104000099
semanticdb-find-table-for-include                           8532
88.441814000  0.0103658947
semanticdb-find-table-for-include-c-mode                    8532
88.244368000  0.0103427529
semanticdb-find-table-for-include-default                   8532
88.193447999  0.0103367848
semanticdb-find-load-unloaded                               8329
46.279124000  0.0055563841
semanticdb-file-table-object                                8329
46.189997999  0.0055456835
semanticdb-create-database                                  2258
41.165646999  0.0182310217
semantic-dependency-tag-file                                7550
40.495623000  0.0053636586
semanticdb-create-table-for-file                            2029
36.623327999  0.0180499398
semanticdb-file-loaded-p                                    2258
30.506237999  0.0135102914
semanticdb-directory-loaded-p                               1913
26.149703000  0.0136694736
semantic--tag-unlink-from-buffer                            31971
14.830222999  0.0004638648
semantic--tag-link-to-buffer                                32046
14.077294000  0.0004392839
semanticdb-get-database                                     229
6.6853550000  0.0291936899
semanticdb-load-database                                    360
6.6024469999  0.0183401305
semanticdb-kill-hook                                        2844
4.7509360000  0.0016705119
semantic--tag-unlink-cache-from-buffer                      1238
4.6449330000  0.0037519652
semantic-tag-components-with-overlays                       64017
4.4400929999  6.935...e-05
semantic--tag-link-cache-to-buffer                          1240
4.3657529999  0.0035207685
semantic-tag-components-with-overlays-default               64017
3.8343440000  5.989...e-05
semanticdb-enable-gnu-global-hook                           1240
2.4876780000  0.0020061919
semanticdb-enable-gnu-global-in-buffer                      1240
2.4800890000  0.0020000717
semantic-tag-components                                     94026
2.0972709999  2.230...e-05
semanticdb-cache-filename                                   2258
2.0192759999  0.0008942763
semanticdb-file-name-directory                              2258
1.8109210000  0.0008020022

On 03/25/2013 06:03 PM, Barry OReilly wrote:
>  > The easiest way is to use GNU Global. It will tell Semantic where it can
>  > find symbols of this name, and it will then visit those symbols and
>  > check if it is the correct definition to your declaration.
>
> I downloaded latest GNU Global (6.2.8), ran gtags at the root, opened my
> Util.h only, issued semantic-analyze-proto-impl-toggle for myUtilFunc,
> waited for 26 minutes, and then Semantic successfully opened Util.cc and
> placed me at myUtilFunc's definition.  To keep whatever caches there are
> warm, I kept the same Emacs session open, including the Util.h and
> Util.cc buffers, went to the same declaration of myUtilFunc in Util.h,
> and issued semantic-analyze-proto-impl-toggle again.  This caused
> another 26 minute wait.  During the command, Emacs visits many .cc
> files, which are in same the source tree but have nothing to do with my
> Util class.
>
> I executed 'global myUtilFunc' at the command line and that returns very
> quickly with two .cc files, one of which is the correct one.  Another
> class has a function of the same name.
>
> Here's the top lines of an elp profile of the
> semantic-analyze-proto-impl-toggle command:
>
> semantic-analyze-proto-impl-toggle                          1
> 1591.447626   1591.447626
> semantic-analyze-tag-references                             1
> 1591.444366   1591.444366
> semantic-analyze-tag-references-default                     1
> 1591.444355   1591.444355
> semantic--analyze-refs-full-lookup                          1
> 1591.078446   1591.078446
> semantic--analyze-refs-full-lookup-with-parents             1
> 1591.078438   1591.078438
> semanticdb-find-tags-collector                              3
> 1590.857997   530.28599900
> semanticdb-find-tags-by-name-method                         61
> 1590.700132   26.077051344
> semanticdb-deep-find-tags-by-name-method                    119
> 1589.770119   13.359412764
> semantic-symref-result-get-tags                             2
> 1589.478069   794.7390345

Well, here's a problem.  When adding GNU Global, when it is called via 
semanticdb-deep-find-tags-by-name, it has to convert all those hits into 
tags.  The only way it can do that is to load the file into a buffer, 
put the cursor at the hit location, and extract the tag.

The problem is that code changes out from under GNU Global, so as you 
edit code, the line numbers change, so we need to snoop around to find 
the tag you're looking for.

I'm guessing we could have a fast mode that doesn't do that, and just 
'trusts' that if it finds a tag by name around some line number, that is 
good enough.

Of course, why are there 1238 hits from global?  I'm guessing from your 
explanation above that the parent of myUtilFunc is some generic word 
like "std" or something.  In order for the impl-proto jump to work, it 
has to find the correct parent of the symbol you want to toggle on on 
the off chance that your tag is in a bunch of different names 
spaces/classes.

The assumption, of course, is that you might have oodles of hits on the 
tag name, and less on the parent name, but your case is the opposite.

A quick hack might be to visit the function 
semantic--analyze-refs-full-lookup and force the if statement to only do 
the simple lookup.  If it "works" and is fast, then we'll know we're on 
the right track, and perhaps that code should do the simple search 
first, see if it is successful, then try the slower version.

> semantic-find-file-noselect                                 1238
> 1542.5541070  1.2460049329

Yep, 1238 file hits.  That's a lot of files to open.  Twice.

Not sure about the 26 minutes though.  That's just crazy.

Eric

> The problem is that code changes out from under GNU Global, so as you
> edit code, the line numbers change, so we need to snoop around to find
> the tag you're looking for.

Well, if I get the feature working reasonably, I'll probably use it when
editing code.

Why go through the trouble of snooping around the code when GNU Global's
incremental updating is fast?  Can't CEDET run 'global -u' and then assume
line number accuracy?

> Of course, why are there 1238 hits from global?  I'm guessing from your
> explanation above that the parent of myUtilFunc is some generic word
> like "std" or something.  In order for the impl-proto jump to work, it
> has to find the correct parent of the symbol you want to toggle on on
> the off chance that your tag is in a bunch of different names
> spaces/classes.

It's not in the std namespace.  The structure is like:
   namespace A { namespace B { namespace C { myUtilFunc declaration } } }

CEDET visited .cc files with no members in the B or C namespace.  Most of
the project is in the A namespace, so I'm not sure if it would open files
without a namespace A.

> The assumption, of course, is that you might have oodles of hits on the
> tag name, and less on the parent name, but your case is the opposite.

Yes, that's true.  I would think this would be true for most projects,
since functions are named very particularly and namespaces typically have
many members.  Thus, shouldn't the default behavior be to get the matches
for the tag first and then use parents to narrow that set of results?

> A quick hack might be to visit the function
> semantic--analyze-refs-full-lookup and force the if statement to only do
> the simple lookup.  If it "works" and is fast, then we'll know we're on
> the right track, and perhaps that code should do the simple search
> first, see if it is successful, then try the slower version.

Well, it was reasonably fast at about one second, but is now incorrect.  It
took me to the other definition of a myUtilFunc rather than the correct one.

On 03/26/2013 10:24 AM, Barry OReilly wrote:
>  > The problem is that code changes out from under GNU Global, so as you
>  > edit code, the line numbers change, so we need to snoop around to find
>  > the tag you're looking for.
>
> Well, if I get the feature working reasonably, I'll probably use it when
> editing code.
>
> Why go through the trouble of snooping around the code when GNU Global's
> incremental updating is fast?  Can't CEDET run 'global -u' and then
> assume line number accuracy?

That seems like a good idea.  I had avoided that in case the user had 
some custom way to call global on their project.  If that seems 
unlikely, I could make it the default and have an option to disable it.

I spent some time looking into if it is possible to skip the full file 
load step and I think doing so will take a bit of time as the entire 
search feature will need to be re-written, and I'll need to figure out 
how to test the two flavors, since there are cases where the files need 
to load.

David is also asking about tag linking when it isn't necessary.  That is 
closely related, and skipping the tag linking will speed things up too.

Eric

On 03/26/2013 07:50 AM, David Engster wrote:
> What I've seen quite often in ELP results from larger parsing workloads
> is that semantic--tag-link and -unlink-from-buffer can take quite some
> time, since they are called for every tag. I usually see roughly about
> 15% of time spend with this during parsing.
>
> I can understand why we would have to link tags to overlays in open
> buffers, but why for files which are opened, parsed and then immediately
> closed again? Couldn't we skip this somehow in this case?

That sounds like a good idea.

The function semanticdb-create-table-for-file-not-in-buffer seems like a 
logical starting point for experimentation, but semantic-fetch-tags 
seems like the wrong place to add such trickery for it to use.  It is 
probably ok to just call semantic-parse-region directly.  Complexity 
goes up from there though since semanticdb next depends on the cached 
tags being linked by overlays.  There would be a ripple effect of tweaks 
needed unless large chunks of the regular infrastructure is skipped 
outright in semanticdb-create-table-for-file-not-in-buffer.

The next puzzle would be using the -not-in-buffer function more 
aggressively.  I think it is underutilized due to the convenience (and 
shorter name of) semanticdb-file-table-object.

Anyway, a neat idea, but I expect some trickiness to it.

Eric

Wooohooo! Thank you Pierre! I'll give it a try during next long
weekend (next week)

On Fri, Mar 22, 2013 at 3:03 PM, kototama kototama <kototamo@...> wrote:
> Hello,
>
> I gave a first try at fully parsing the Clojure project definitions. I
> build an AST from the semantic tags, this seems to be fast enough and
> in general project definitions are small.
>
> First I wanted to use just plain lists, no tags, since they contain
> less information, to represent the AST but then I was stuck since
> EXPANDFULL works with tags. Is there some equivalent that works with
> list? That is, how can I call several time a rule to build a list of
> lists?
>
> Otherwise it seems to work correctly on the few examples I tried. I
> implemented a function that should make Alex happy:
>
> wisent-clojure-project-dependencies
>
> "Returns a list of lists representing the project dependencies."
>
>
> Regards,
> Pierre



-- 
With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)
Skype: alex.ott

P.S: the code is here:

https://github.com/kototama/clojure-semantic

the function is defined in clojure-ast.el

On Fri, Mar 22, 2013 at 3:03 PM, kototama kototama <kototamo@...> wrote:
> Hello,
>
> I gave a first try at fully parsing the Clojure project definitions. I
> build an AST from the semantic tags, this seems to be fast enough and
> in general project definitions are small.
>
> First I wanted to use just plain lists, no tags, since they contain
> less information, to represent the AST but then I was stuck since
> EXPANDFULL works with tags. Is there some equivalent that works with
> list? That is, how can I call several time a rule to build a list of
> lists?
>
> Otherwise it seems to work correctly on the few examples I tried. I
> implemented a function that should make Alex happy:
>
> wisent-clojure-project-dependencies
>
> "Returns a list of lists representing the project dependencies."
>
>
> Regards,
> Pierre

kototama kototama writes:
> I gave a first try at fully parsing the Clojure project definitions. I
> build an AST from the semantic tags, this seems to be fast enough and
> in general project definitions are small.
>
> First I wanted to use just plain lists, no tags, since they contain
> less information, to represent the AST but then I was stuck since
> EXPANDFULL works with tags. Is there some equivalent that works with
> list? That is, how can I call several time a rule to build a list of
> lists?

This might be completely off the mark, but you could take a look at the
Wisent C grammar in the 'obsolete' directory of CEDET trunk. It's in

obsolete/lisp/cedet/semantic/wisent/c.wy and c.el.

David Ponce builds ASTs there, using the package semantic/ast.el which
is currently still in trunk. I frankly never had the time to really
understand what exactly he's doing there.

-David

> You pass... what? Are those seriously 420 *separate* include
> directories, meaning do you pass those 420 directories to the compiler
> via "-I"? Does that even fit on the command line?

That's the number of include directories in the whole project, including UT
and integration test code.  A given invocation of g++ has a subset.  The
compilation of the .cc in question has 19 -I flags.

When I do:
   rm /tmp/inc-list
   for arg in `find <project-root> -name "*.h"` ; do dirname $arg >>
/tmp/inc-list ;done
   cat /tmp/inc-list | sort | uniq | wc -l
I get 428

> If you even on repeated jumps to the same location get the same
> performance, then that's it.

To clarify, if I do it again in the same session it's fast.  It's closing
and opening Emacs again that a second attempt is slow.

> But you did byte-compile CEDET, did you?

Yes, I byte compiled CEDET.  I also moved the semanticdb location out of my
NFS-hosted home directory to a localhost directory.  I also tried removing
my c-mode hooks, since those are not byte compiled.  That did not improve
performance.

> I remember testing out Boost a while ago though, and it was pretty much
> OK. Could you tell me which headers from Boost you're using?

I created a dependency file using g++ -MMD -MT flags and found 1166 Boost
dependency headers.  (We normally exclude Boost headers from our auto
dependency files because they significantly slowed down the initialization
of make.)

> Could you look how much memory your Emacs process occupies?

I ran 'top' while I did the semantic-ia-fast-jump.  Memory usage didn't go
above 1.2%.

> You could take a look with the ELP profiler what is taking so long.

Thanks for your guidance.

Looking at the output, it appears to be a matter of
semanticdb-find-table-for-include* functions being called for each include
(1238 is pretty close to 1166 Boost headers + 63 source tree headers).  I
suppose cutting down on the 420 include paths wouldn't help since the
performance seems to be a function of the number of files actually
included.  Does that sound correct to you?

semantic-ia-fast-jump                                       1
54.430523     54.430523
semantic-analyze-current-context                            1
53.919461     53.919461
semantic-analyze-current-context-default                    1
53.919371     53.919371
semantic-calculate-scope                                    1
53.917867     53.917867
semantic-analyze-scoped-types                               1
53.904351     53.904351
semantic-analyze-scoped-types-default                       1
53.904343     53.904343
semanticdb-find-translate-path                              2
37.265932     18.632966
semanticdb-find-translate-path-default                      2
37.265915     18.6329575
semanticdb-find-translate-path-includes-default             2
37.265886     18.632943
semantic-ctxt-scoped-types                                  1
37.182581     37.182581
semantic-ctxt-scoped-types-c++-mode                         1
37.182572     37.182572
semanticdb-find-tags-by-class                               1
37.182231     37.182231
semanticdb-find-tags-collector                              1
37.182226     37.182226
semanticdb-find-translate-path-includes--internal           1
37.122164     37.122164
semanticdb-find-table-for-include                           1238
36.202804999  0.0292429765
semanticdb-find-table-for-include-c-mode                    1238
36.193572000  0.0292355185
semanticdb-find-table-for-include-default                   1238
36.186072999  0.0292294612
semantic-dependency-tag-file                                1193
31.026644999  0.0260072464
semantic-sort-tags-by-name-then-type-increasing             3310
13.898089999  0.0041988187
semanticdb-find-load-unloaded                               1186
4.8754469999  0.0041108322
semanticdb-file-table-object                                1186
4.849719      0.0040891391
semanticdb-find-load-unloaded-default                       1164
4.8460080000  0.0041632371
semantic-tag-lessp-name-then-type                           1464132
3.5792250000  2.444...e-06
semantic-dependency-find-file-on-path                       1180
2.6593979999  0.0022537271
semanticdb-get-database                                     131
2.432285      0.0185670610
semanticdb-create-database                                  134
2.2880920000  0.0170753134
semanticdb-load-database                                    129
1.9242220000  0.0149164496
semanticdb-create-table-for-file-not-in-buffer              4
0.6269579999  0.1567394999
semantic-find-file-noselect                                 4
0.617115      0.15427875
semantic-tag-copy                                           7799
0.5927260000  7.600...e-05
semantic-ia--fast-jump-helper                               1
0.511037      0.511037
semantic-something-to-tag-table                             5536
0.3624840000  6.547...e-05
semanticdb-find-tags-by-class-method                        2325
0.3351900000  0.0001441677
semanticdb-directory-loaded-p                               1143
0.2964010000  0.0002593184
semantic-find-tags-included                                 1163
0.2802740000  0.0002409922
semantic-find-tags-included-c-mode                          1163
0.2713890000  0.0002333525
semantic-tag-file-name                                      16898
0.2195980000  1.299...e-05
semanticdb-refresh-table                                    3508
0.1942109999  5.536...e-05
semanticdb-in-buffer-p                                      5881
0.1812609999  3.082...e-05
semanticdb-cache-filename                                   134
0.1676630000  0.0012512164
semanticdb-file-name-directory                              134
0.1620419999  0.0012092686
semanticdb-file-table                                       1146
0.1560520000  0.0001361710
semanticdb-table                                            1408
0.1548770000  0.0001099978
semantic-go-to-tag                                          1
0.149445      0.149445
semantic-tag-in-buffer-p                                    18135
0.1443660000  7.960...e-06
semanticdb-full-filename                                    4330
0.1235819999  2.854...e-05
semanticdb-file-loaded-p                                    134
0.1184639999  0.0008840597
semanticdb-find-need-cache-update-p                         2
0.116726      0.058363
semanticdb-get-table-index                                  1183
0.1166690000  9.862...e-05
semanticdb-find-incomplete-cache-entries-p                  2
0.116533      0.0582665
semantic--tag-put-property                                  15991
0.1156299999  7.230...e-06
semanticdb-needs-refresh-p                                  1161
0.1093860000  9.421...e-05
semanticdb-abstract-table-child-p                           2453
0.1019999999  4.158...e-05
semantic-tag-of-type-p                                      47130
0.0988480000  2.097...e-06
semantic-tag-include-filename                               2431
0.0965359999  3.971...e-05
semanticdb-file-table-object-put-hash                       1149
0.0847220000  7.373...e-05
semantic-tag-include-filename-default                       2431
0.0783770000  3.224...e-05
semanticdb-find-search-index                                1163
0.0747090000  6.423...e-05
semanticdb-equivalent-mode-for-search                       1162
0.0689760000  5.935...e-05
semantic-tag-faux-p                                         52790
0.0675790000  1.280...e-06
semanticdb-refresh-references                               4
0.0608010000  0.0152002500
semanticdb-add-reference                                    25
0.060522      0.00242088
semantic-tag-with-position-p                                9169
0.0527819999  5.756...e-06
semantic-new-buffer-fcn                                     5
0.048553      0.0097106
semanticdb-semantic-init-hook-fcn                           3
0.0481959999  0.0160653333
semantic-overlay-p                                          48977
0.0472740000  9.652...e-07
semantic--tag-link-to-buffer                                91
0.0410510000  0.0004511098
semanticdb-equivalent-mode                                  1162
0.0358810000  3.087...e-05
semantic--set-buffer-cache                                  3
0.034814      0.0116046666
semanticdb-synchronize-table                                6
0.0347900000  0.0057983333
semantic-tag-make-plist                                     18111
0.0330449999  1.824...e-06
semanticdb-get-tags                                         3498
0.0312509999  8.933...e-06
semantic--tag-unlink-from-buffer                            73
0.0297309999  0.0004072739
semantic-tag-prototype-p                                    3643
0.0294490000  8.083...e-06
semanticdb-project-database-file                            129
0.0274409999  0.0002127209
semanticdb-table-p                                          1162
0.0212240000  1.826...e-05
semanticdb-abstract-table-list-p                            129
0.0159929999  0.0001239767
semantic-tag                                                2120
0.0158770000  7.489...e-06
semanticdb-table-child-p                                    1163
0.0131949999  1.134...e-05
semanticdb-abstract-search-index-child-p                    1163
0.0129030000  1.109...e-05
semantic--tag-link-cache-to-buffer                          3
0.008203      0.0027343333
semantic-tag-components-with-overlays                       164
0.0080019999  4.879...e-05
semanticdb-find-throttle-active-p                           4874
0.0068840000  1.412...e-06
semanticdb-kill-hook                                        148
0.0060930000  4.116...e-05
semantic-tag-components-with-overlays-default               164
0.0058460000  3.564...e-05
semantic--tag-unlink-cache-from-buffer                      2
0.0056869999  0.0028434999
semantic-analyze-scoped-tags                                1
0.005408      0.005408
semantic-analyze-scoped-tags-default                        1
0.0054        0.0054
semantic-tag-prototype-p-default                            3643
0.0052740000  1.447...e-06
semanticdb-create-table-for-file                            3
0.004982      0.0016606666
semantic-idle-scheduler-function                            6
0.0032009999  0.0005334999
semantic-idle-core-handler                                  5
0.003149      0.0006298
semantic-tag-components                                     243
0.0030909999  1.272...e-05
semanticdb-file-table-object-from-hash                      1186
0.0030629999  2.582...e-06
semantic-analyze-scoped-type-parts                          4
0.002043      0.00051075
semanticdb-file-name-non-directory                          134
0.0019589999  1.461...e-05
semanticdb-get-database-tables                              129
0.001842      1.427...e-05
semantic-tag-components-c-mode                              243
0.0016879999  6.946...e-06
semanticdb-minor-mode-p                                     1285
0.0016569999  1.289...e-06
semantic-analyze-scope-nested-tags                          1
0.001587      0.001587
semantic-analyze-scope-nested-tags-default                  1
0.001579      0.001579
semanticdb-set-buffer                                       27
0.0009169999  3.396...e-05
semantic-tag-buffer                                         39
0.0007500000  1.923...e-05
semantic-analyze-find-tag-sequence                          1
0.000681      0.000681
semantic-analyze-find-tag-sequence-default                  1
0.000673      0.000673
semanticdb-get-buffer                                       27
0.0006640000  2.459...e-05
semanticdb-find-results-p                                   61
0.0005570000  9.131...e-06
semantic-up-context                                         7
0.000463      6.614...e-05
semantic-analyze-scope-calculate-access                     4
0.000452      0.000113
semantic-analyze-scope-calculate-access-default             4
0.0004250000  0.0001062500
semantic-find-first-tag-by-name                             14
0.0003890000  2.778...e-05
semantic-ctxt-current-assignment                            1
0.000387      0.000387
semantic-ctxt-current-assignment-default                    1
0.000378      0.000378
semantic--tag-run-hooks                                     180
0.0003619999  2.011...e-06
semantic-tag-components-default                             239
0.0003370000  1.410...e-06
semantic-get-local-variables                                2
0.000322      0.000161
semantic-narrow-to-context                                  2
0.000308      0.000154
semantic-idle-scheduler-refresh-tags                        32
0.0003079999  9.624...e-06
semantic-get-local-variables-c++-mode                       2
0.000301      0.0001505
semantic-tag-similar-p                                      9
0.0002890000  3.211...e-05
semantic-scope-cache                                        5
0.000283      5.66e-05
semantic-get-all-local-variables                            1
0.000281      0.000281
semantic-find-tag-by-overlay                                15
0.0002750000  1.833...e-05
semantic-get-all-local-variables-default                    1
0.000273      0.000273
semanticdb-cache-get                                        4
0.000272      6.8e-05
semantic-overlay-live-p                                     251
0.0002439999  9.721...e-07
semantic-get-local-arguments                                2
0.00024       0.00012
semantic-tag-similar-p-default                              9
0.000229      2.544...e-05
semantic-scope-reset-cache                                  3
0.0002239999  7.466...e-05
semantic-get-local-variables-default                        2
0.000212      0.000106
semanticdb-notify-references                                1
0.000197      0.000197
semantic-analyze-scope-lineage-tags                         2
0.00019       9.5e-05
semantic-beginning-of-command                               1
0.000182      0.000182
semanticdb-create-table                                     3
0.00018       6e-05
semantic-analyze-scope-lineage-tags-default                 2
0.000175      8.75e-05
semantic-beginning-of-command-default                       1
0.000174      0.000174
semantic-end-of-command                                     1
0.00017       0.00017
semantic-ctxt-current-symbol-and-bounds                     1
0.00017       0.00017
semantic-overlay-buffer                                     172
0.0001669999  9.709...e-07
semantic-end-of-command-default                             1
0.000163      0.000163
semantic-ctxt-current-symbol-and-bounds-default             1
0.000161      0.000161
semantic-current-tag-of-class                               7
0.0001559999  2.228...e-05
semantic-ctxt-current-symbol                                1
0.000145      0.000145
semantic-beginning-of-context                               2
0.000144      7.2e-05
semantic-end-of-context                                     2
0.000143      7.15e-05
semantic-current-tag                                        6
0.00014       2.333...e-05
semantic-narrow-to-tag                                      7
0.0001370000  1.957...e-05
semantic-ctxt-current-symbol-default                        1
0.000137      0.000137
semantic-beginning-of-context-default                       2
0.000132      6.6e-05
semantic-end-of-context-default                             2
0.00013       6.5e-05
semantic-make-overlay                                       92
0.0001249999  1.358...e-06
semantic-clear-toplevel-cache                               3
0.000114      3.8e-05
semanticdb-includes-in-table                                4
0.000108      2.7e-05
semantic-overlay-delete                                     74
0.0001049999  1.418...e-06
semantic-overlay-put                                        94
0.0001019999  1.085...e-06
semantic-analyze-context                                    1
9.8e-05       9.8e-05
semantic-analyze-type-constants-c-mode                      78
9.499...e-05  1.217...e-06
semantic-overlay-end                                        96
9.199...e-05  9.583...e-07
semantic-overlay-start                                      96
9.099...e-05  9.479...e-07
semantic-completable-tags-from-type                         4
9e-05         2.25e-05
semantic-analyze-scoped-inherited-tag-map                   10
8.700...e-05  8.700...e-06
semantic--tag-attribute-similar-p                           6
8.7e-05       1.45e-05
semantic-ctxt-current-function                              1
8.5e-05       8.5e-05
semantic--tag-similar-names-p                               6
7.9e-05       1.316...e-05
semantic-decorate-flush-pending-decorations                 32
7.899...e-05  2.468...e-06
semantic-ctxt-current-function-default                      1
7.7e-05       7.7e-05
semantic-default-c-setup                                    3
7.3e-05       2.433...e-05
semantic-analyze-split-name                                 4
6.8e-05       1.7e-05
semanticdb-set-dirty                                        3
6.500...e-05  2.166...e-05
semantic-scope-tag-clone-with-scope                         5
6.3e-05       1.26e-05
semantic-fetch-tags                                         34
5.999...e-05  1.764...e-06
semantic-get-local-arguments-default                        2
5.7e-05       2.85e-05
semantic-get-cache-data                                     3
5.6e-05       1.866...e-05
semantic-up-context-default                                 7
5.200...e-05  7.428...e-06
semantic-analyze-scoped-inherited-tags                      4
5.2e-05       1.3e-05
semantic-find-tags-by-scope-protection                      4
5.100...e-05  1.275...e-05
semantic--tag-attribute-similar-p-c-mode                    6
4.6e-05       7.666...e-06
semantic-lex-init                                           3
4.499...e-05  1.499...e-05
semantic-scope-cache-p                                      4
4.2e-05       1.05e-05
semantic--tag-similar-names-p-c-mode                        6
4.099...e-05  6.833...e-06
semantic-ctxt-current-class-list                            1
4e-05         4e-05
semantic-lex-spp-set-dynamic-table                          3
3.9e-05       1.3e-05
semantic-analyze-split-name-c-mode                          4
3.8e-05       9.5e-06
semantic-overlay-get                                        37
3.7e-05       1e-06
semantic-idle-scheduler-mode                                1
3.2e-05       3.2e-05
semantic-ctxt-current-class-list-default                    1
3.2e-05       3.2e-05
semantic-overlays-at                                        19
3.100...e-05  1.631...e-06
semantic-c-by--install-parser                               3
2.600...e-05  8.666...e-06
semantic-cache-data-to-buffer                               1
2.4e-05       2.4e-05
semantic-cache-data-post-command-hook                       1
2.4e-05       2.4e-05
semantic-tag-resolve-proxy                                  15
2.300...e-05  1.533...e-06
semantic-find-tags-by-scope-protection-c-mode               4
2.2e-05       5.5e-06
semantic-tag-type-superclasses                              10
2e-05         2.000...e-06
semantic-lex-spp-symbol-set                                 3
1.999...e-05  6.666...e-06
semantic--tag-attribute-similar-p-default                   6
1.800...e-05  3.000...e-06
semanticdb-expand-nested-tag-c++-mode                       14
1.800...e-05  1.285...e-06
semantic-clear-unmatched-syntax-cache                       3
1.300...e-05  4.333...e-06
semantic-scope-cache-child-p                                1
1.1e-05       1.1e-05
semantic-clear-parser-warnings                              3
9e-06         3e-06
semantic--tag-similar-names-p-default                       6
8e-06         1.333...e-06
semantic-lex-spp-save-table                                 3
7e-06         2.333...e-06
semantic-set-unmatched-syntax-cache                         3
6e-06         2e-06
semanticdb-find-result-nth                                  3
6e-06         2e-06
semantic-fetch-tags-fast                                    2
4e-06         2e-06
semantic-overlay-lists                                      3
4e-06         1.333...e-06
semanticdb-find-log-activity                                2
4e-06         2e-06
semantic-tag-name                                           3
3e-06         1e-06
semantic-idle-scheduler-setup-timers                        1
3e-06         3e-06
semanticdb-find-result-length                               1
3e-06         3e-06
semantic-equivalent-tag-p                                   1
2e-06         2e-06
semantic-refresh-tags-safe                                  2
2e-06         1e-06
semantic-analyze-dereference-alias                          1
2e-06         2e-06
semanticdb-find-log-new-search                              1
1e-06         1e-06

On 03/19/2013 11:06 AM, Barry OReilly wrote:
>  > You pass... what? Are those seriously 420 *separate* include
>  > directories, meaning do you pass those 420 directories to the compiler
>  > via "-I"? Does that even fit on the command line?
>
> That's the number of include directories in the whole project, including
> UT and integration test code.  A given invocation of g++ has a subset.
> The compilation of the .cc in question has 19 -I flags.

I had similar problems where I work.  I ended up writing a custom EDE 
project to cut back on the # of include paths needed for a given source, 
since different areas had different sets of includes.  Even so, for me 
if I pulled in sources from too many modules, Emacs would eventually 
load so much data, the machine spent all its time swapping.

Writing a custom EDE project is not for everyone though. ;)

For your case, you might be able to tag subsets of your project as 
independent projects instead.  That depends on if they cross-ref each other.

Lastly, another idea is to implement a :locate-fcn for your 
cpp-root-project that knows which includes belong to which subsections 
of your project.  In that way, it will only load in database caches for 
include directories that matter, instead of for the entire project.

>  > You could take a look with the ELP profiler what is taking so long.
>
> Thanks for your guidance.
>
> Looking at the output, it appears to be a matter of
> semanticdb-find-table-for-include* functions being called for each
> include (1238 is pretty close to 1166 Boost headers + 63 source tree
> headers).  I suppose cutting down on the 420 include paths wouldn't help
> since the performance seems to be a function of the number of files
> actually included.  Does that sound correct to you?
>
> semantic-ia-fast-jump                                       1

[ ... ]

> semanticdb-find-table-for-include                           1238
> 36.202804999  0.0292429765
> semanticdb-find-table-for-include-c-mode                    1238
> 36.193572000  0.0292355185
> semanticdb-find-table-for-include-default                   1238
> 36.186072999  0.0292294612
> semantic-dependency-tag-file                                1193
> 31.026644999  0.0260072464

This shows that it is snooping around your file system for include 
files.  Once found, the path is cached in the tag and doesn't need to be 
looked up a second time.

For this case, shrinking the include path will help.  It could also help 
if you add a ':locate-fcn' as describe above which is somehow smarter 
about identifying where your files are than a simple rummage around the FS.

> semantic-sort-tags-by-name-then-type-increasing             3310
> 13.898089999  0.0041988187

My first thought is that this was part of the typecache which is built 
up for your first completion, but I don't see the rest of the API in the 
ELP output.  This is another complex data structure that isn't saved, 
and is a huge database of all your types.  It gets rebuilt regularly, 
but secondary builds are usually best-case scenario with most of the 
list already sorted.

For this case, tuning the project to minimize the number of include 
files needed will help.

For example, if your files include "unecessary_defs.hh", you might 
choose to remove that from your include path if you never need to access 
those names via semantic.  My system at work had this monster-sized 
util.h which had mostly fcns in it, and a bunch of crazy data types I 
never used which weren't really needed for completing stuff in the code 
I worked on, so I could skip loading it.

For my system at work, it used to take 8 minutes the first time for a 
completion, and I managed to get it down to something pretty reasonable, 
and instantaneous the second time.  In this case, it's just big so it 
will take effort to tune things to minimize the scope.

Eric

Barry OReilly writes:
>> You pass... what? Are those seriously 420 *separate* include
>> directories, meaning do you pass those 420 directories to the compiler
>> via "-I"? Does that even fit on the command line?
>
> That's the number of include directories in the whole project, including UT and
> integration test code.  A given invocation of g++ has a subset.  The
> compilation of the .cc in question has 19 -I flags.

Just to be clear: if the include file names are relative, like

#include "../inc/something.h"

then Semantic will find them and you don't have to explicitly pass it to
the project.

>> If you even on repeated jumps to the same location get the same
>> performance, then that's it.
>
> To clarify, if I do it again in the same session it's fast.  It's closing and
> opening Emacs again that a second attempt is slow.

Oh, I see. Well, then never close Emacs. ;-)

Actually, hearing that Semantic is "fast" once it has a warm cache on
such a large-scale project is pretty good news.

>> But you did byte-compile CEDET, did you?
>
> Yes, I byte compiled CEDET.  I also moved the semanticdb location out of my
> NFS-hosted home directory to a localhost directory.  I also tried removing my
> c-mode hooks, since those are not byte compiled.  That did not improve
> performance.

The actual loading of the files is only ~10%, so there's not much to
gain here:

semanticdb-find-load-unloaded                               1186       
4.8754469999  0.0041108322

(That's still pretty slow, though. Loading 1200 small files shouldn't
take 5 seconds. How large is your ~/.emacs.d/semanticdb directory?).

>> Could you look how much memory your Emacs process occupies?
>
> I ran 'top' while I did the semantic-ia-fast-jump.  Memory usage didn't go
> above 1.2%.

Unless you have a Petabyte RAM, that's good news. :-)

>> You could take a look with the ELP profiler what is taking so long.
>
> Looking at the output, it appears to be a matter of
> semanticdb-find-table-for-include* functions being called for each include
> (1238 is pretty close to 1166 Boost headers + 63 source tree headers).  I
> suppose cutting down on the 420 include paths wouldn't help since the
> performance seems to be a function of the number of files actually included. 
> Does that sound correct to you?

> semanticdb-find-table-for-include-default                   1238       
> 36.186072999  0.0292294612
> semantic-dependency-tag-file                                1193       
> 31.026644999  0.0260072464

Unless you're working on an pretty old machine, that should be way
faster. Like Eric, I think that those hundreds of include paths are
slowing things down considerably.

To verify, I created a test file including a whole bunch of Boost
libraries. gcc says it has ~870 header dependencies, so that's not too
shabby. If I let Emacs parse the necessary headers and then restart it,
jumping results in the following:

semanticdb-find-table-for-include-default                   958         2.2813617300  0.0023813796
semanticdb-find-load-unloaded                               958         1.3286687080  0.0013869193
semanticdb-find-load-unloaded-default                       924         1.3213180550  0.0014299978
semanticdb-file-table-object                                958         1.3197434490  0.0013776027
semantic-tag-lessp-name-then-type                           1199175     1.0901007759  9.090...e-07
semantic-dependency-tag-file                                972         0.9946550759  0.0010233076

This is without any project definition (Boost is installed as a system
library). You see that things are much faster, and most of the time is
actually spend loading the tag data. After they are in memory, the time
for jumping is roughly cut down by half.

So I'd suggest a bottom-up approach: start with a minimal subproject and
work your way up.

-David

Profiling showed that the reduced performance is due to
semanticdb-find-table-for-include* functions getting pounded even though
taking an average of 30ms, so I took a closer look at their
implementation.  When I saw that evaluations are skipped over based on
semanticdb-find-default-throttle values, I decided to try removing system
from the throttle.  I didn't try this sooner because I thought a nil value
for :system-include-path would be sufficient.  However, this improved
performance to about a second or two.

Giving up Boost headers in exchange for a speedup from 45s to 2s is
certainly worth it, so this solution will work for the time being.  Thanks
for your help!

Thanks for your reply David.  My emails go through moderator approval, so
they may appear out of sequence relative to yours.

> (That's still pretty slow, though. Loading 1200 small files shouldn't
> take 5 seconds. How large is your ~/.emacs.d/semanticdb directory?).

'du -sh' currently shows 51M for the semanticdb directory.  It may have
increased in size in the last couple days.  I think I remember it being 41M
yesterday.

The Boost headers account for 24788 KiB, determined from
   du -s `ls | grep boost` | sed 's/\t.*//g' | xargs python -c 'import sys;
print(sum(map(int, sys.argv[1:])))'

A similar command indicates 17504 KiB for files within the project source
tree.

> To verify, I created a test file including a whole bunch of Boost
> libraries. gcc says it has ~870 header dependencies, so that's not too
> shabby.

You are certainly getting better performance out of
semanticdb-find-table-for-include-default.

Mine:  semanticdb-find-table-for-include-default
1238        36.186072999  0.0292294612
Yours: semanticdb-find-table-for-include-default
958         2.2813617300  0.0023813796

What was the value of your semanticdb-find-default-throttle for the run?

On 03/20/2013 03:12 PM, Barry OReilly wrote:
> Thanks for your reply David.  My emails go through moderator approval,
> so they may appear out of sequence relative to yours.

Sorry about being slow.  Being a human spam filter is a bit of a pain, 
but it's better than having all that junk in the main list.

Eric

> Lastly, another idea is to implement a :locate-fcn for your
> cpp-root-project that knows which includes belong to which subsections
> of your project.  In that way, it will only load in database caches for
> include directories that matter, instead of for the entire project.

I can reduce the include directories considerably by mapping the location
in the source tree to a list of include paths.  For example, suppose
Util.cc uses a struct Packet.h, and I'm executing semantic-ia-fast-jump
from inside Util.cc and expect to be taken to Packet.h .  I can write the
Elisp that if I'm given the path to Util.cc within the project, I can
return a smaller list of include paths for EDE to use which will contain
Packet.h , if I know how to hook that into EDE.

I looked into locate-fcn as you recommended, in cpp-root.el, and see that
it would call it with
   name=Packet.h
   dir=(project root)

These input parameters don't give me the information I need: the path to
Util.cc.  Bizarrely, the value of buffer-file-name at this point is my
Util.h, not Util.cc .

> Sorry about being slow.  Being a human spam filter is a bit of a pain,
> but it's better than having all that junk in the main list.

Is there a way to white list me?

On 03/21/2013 10:25 AM, Barry OReilly wrote:
>  > Lastly, another idea is to implement a :locate-fcn for your
>  > cpp-root-project that knows which includes belong to which subsections
>  > of your project.  In that way, it will only load in database caches for
>  > include directories that matter, instead of for the entire project.
>
> I can reduce the include directories considerably by mapping the
> location in the source tree to a list of include paths.  For example,
> suppose Util.cc uses a struct Packet.h, and I'm executing
> semantic-ia-fast-jump from inside Util.cc and expect to be taken to
> Packet.h .  I can write the Elisp that if I'm given the path to Util.cc
> within the project, I can return a smaller list of include paths for EDE
> to use which will contain Packet.h , if I know how to hook that into EDE.
>
> I looked into locate-fcn as you recommended, in cpp-root.el, and see
> that it would call it with
>     name=Packet.h
>     dir=(project root)
>
> These input parameters don't give me the information I need: the path to
> Util.cc.  Bizarrely, the value of buffer-file-name at this point is my
> Util.h, not Util.cc .

Hmmm, I didn't expect the current buffer to change immediately, but it 
will change along the way.  For example, if file A.cpp includes B.h, and 
B.h includes C.h, then B.h should be current when looking for C.h.

I also see ede/cpp-root's impl for ede-expand-filename-impl is calling 
(call-next-method) before the locate fcn, so perhaps something funny is 
going on in there.

It is probably safe to remove that first call.  It's there to allow you 
to use an external locate routine, such as GNU Global, and it's done 
again at the end anyway.

>  > Sorry about being slow.  Being a human spam filter is a bit of a pain,
>  > but it's better than having all that junk in the main list.
>
> Is there a way to white list me?

Uh, sure.  Didn't think of that.  You could also join the mailing list. 
  Either cedet-semantic or cedet-devel.

Eric

On 03/18/2013 08:11 PM, Jai Dayal wrote:
> HI,
>
>   I am trying to jump to the definition of a struct, but I get this error:
>
> eieio-class-precedence-list: Variable binding depth exceeds
> max-specpdl-size [2 times]
>
> I am using Ott's latest CEDET and I am using EDE cpp projects.  Any idea
> what I'm doing wrong?

Hi,

To find out, do:

M-x toggle-debug-on-error RET

and reproduce the failure.  The resulting stack will help identify the 
issue.

Run the command again to make it stop showing stack traces all the time.

Eric