Thread: [CEDET-devel] support for aliases through using statements
Brought to you by:
zappo
From: David E. <de...@ra...> - 2009-08-23 17:15:57
Attachments:
using-ns-patch.diff
|
Hi Eric, first of all, thanks for the work on the "fully qualified names in the parent class" problem. It works fine for me now. So back to the original problem. ;-) Before you dive into old threads: the problem was basically this construction namespace foo { struct aStruct {}; } namespace bar { using foo::aStruct; } which makes bar::aStruct an alias for foo::aStruct. I attached a patch which tries to resolve this construction through an additional dereferencer. The code got more complicated than I thought, since it has to deal with all kinds of cases: fully qualified and unqualified names, aliases of aliases, 'using' vs 'using namespace', and so on. I've now also added support for dealing with a proper namespace alias like namespace fooalias = foo; which is now parsed through the latest changes in c.by which you've done. Needless to say, you can do all kinds of funky stuff with this kind of aliasing in C++. I've written a bunch of new unit tests which at least try to cover the basic stuff, and I hope together with the comments they make clear what the code tries to resolve. The attached code kind of evolved in the last weeks, so it is probably in need of some refactoring, but before I invest more time, I thought it would be better if you could take a look and see if I'm on the right track. Regards, -David |
From: Eric M. L. <er...@si...> - 2009-08-25 02:56:59
|
Hi David, Thanks for keeping your patch up to date. Sadly, I am confused here, and partly because I don't remember how this stuff works very well, and I don't think it should involve the large number of search commands you need to run to make it function. Let me ask this question. In this example: namespace bar { typedef foo::aStruct aStruct; } is that roughly the same as your example: namespace bar { using foo::aStruct; } ? A search like this: (semanticdb-typecache-find "bar::aStruct") would find the typedef (first case) easily and ask for that to be dereference. It would miss the second, and this is where I'm confused about what is being dereferenced, and I wish I had a moment to spare to try it out today. Instead, I can let you contemplate the thought of identifying key patterns, like that using statement, and converting it into some sort of meta tag in the typecache. That would simplify your C specific code here in favor of some override method provided by the typecache to do the translation. The translations would happen once per file changed, and get re-used often. This new info would be available only via restricted use cases, so it may be too narrow. I can investigate more when I finish the current lex bug I'm working on. Sorry for being so persnickety with your patches. I don't know if you were on the mailing list back when I did all the performance enhancements for the smart completion engine. I got a smart completion example of mine down from 8 minutes ( I thought it was hung ) to around a second, and it was by eliminating searches similar to these in favor of these misc caches and scopes. Thanks! Eric On Sun, 2009-08-23 at 19:15 +0200, David Engster wrote: > Hi Eric, > > first of all, thanks for the work on the "fully qualified names in the > parent class" problem. It works fine for me now. > > So back to the original problem. ;-) > > Before you dive into old threads: the problem was basically this > construction > > namespace foo { > struct aStruct {}; > } > > namespace bar { > using foo::aStruct; > } > > which makes bar::aStruct an alias for foo::aStruct. > > I attached a patch which tries to resolve this construction through an > additional dereferencer. The code got more complicated than I thought, > since it has to deal with all kinds of cases: fully qualified and > unqualified names, aliases of aliases, 'using' vs 'using namespace', and > so on. I've now also added support for dealing with a proper namespace > alias like > > namespace fooalias = foo; > > which is now parsed through the latest changes in c.by which you've > done. > > Needless to say, you can do all kinds of funky stuff with this kind of > aliasing in C++. I've written a bunch of new unit tests which at least > try to cover the basic stuff, and I hope together with the comments they > make clear what the code tries to resolve. > > The attached code kind of evolved in the last weeks, so it is probably > in need of some refactoring, but before I invest more time, I thought it > would be better if you could take a look and see if I'm on the right > track. |
From: David E. <de...@ra...> - 2009-08-25 09:47:44
|
Eric M. Ludlam <er...@si...> writes: > Thanks for keeping your patch up to date. Thank git for that; it makes creating and merging branches much simpler. > Sadly, I am confused here, and partly because I don't remember how this > stuff works very well, and I don't think it should involve the large > number of search commands you need to run to make it function. Yes. I'm not too happy with the amount of searching, either. > Let me ask this question. In this example: > > namespace bar { > typedef foo::aStruct aStruct; > } > > is that roughly the same as your example: > > namespace bar { > using foo::aStruct; > } > > ? Roughly, yes. It's C++, so I'm sure there's some subtle difference, probably regarding symbol lookup. But I don't think that matters much here. If code really relies on stuff like that, it's doomed anyway. ;-) > A search like this: > > (semanticdb-typecache-find "bar::aStruct") > > would find the typedef (first case) easily and ask for that to be > dereference. It would miss the second, and this is where I'm confused > about what is being dereferenced, and I wish I had a moment to spare to > try it out today. > > Instead, I can let you contemplate the thought of identifying key > patterns, like that using statement, and converting it into some sort of > meta tag in the typecache. That would simplify your C specific code > here in favor of some override method provided by the typecache to do > the translation. The translations would happen once per file changed, > and get re-used often. This new info would be available only via > restricted use cases, so it may be too narrow. That's a good idea. Where would I introduce such a 'metatype'? The problem I have here is that they are not parsed as such, like 'typedef', so I would have to somehow create them when I encounter the above construction. Should this be done in the dereferencing-step, or before? How do you think one should best handle a complete namespace alias, like namespace foo { using namespace bar; } or namespace foo = bar; Currently, I'm searching the types in that namespace. I don't think creating a metatype for every type in foo would be more efficient (just consider something like 'namespace stdalias = std;'). I think this has to be done only once for each type, since after that they are in the typecache? > Sorry for being so persnickety with your patches. Don't worry about it. I was already a bit concerned regarding speed. > I don't know if you were on the mailing list back when I did all the > performance enhancements for the smart completion engine. I got a > smart completion example of mine down from 8 minutes ( I thought it > was hung ) to around a second, and it was by eliminating searches > similar to these in favor of these misc caches and scopes. Yes, I do remember that. I can imagine that these semanticdb-searches could slow things down considerably for large codebases. -David |
From: Eric M. L. <er...@si...> - 2009-08-25 22:59:49
|
On Tue, 2009-08-25 at 11:47 +0200, David Engster wrote: > Eric M. Ludlam <er...@si...> writes: > > > > Instead, I can let you contemplate the thought of identifying key > > patterns, like that using statement, and converting it into some sort of > > meta tag in the typecache. That would simplify your C specific code > > here in favor of some override method provided by the typecache to do > > the translation. The translations would happen once per file changed, > > and get re-used often. This new info would be available only via > > restricted use cases, so it may be too narrow. > > That's a good idea. Where would I introduce such a 'metatype'? The > problem I have here is that they are not parsed as such, like 'typedef', > so I would have to somehow create them when I encounter the above > construction. Should this be done in the dereferencing-step, or before? > > How do you think one should best handle a complete namespace alias, like > > namespace foo { > using namespace bar; > } > > or > > namespace foo = bar; > > Currently, I'm searching the types in that namespace. I don't think > creating a metatype for every type in foo would be more efficient (just > consider something like 'namespace stdalias = std;'). I think this has > to be done only once for each type, since after that they are in the > typecache? Personally, I find the typecache code to be a little tricky due to the many different caches that are created and recycled in different ways, all in the name of recycling as much work as possible. The idea is that every file has a typecache. The cache object (at the top of semanticdb-typecache.el) has a "filestream", which is this file's types converted into the unified typecache model, and an "includestream", which is all the files this file includes. When calculating the includestream, it asks each included file for it's filestream, and merges all those things together. Thus, if you look in this method: (defmethod semanticdb-typecache-file-tags ((table semanticdb-table)) "Update the typecache for TABLE, and return the file-tags. File-tags are those that belong to this file only, and excludes all included files." (let* (;(idx (semanticdb-get-table-index table)) (cache (semanticdb-get-typecache table)) ) ;; Make sure our file-tags list is up to date. (when (not (oref cache filestream)) (let ((tags (semantic-find-tags-by-class 'type table))) (when tags (setq tags (semanticdb-typecache-safe-tag-list tags table)) (oset cache filestream (semanticdb-typecache-merge-streams tags nil))))) ;; Return our cache. (oref cache filestream) )) you see the function call `semantic-find-tags-by-class' which is applied to the tags in the specified buffer's tags table (derived from the sematnicdb table.) My first guess for augmenting this is to add an override method like `semanticdb-typecache-find-tags-for-typecache', whose default implementation is as written above. Then, in C/C++, it would also do that line, but also scan through that list looking for things to convert into some sort of alias tag. These alias tags would then be used in the completion engine in the simplified way that typedefs are. (cross fingers.) As for what kind of tag to make, you could make up something completely new, such as converting this: namespace bar { class beep; } namespace foo { using namespace bar; } like this: (semantic-tag "bar" 'metatype :parent "foo") which if I have the translation right, means that foo::beep can convert into bar::beep? Hmmm. Now I'm not sure how that will work. I wonder if semanticdb-typecache-find needs to somehow identify the above proposed 'metatype, and perform the alias operation itself in the middle of the name. That seems weird. Eric |
From: David E. <de...@ra...> - 2009-08-28 16:14:13
|
Eric M. Ludlam <er...@si...> writes: > Thus, if you look in this method: > > (defmethod semanticdb-typecache-file-tags ((table semanticdb-table)) > "Update the typecache for TABLE, and return the file-tags. > File-tags are those that belong to this file only, and excludes > all included files." > (let* (;(idx (semanticdb-get-table-index table)) > (cache (semanticdb-get-typecache table)) > ) > > ;; Make sure our file-tags list is up to date. > (when (not (oref cache filestream)) > (let ((tags (semantic-find-tags-by-class 'type table))) > (when tags > (setq tags (semanticdb-typecache-safe-tag-list tags table)) > (oset cache filestream (semanticdb-typecache-merge-streams tags > nil))))) > > ;; Return our cache. > (oref cache filestream) > )) > > you see the function call `semantic-find-tags-by-class' which is applied > to the tags in the specified buffer's tags table (derived from the > sematnicdb table.) My first guess for augmenting this is to add an > override method like `semanticdb-typecache-find-tags-for-typecache', > whose default implementation is as written above. I'm confused. :-) Where should this new override semanticdb-typecache-find-tags-for-typecache be called? > Then, in C/C++, it would also do that line, but also scan through that > list looking for things to convert into some sort of alias tag. These > alias tags would then be used in the completion engine in the simplified > way that typedefs are. (cross fingers.) This would still need a dereferencer, right? And this one would still have to look for TYPE in the aliased namespace? (Which could also be an alias.) I'm not sure if that really would be faster, but maybe I'm just not understanding it. To (hopefully) make things clearer, here's what my code does. Let's take namespace foo { using namespace bar; } When you ask for 'foo::TYPE': 1) Find namespace 'foo' (which can be scattered over several files) 2) Search for all 'using' statements in that namespace. -> We find 'using namespace bar;' 3) Go to step 0), but this time search for 'bar::TYPE' 4) If we find nothing take next 'using' statement. -> Do until TYPE is found, or until there are no more using statements. If I knew beforehand that 'foo' is an alias for 'bar', I could the skip the search for using statements, but isn't that pretty fast? I don't know. I would think that searching for TYPE in the namespace is what really could be slow, but I don't see how to avoid that. BTW, couldn't I create these metatypes also in the dereferencer, after step 2? Then I would only search for using statements the first time a type is looked up in an aliased namespace. -David |
From: Eric M. L. <er...@si...> - 2009-08-29 02:23:38
|
On Fri, 2009-08-28 at 18:13 +0200, David Engster wrote: > Eric M. Ludlam <er...@si...> writes: > > Thus, if you look in this method: > > > > (defmethod semanticdb-typecache-file-tags ((table semanticdb-table)) > > "Update the typecache for TABLE, and return the file-tags. > > File-tags are those that belong to this file only, and excludes > > all included files." > > (let* (;(idx (semanticdb-get-table-index table)) > > (cache (semanticdb-get-typecache table)) > > ) > > > > ;; Make sure our file-tags list is up to date. > > (when (not (oref cache filestream)) > > (let ((tags (semantic-find-tags-by-class 'type table))) > > (when tags > > (setq tags (semanticdb-typecache-safe-tag-list tags table)) > > (oset cache filestream (semanticdb-typecache-merge-streams tags > > nil))))) > > > > ;; Return our cache. > > (oref cache filestream) > > )) > > > > you see the function call `semantic-find-tags-by-class' which is applied > > to the tags in the specified buffer's tags table (derived from the > > sematnicdb table.) My first guess for augmenting this is to add an > > override method like `semanticdb-typecache-find-tags-for-typecache', > > whose default implementation is as written above. > > I'm confused. :-) > > Where should this new override > semanticdb-typecache-find-tags-for-typecache be called? > > > Then, in C/C++, it would also do that line, but also scan through that > > list looking for things to convert into some sort of alias tag. These > > alias tags would then be used in the completion engine in the simplified > > way that typedefs are. (cross fingers.) > > This would still need a dereferencer, right? And this one would still > have to look for TYPE in the aliased namespace? (Which could also be an > alias.) > > I'm not sure if that really would be faster, but maybe I'm just not > understanding it. > > To (hopefully) make things clearer, here's what my code does. Let's take > > namespace foo { > using namespace bar; > } > > When you ask for 'foo::TYPE': > > 1) Find namespace 'foo' (which can be scattered over several files) > 2) Search for all 'using' statements in that namespace. > -> We find 'using namespace bar;' > 3) Go to step 0), but this time search for 'bar::TYPE' > 4) If we find nothing take next 'using' statement. > > -> Do until TYPE is found, or until there are no more using statements. > > If I knew beforehand that 'foo' is an alias for 'bar', I could the skip > the search for using statements, but isn't that pretty fast? I don't > know. I would think that searching for TYPE in the namespace is what > really could be slow, but I don't see how to avoid that. > > BTW, couldn't I create these metatypes also in the dereferencer, after > step 2? Then I would only search for using statements the first time a > type is looked up in an aliased namespace. Hi David, I think I may be missing something from my understanding of the problem. Let me try and sum-up my previous email in a different way. I'm starting with the premise that we want to avoid the searches in a dereferencer. I'm sure the thing you wrote works fine, I'm just optimizing here. In fact, I'm only trying to optimize a single line, which was a global search across all tables. A search is basically a linear scan of all the tags. Not a big deal with small projects. More problematic for a large number of tables. The typecache happens to be a tool that already performs a linear scan of all the tags across all tables, but it tackles the problem via divide/concur/cache mechanism, which makes it faster. The question is then, can we leave special tags behind in the typecache during the scan? I'm not entirely sure after reading your mail. I was hoping something like a typedef could be left behind while building the cache. We would then need a dereferencer for the metatag, but that dereferencer would look just like the typedef one which has no searches in it. The trick is that if you find one of your special namespace entries, it would leave behind something with a name that matches what would be searched for. This tag would be found, and then the dereferences would just return a new name of whatever it referenced. What makes me uncertain if this will work now is that this only works for a fully qualified type. If I understand the problem right, your aliases are for names in the middle of a fully qualified name. As such, I think I may change my mind on a possible solution. Perhaps the typecache instead collect an alias table. The dereferencer could then just scan the table, instead of scanning the entire universe. Perhaps a simple alias table is needed for both the typecache, and the scope. I'm not sure how it all works out. Or perhaps this is a premature optimization, and I'm just speculating. Using ELP in the past has usually been the best thing. You could do before/after runs of the unit test suite, and see what the delta is. That would be easy. Thanks Eric |
From: David E. <de...@ra...> - 2009-08-29 10:38:38
|
Eric M. Ludlam <er...@si...> writes: > I'm starting with the premise that we want to avoid the searches in a > dereferencer. I'm sure the thing you wrote works fine, I'm just > optimizing here. In fact, I'm only trying to optimize a single line, > which was a global search across all tables. I'm not sure which line you are referring to. There is (semanticdb-find-tags-by-name name) in semantic-c-check-type-namespace-alias, and (semanticdb-find-tags-by-name (car typename)) in semantic-c-dereference-type-alias. Both do a search for the aliased namespace. Then, there's (semantic-deep-find-tags-by-name typename result) which searches for the type in the namespace. > The question is then, can we leave special tags behind in the typecache > during the scan? I'm not entirely sure after reading your mail. I was > hoping something like a typedef could be left behind while building the > cache. We would then need a dereferencer for the metatag, but that > dereferencer would look just like the typedef one which has no searches > in it. > > The trick is that if you find one of your special namespace entries, it > would leave behind something with a name that matches what would be > searched for. This tag would be found, and then the dereferences would > just return a new name of whatever it referenced. > > What makes me uncertain if this will work now is that this only works > for a fully qualified type. If I understand the problem right, your > aliases are for names in the middle of a fully qualified name. Well, they can be completely unqualified. Consider --- someheader --- namespace foo { using namespace std; } ------------------ --- somefile --- #include "someheader" int main(void) { using namespace foo; vector<int> v; v.//complete here } Even if I had a metatag which tells me that foo is an alias for std, I'd still have to lookup 'vector' in std. > Or perhaps this is a premature optimization, and I'm just speculating. > Using ELP in the past has usually been the best thing. > > You could do before/after runs of the unit test suite, and see what the > delta is. That would be easy. There is no delta, since the dereferencer doesn't do anything for the unit tests, as they're all un-aliased types. Maybe we should a include a completion in the unit tests which deliberately fails, to see the impact when all dereferencers fire up and do their job. I tried to create something like a 'worst case' scenario (see the attached file). I include almost the full STL, then create a namespace 'foo' which is an alias for 'std'. When I then fire up the analyzer, I get the following profile: semantic-analyze-current-context 1 15.577188 15.577188 semantic-analyze-current-context-default 1 14.714472 14.714472 semantic-analyze-find-tag-sequence 1 12.615496 12.615496 semantic-analyze-find-tag-sequence-default 1 12.615482 12.615482 semantic-sort-tags-by-name-then-type-increasing 436 10.639908000 0.0244034587 semantic-tag-lessp-name-then-type 655951 3.1323869999 4.775...e-06 semanticdb-find-translate-path 7 2.2029980000 0.3147140000 semanticdb-find-translate-path-default 7 2.202899 0.3146998571 semanticdb-find-translate-path-includes-default 7 2.2027309999 0.3146758571 semanticdb-file-table-object 375 2.1800860000 0.0058135626 semanticdb-find-tags-collector 5 2.1140149999 0.4228029999 [...] semantic-c-dereference-namespace 2 0.027656 0.013828 semantic-c-dereference-type-alias 1 0.026057 0.026057 It seems the dereferencer doesn't really matter, but maybe the example is somewhat wrong; I'm not sure. Regards, David --- Test case: #include <algorithm> #include <ccomplex> #include <climits> #include <cstdarg> #include <ctgmath> #include <deque> #include <limits> #include <ostream> #include <stack> #include <tuple> #include <vector> #include <array> #include <cctype> #include <clocale> #include <cstdbool> #include <ctime> #include <exception> #include <iomanip> #include <list> #include <stdexcept> #include <typeinfo> #include <cerrno> #include <cmath> #include <cstddef> #include <cwchar> #include <locale> #include <queue> #include <streambuf> #include <cfenv> #include <complex> #include <cstdint> #include <cwctype> #include <iosfwd> #include <map> #include <random> #include <string> #include <unordered_map> #include <bitset> #include <cfloat> #include <cstdio> #include <iostream> #include <memory> #include <regex> #include <unordered_set> #include <cinttypes> #include <csetjmp> #include <cstdlib> #include <fstream> #include <istream> #include <new> #include <set> #include <utility> #include <cassert> #include <ciso646> #include <csignal> #include <cstring> #include <functional> #include <iterator> #include <numeric> #include <sstream> #include <valarray> namespace foo { using namespace std; } int main(void) { foo::vector<int> v; v. } ---- Profile: semantic-analyze-current-context 1 15.577188 15.577188 semantic-analyze-current-context-default 1 14.714472 14.714472 semantic-analyze-find-tag-sequence 1 12.615496 12.615496 semantic-analyze-find-tag-sequence-default 1 12.615482 12.615482 semantic-sort-tags-by-name-then-type-increasing 436 10.639908000 0.0244034587 semantic-tag-lessp-name-then-type 655951 3.1323869999 4.775...e-06 semanticdb-find-translate-path 7 2.2029980000 0.3147140000 semanticdb-find-translate-path-default 7 2.202899 0.3146998571 semanticdb-find-translate-path-includes-default 7 2.2027309999 0.3146758571 semanticdb-file-table-object 375 2.1800860000 0.0058135626 semanticdb-find-tags-collector 5 2.1140149999 0.4228029999 semantic-analyze-scoped-types 4 2.0995730000 0.5248932500 semantic-analyze-scoped-types-default 4 2.0995009999 0.5248752499 semantic-ctxt-scoped-types 4 2.0994539999 0.5248634999 semantic-ctxt-scoped-types-c++-mode 4 2.099407 0.52485175 semantic-calculate-scope 1 2.097087 2.097087 semanticdb-find-tags-by-class 4 2.089528 0.522382 semanticdb-find-translate-path-includes--internal 2 2.026349 1.0131745 semanticdb-find-table-for-include 317 1.9044870000 0.0060078454 semanticdb-find-table-for-include-default 317 1.8996039999 0.0059924416 semanticdb-get-database 14 1.856218 0.1325869999 semanticdb-create-database 15 1.8510979999 0.1234065333 semanticdb-load-database 12 1.8325850000 0.1527154166 semanticdb-find-load-unloaded 311 1.763112 0.0056691704 semanticdb-find-load-unloaded-default 247 1.7506250000 0.0070875506 semantic-decorate-add-decorations 301 1.1829669999 0.0039301229 semantic-refresh-tags-safe 3 0.862528 0.2875093333 semantic-fetch-tags 14 0.8625099999 0.0616078571 semantic--set-buffer-cache 2 0.5229929999 0.2614964999 semantic-decorate-tags-after-full-reparse 2 0.522069 0.2610345 semantic-decoration-on-includes-highlight 64 0.501624 0.007837875 semantic-decoration-on-includes-highlight-default 64 0.5004510000 0.0078195468 semantic-tag-copy 3811 0.4500199999 0.0001180844 semantic-analyze-scoped-type-parts 7 0.3445380000 0.0492197142 semantic-analyze-tag-type 1 0.340777 0.340777 semantic-analyze-type 1 0.3407670000 0.3407670000 semantic-analyze-dereference-metatype-stack 1 0.338855 0.338855 semantic-analyze-dereference-metatype 2 0.338837 0.1694185 semantic-analyze-dereference-metatype-c-mode 2 0.3387280000 0.1693640000 semantic-analyze-scoped-inherited-tags 6 0.331399 0.0552331666 semantic-analyze-scoped-inherited-tag-map 9 0.3313939999 0.0368215555 semantic-c-dereference-member-of 2 0.3109969999 0.1554984999 semantic-new-buffer-fcn 1 0.254285 0.254285 semantic-decoration-mode 1 0.219424 0.219424 semantic-decoration-mode-setup 1 0.219384 0.219384 semantic--tag-put-property 5935 0.2038390000 3.434...e-05 semanticdb-refresh-table 753 0.1861490000 0.0002472098 semantic-something-to-tag-table 1086 0.1793570000 0.0001651537 semantic-tag-with-position-p 4560 0.1777190000 3.897...e-05 semanticdb-get-table-index 274 0.1762299999 0.0006431751 semantic-tag-components-with-overlays 596 0.1717100000 0.0002881040 semantic-tag-components-with-overlays-default 596 0.1645049999 0.0002760151 semanticdb-find-search-index 235 0.1596429999 0.0006793319 semantic-tag-make-plist 6181 0.1525790000 2.468...e-05 semantic-tag-of-type-p 29117 0.1499450000 5.149...e-06 semanticdb-find-need-cache-update-p 7 0.1377140000 0.0196734285 semanticdb-find-incomplete-cache-entries-p 7 0.1373759999 0.0196251428 semanticdb-abstract-table-child-p 557 0.1333470000 0.0002394021 semanticdb-table-child-p 512 0.1320040000 0.0002578203 semantic--tag-link-to-buffer 298 0.1134949999 0.0003808557 semantic-parse-region 5 0.100303 0.0200606 semantic-parse-region-c-mode 5 0.100206 0.0200412000 semantic-parse-region-default 5 0.1001430000 0.0200286 semantic-tag-file-name 5229 0.0970610000 1.856...e-05 semantic-repeat-parse-whole-stream 5 0.093378 0.0186756 semantic-tag-in-buffer-p 6034 0.0841019999 1.393...e-05 semantic-parse-stream 75 0.0823349999 0.0010977999 semanticdb-find-tags-by-class-method 524 0.0701199999 0.0001338167 semantic-dependency-find-file-on-path 261 0.065377 0.0002504865 semanticdb-file-table 234 0.0641979999 0.0002743504 semantic-parse-stream-default 1 0.062927 0.062927 semanticdb-full-filename 892 0.0556190000 6.235...e-05 semantic-decorate-tag 155 0.0551050000 0.0003555161 semantic-overlay-p 22775 0.0525950000 2.309...e-06 semanticdb-needs-refresh-p 247 0.0509880000 0.0002064291 semanticdb-in-buffer-p 1382 0.0505340000 3.656...e-05 semanticdb-directory-loaded-p 233 0.0491140000 0.0002107896 semantic-tag-boundary-highlight 91 0.0430130000 0.0004726703 semantic--dependency-find-file-on-path 261 0.0426429999 0.0001633831 semantic-tag-boundary-highlight-default 91 0.0420030000 0.0004615714 semantic-tag-create-secondary-overlay 155 0.0418080000 0.0002697290 semanticdb-get-tags 1007 0.0394749999 3.920...e-05 semanticdb-refresh-references 2 0.037945 0.0189725 semanticdb-add-reference 64 0.037379 0.0005840468 semanticdb-semantic-init-hook-fcn 1 0.034149 0.034149 semanticdb-table 218 0.0341469999 0.0001566376 semantic--tag-link-cache-to-buffer 1 0.031986 0.031986 semantic-c-dereference-namespace 2 0.027656 0.013828 semantic-c-dereference-type-alias 1 0.026057 0.026057 semanticdb-equivalent-mode-for-search 247 0.0260299999 0.0001053846 semantic-tag-buffer 532 0.0258970000 4.867...e-05 semanticdb-find-tags-by-name 1 0.024531 0.024531 semantic-tag-components 951 0.0224859999 2.364...e-05 semantic-analyze-scope-nested-tags 4 0.0212569999 0.0053142499 semantic-analyze-scope-nested-tags-default 4 0.021198 0.0052995 semanticdb-find-tags-by-name-method 235 0.0186320000 7.928...e-05 [snipped] |
From: Eric M. L. <er...@si...> - 2009-08-29 12:04:35
|
On Sat, 2009-08-29 at 12:38 +0200, David Engster wrote: > Eric M. Ludlam <er...@si...> writes: > > I'm starting with the premise that we want to avoid the searches in a > > dereferencer. I'm sure the thing you wrote works fine, I'm just > > optimizing here. In fact, I'm only trying to optimize a single line, > > which was a global search across all tables. > > I'm not sure which line you are referring to. > > There is > > (semanticdb-find-tags-by-name name) > > in semantic-c-check-type-namespace-alias, and > > (semanticdb-find-tags-by-name (car typename)) > > in semantic-c-dereference-type-alias. Both do a search for the aliased > namespace. Then, there's These two are troublesome because they search "everything". If there was some search path to add to the search, then this wouldn't be a problem. I also know that this is important to do sometimes. For example, semantic-ctxt-scoped-types has to do this a couple times to build the scope. Fortunately in this case, the scope is cached, and can be recycled between requests. Actually, I wonder if the search for using statements in the scope can be used instead. Maybe I was completely off track, and we could add a :aliases slot to the scope? > (semantic-deep-find-tags-by-name typename result) This one is fine since you are passing the results of some other search in. The tough ones are when you have to search all the active tables. > which searches for the type in the namespace. > > > The question is then, can we leave special tags behind in the typecache > > during the scan? I'm not entirely sure after reading your mail. I was > > hoping something like a typedef could be left behind while building the > > cache. We would then need a dereferencer for the metatag, but that > > dereferencer would look just like the typedef one which has no searches > > in it. > > > > The trick is that if you find one of your special namespace entries, it > > would leave behind something with a name that matches what would be > > searched for. This tag would be found, and then the dereferences would > > just return a new name of whatever it referenced. > > > > What makes me uncertain if this will work now is that this only works > > for a fully qualified type. If I understand the problem right, your > > aliases are for names in the middle of a fully qualified name. > > Well, they can be completely unqualified. Consider > > --- someheader --- > > namespace foo { > using namespace std; > } > > ------------------ > > --- somefile --- > > #include "someheader" > > int main(void) { > using namespace foo; > vector<int> v; > v.//complete here > } > > Even if I had a metatag which tells me that foo is an alias for std, I'd > still have to lookup 'vector' in std. Yes. This is the key functionality of the metaclass dereference loop. Your dereferencer only needs to convert foo::vector into std::vector. The outer loop will then do the second lookup. What I think I now understand is that you cannot convert foo::vector to std::vector unless you search in std:: for vector to validate that translation? > > Or perhaps this is a premature optimization, and I'm just speculating. > > Using ELP in the past has usually been the best thing. > > > > You could do before/after runs of the unit test suite, and see what the > > delta is. That would be easy. > > There is no delta, since the dereferencer doesn't do anything for the > unit tests, as they're all un-aliased types. Maybe we should a include a > completion in the unit tests which deliberately fails, to see the impact > when all dereferencers fire up and do their job. Well, then that is good. It hadn't occurred to me that might happen. Only complex code will suffer complex lookups. Or perhaps failures in lookup will also go through this. > I tried to create something like a 'worst case' scenario (see the > attached file). I include almost the full STL, then create a namespace > 'foo' which is an alias for 'std'. When I then fire up the analyzer, I > get the following profile: > > semantic-analyze-current-context 1 15.577188 15.577188 > semantic-analyze-current-context-default 1 14.714472 14.714472 > semantic-analyze-find-tag-sequence 1 12.615496 12.615496 > semantic-analyze-find-tag-sequence-default 1 12.615482 12.615482 > semantic-sort-tags-by-name-then-type-increasing 436 10.639908000 0.0244034587 > semantic-tag-lessp-name-then-type 655951 3.1323869999 4.775...e-06 > semanticdb-find-translate-path 7 2.2029980000 0.3147140000 > semanticdb-find-translate-path-default 7 2.202899 0.3146998571 > semanticdb-find-translate-path-includes-default 7 2.2027309999 0.3146758571 > semanticdb-file-table-object 375 2.1800860000 0.0058135626 > semanticdb-find-tags-collector 5 2.1140149999 0.4228029999 > > [...] > > semantic-c-dereference-namespace 2 0.027656 0.013828 > semantic-c-dereference-type-alias 1 0.026057 0.026057 > > It seems the dereferencer doesn't really matter, but maybe the example > is somewhat wrong; I'm not sure. > > Regards, > David > > --- Test case: > > #include <algorithm> > #include <ccomplex> > #include <climits> > #include <cstdarg> > #include <ctgmath> > #include <deque> > #include <limits> > #include <ostream> > #include <stack> > #include <tuple> > #include <vector> > #include <array> > #include <cctype> > #include <clocale> > #include <cstdbool> > #include <ctime> > #include <exception> > #include <iomanip> > #include <list> > #include <stdexcept> > #include <typeinfo> > #include <cerrno> > #include <cmath> > #include <cstddef> > #include <cwchar> > #include <locale> > #include <queue> > #include <streambuf> > #include <cfenv> > #include <complex> > #include <cstdint> > #include <cwctype> > #include <iosfwd> > #include <map> > #include <random> > #include <string> > #include <unordered_map> > #include <bitset> > #include <cfloat> > #include <cstdio> > #include <iostream> > #include <memory> > #include <regex> > #include <unordered_set> > #include <cinttypes> > #include <csetjmp> > #include <cstdlib> > #include <fstream> > #include <istream> > #include <new> > #include <set> > #include <utility> > #include <cassert> > #include <ciso646> > #include <csignal> > #include <cstring> > #include <functional> > #include <iterator> > #include <numeric> > #include <sstream> > #include <valarray> > > namespace foo { > using namespace std; > } > > int main(void) > { > foo::vector<int> v; > v. > } > This certainly seems like a good way to exercise the system. > > ---- Profile: > > > semantic-analyze-current-context 1 15.577188 15.577188 > semantic-analyze-current-context-default 1 14.714472 14.714472 > semantic-analyze-find-tag-sequence 1 12.615496 12.615496 > semantic-analyze-find-tag-sequence-default 1 12.615482 12.615482 > semantic-sort-tags-by-name-then-type-increasing 436 10.639908000 0.0244034587 > semantic-tag-lessp-name-then-type 655951 3.1323869999 4.775...e-06 These two are our typecache overhead. > semanticdb-find-translate-path 7 2.2029980000 0.3147140000 > semanticdb-find-translate-path-default 7 2.202899 0.3146998571 > semanticdb-find-translate-path-includes-default 7 2.2027309999 0.3146758571 I think these used to be called once. It is needed for every search that doesn't specify a search path. The stuff I added for creating mini-scopes around a parent class to search for that parent's parent class probably increased this number. > semanticdb-file-table-object 375 2.1800860000 0.0058135626 > semanticdb-find-tags-collector 5 2.1140149999 0.4228029999 > semantic-analyze-scoped-types 4 2.0995730000 0.5248932500 > semantic-analyze-scoped-types-default 4 2.0995009999 0.5248752499 > semantic-ctxt-scoped-types 4 2.0994539999 0.5248634999 > semantic-ctxt-scoped-types-c++-mode 4 2.099407 0.52485175 These two are the overhead for the wide searches I talked about above that gets cached in the local scope. > semantic-calculate-scope 1 2.097087 2.097087 > semanticdb-find-tags-by-class 4 2.089528 0.522382 > semanticdb-find-translate-path-includes--internal 2 2.026349 1.0131745 > semanticdb-find-table-for-include 317 1.9044870000 0.0060078454 > semanticdb-find-table-for-include-default 317 1.8996039999 0.0059924416 These two show that you included lots of include files. :) > semanticdb-get-database 14 1.856218 0.1325869999 > semanticdb-create-database 15 1.8510979999 0.1234065333 > semanticdb-load-database 12 1.8325850000 0.1527154166 These show you ran the profiler test before the idle timer caught all the includes. > semanticdb-find-load-unloaded 311 1.763112 0.0056691704 > semanticdb-find-load-unloaded-default 247 1.7506250000 0.0070875506 I struggled with these a lot. Huzzah for the small time on them. :) > semantic-decorate-add-decorations 301 1.1829669999 0.0039301229 Hmmm. > semantic-refresh-tags-safe 3 0.862528 0.2875093333 > semantic-fetch-tags 14 0.8625099999 0.0616078571 > semantic--set-buffer-cache 2 0.5229929999 0.2614964999 > semantic-decorate-tags-after-full-reparse 2 0.522069 0.2610345 Hmmm again. Did you use a variant of semantic-elp-analyze to run your test? This fcn shouldn't show up unless you run the profiler test and request by hand. I apologize for not describing the profiling tool in Semantic earlier. Unless you did, and something is broken leaking out the decorator. It seems speedy enough though. The semantic wrapper for elp is helpful because it forces many caches to be deleted so it turns a given test into a worse case scenario. It will also save the results in an save-file that can be reloaded and browsed later, allowing comparisons between runs. > semantic-decoration-on-includes-highlight 64 0.501624 0.007837875 > semantic-decoration-on-includes-highlight-default 64 0.5004510000 0.0078195468 > semantic-tag-copy 3811 0.4500199999 0.0001180844 > semantic-analyze-scoped-type-parts 7 0.3445380000 0.0492197142 > semantic-analyze-tag-type 1 0.340777 0.340777 > semantic-analyze-type 1 0.3407670000 0.3407670000 > semantic-analyze-dereference-metatype-stack 1 0.338855 0.338855 > semantic-analyze-dereference-metatype 2 0.338837 0.1694185 > semantic-analyze-dereference-metatype-c-mode 2 0.3387280000 0.1693640000 > semantic-analyze-scoped-inherited-tags 6 0.331399 0.0552331666 > semantic-analyze-scoped-inherited-tag-map 9 0.3313939999 0.0368215555 > semantic-c-dereference-member-of 2 0.3109969999 0.1554984999 > semantic-new-buffer-fcn 1 0.254285 0.254285 > semantic-decoration-mode 1 0.219424 0.219424 > semantic-decoration-mode-setup 1 0.219384 0.219384 > semantic--tag-put-property 5935 0.2038390000 3.434...e-05 > semanticdb-refresh-table 753 0.1861490000 0.0002472098 > semantic-something-to-tag-table 1086 0.1793570000 0.0001651537 > semantic-tag-with-position-p 4560 0.1777190000 3.897...e-05 > semanticdb-get-table-index 274 0.1762299999 0.0006431751 > semantic-tag-components-with-overlays 596 0.1717100000 0.0002881040 > semantic-tag-components-with-overlays-default 596 0.1645049999 0.0002760151 > semanticdb-find-search-index 235 0.1596429999 0.0006793319 > semantic-tag-make-plist 6181 0.1525790000 2.468...e-05 > semantic-tag-of-type-p 29117 0.1499450000 5.149...e-06 > semanticdb-find-need-cache-update-p 7 0.1377140000 0.0196734285 > semanticdb-find-incomplete-cache-entries-p 7 0.1373759999 0.0196251428 > semanticdb-abstract-table-child-p 557 0.1333470000 0.0002394021 > semanticdb-table-child-p 512 0.1320040000 0.0002578203 > semantic--tag-link-to-buffer 298 0.1134949999 0.0003808557 > semantic-parse-region 5 0.100303 0.0200606 > semantic-parse-region-c-mode 5 0.100206 0.0200412000 > semantic-parse-region-default 5 0.1001430000 0.0200286 > semantic-tag-file-name 5229 0.0970610000 1.856...e-05 > semantic-repeat-parse-whole-stream 5 0.093378 0.0186756 > semantic-tag-in-buffer-p 6034 0.0841019999 1.393...e-05 > semantic-parse-stream 75 0.0823349999 0.0010977999 > semanticdb-find-tags-by-class-method 524 0.0701199999 0.0001338167 > semantic-dependency-find-file-on-path 261 0.065377 0.0002504865 > semanticdb-file-table 234 0.0641979999 0.0002743504 > semantic-parse-stream-default 1 0.062927 0.062927 > semanticdb-full-filename 892 0.0556190000 6.235...e-05 > semantic-decorate-tag 155 0.0551050000 0.0003555161 > semantic-overlay-p 22775 0.0525950000 2.309...e-06 > semanticdb-needs-refresh-p 247 0.0509880000 0.0002064291 > semanticdb-in-buffer-p 1382 0.0505340000 3.656...e-05 > semanticdb-directory-loaded-p 233 0.0491140000 0.0002107896 > semantic-tag-boundary-highlight 91 0.0430130000 0.0004726703 > semantic--dependency-find-file-on-path 261 0.0426429999 0.0001633831 > semantic-tag-boundary-highlight-default 91 0.0420030000 0.0004615714 > semantic-tag-create-secondary-overlay 155 0.0418080000 0.0002697290 > semanticdb-get-tags 1007 0.0394749999 3.920...e-05 > semanticdb-refresh-references 2 0.037945 0.0189725 > semanticdb-add-reference 64 0.037379 0.0005840468 > semanticdb-semantic-init-hook-fcn 1 0.034149 0.034149 > semanticdb-table 218 0.0341469999 0.0001566376 > semantic--tag-link-cache-to-buffer 1 0.031986 0.031986 > semantic-c-dereference-namespace 2 0.027656 0.013828 > semantic-c-dereference-type-alias 1 0.026057 0.026057 Wow, I'm impressed you can have those searches in these routines, and have them be this speedy. I think you have proven your point well. Thanks for putting in the extra effort. Unless my first comment about adding some data to the scope calculation resonates with you, I think your patch is as safe to do as anything else, and you can check it in. Thanks! Eric |
From: David E. <de...@ra...> - 2009-08-29 15:46:09
|
Eric M. Ludlam <er...@si...> writes: > On Sat, 2009-08-29 at 12:38 +0200, David Engster wrote: >> There is >> >> (semanticdb-find-tags-by-name name) >> >> in semantic-c-check-type-namespace-alias, and >> >> (semanticdb-find-tags-by-name (car typename)) >> >> in semantic-c-dereference-type-alias. Both do a search for the aliased >> namespace. Then, there's > > These two are troublesome because they search "everything". If there > was some search path to add to the search, then this wouldn't be a > problem. > > I also know that this is important to do sometimes. For example, > semantic-ctxt-scoped-types has to do this a couple times to build the > scope. Fortunately in this case, the scope is cached, and can be > recycled between requests. It seems that there's some caching going on in the dereferencer as well. Actually, I had a wrong memory of what the dereferencer does. It does not look up the new symbol, it just gives back its name. In the attached example (which included the STL), it just gives back "std::vector" as the possible type. It could be that this type is also an alias, but as far as I see, the dereferencer is then called again from the top loop and so it can be dereferenced again. It has to lookup the namespace 'bar' though, but that one was very small. Hence, the real worst case is when the dereferencer has to look up 'std', for example by doing this kind of extension of the std-namespace: #include <.... whole STL ...> namespace bar { struct aStruct { int whatever; }; } namespace std { using namespace bar; } int main(void) { using namespace std; aStruct foo; foo. // complete here } In this case, the dereferencer has to search the namespace 'std' for all using statements, and for this it has to do (semanticdb-find-tags-by-name "std") When I do this manually in the buffer, this is practically killing Emacs. However, in the dereferencer this search is almost instantaneous. I don't know why. > Yes. This is the key functionality of the metaclass dereference loop. > Your dereferencer only needs to convert foo::vector into std::vector. > The outer loop will then do the second lookup. > > What I think I now understand is that you cannot convert foo::vector to > std::vector unless you search in std:: for vector to validate that > translation? Well, as I've written above, I'm afraid I was mistaken in my description. It actually returns just the new symbol, as all the other dereferencers do. > Hmmm again. Did you use a variant of semantic-elp-analyze to run your > test? This fcn shouldn't show up unless you run the profiler test and > request by hand. I tried, but it didn't include the dereferencers, so I simply instrumented ELP manually. > Wow, I'm impressed you can have those searches in these routines, and > have them be this speedy. > > I think you have proven your point well. Thanks for putting in the > extra effort. Unless my first comment about adding some data to the > scope calculation resonates with you, I think your patch is as safe to > do as anything else, and you can check it in. After that manual run of semanticdb-find-tags-by-name, I'm a bit worried that I did not really catch the worst case scenario. I will have to experiment a bit more with this code, and also think about this metatype-idea. -David |
From: Eric M. L. <er...@si...> - 2009-08-29 18:13:05
|
On Sat, 2009-08-29 at 17:45 +0200, David Engster wrote: > Eric M. Ludlam <er...@si...> writes: > > On Sat, 2009-08-29 at 12:38 +0200, David Engster wrote: > > In this case, the dereferencer has to search the namespace 'std' for all > using statements, and for this it has to do > > (semanticdb-find-tags-by-name "std") > > When I do this manually in the buffer, this is practically killing > Emacs. However, in the dereferencer this search is almost > instantaneous. I don't know why. The current buffer might be wrong when it looks things up in the dereferencer, or perhaps it doesn't hit that line? If you know you are looking up a type, which std is, since it is a namespace, then you can instead look it up in the typecache. That will be much faster since there is only one answer. The typecache will merge all the std namespaces together for you ahead of time. > > > Hmmm again. Did you use a variant of semantic-elp-analyze to run your > > test? This fcn shouldn't show up unless you run the profiler test and > > request by hand. > > I tried, but it didn't include the dereferencers, so I simply > instrumented ELP manually. It has long lists of symbols to instrument. You need to edit semantic-elp to add new fcns when you introduce them. Eric |
From: David E. <de...@ra...> - 2009-09-02 12:59:03
|
Eric M. Ludlam <er...@si...> writes: > On Sat, 2009-08-29 at 17:45 +0200, David Engster wrote: >> Eric M. Ludlam <er...@si...> writes: >> > On Sat, 2009-08-29 at 12:38 +0200, David Engster wrote: >> In this case, the dereferencer has to search the namespace 'std' for all >> using statements, and for this it has to do >> >> (semanticdb-find-tags-by-name "std") [...] > The current buffer might be wrong when it looks things up in the > dereferencer, or perhaps it doesn't hit that line? > > If you know you are looking up a type, which std is, since it is a > namespace, then you can instead look it up in the typecache. That will > be much faster since there is only one answer. The typecache will > merge all the std namespaces together for you ahead of time. I've committed the new namespace/using dereferencer. I've refactored the code quite a bit and removed some code duplication, so that it's smaller and easier to understand. Also, I removed the recursive searches since the dereferencer seems to be called again from the top loop anyway when dealing with "aliases of aliases". Much more important, I was able to eliminate the global semanticdb-searches by using the typecache, so we shouldn't have to worry about performance anymore. :-) I've also committed the new unit tests. Strangely, one of the tests is working with 'make utest', but not interactively. I will have to investigate that one further. Regards, David |
From: Eric M. L. <er...@si...> - 2009-09-03 00:31:21
|
On Wed, 2009-09-02 at 14:58 +0200, David Engster wrote: > Eric M. Ludlam <er...@si...> writes: > > On Sat, 2009-08-29 at 17:45 +0200, David Engster wrote: > >> Eric M. Ludlam <er...@si...> writes: > >> > On Sat, 2009-08-29 at 12:38 +0200, David Engster wrote: > >> In this case, the dereferencer has to search the namespace 'std' for all > >> using statements, and for this it has to do > >> > >> (semanticdb-find-tags-by-name "std") > > [...] > > > The current buffer might be wrong when it looks things up in the > > dereferencer, or perhaps it doesn't hit that line? > > > > If you know you are looking up a type, which std is, since it is a > > namespace, then you can instead look it up in the typecache. That will > > be much faster since there is only one answer. The typecache will > > merge all the std namespaces together for you ahead of time. > > I've committed the new namespace/using dereferencer. I've refactored the > code quite a bit and removed some code duplication, so that it's smaller > and easier to understand. Also, I removed the recursive searches since > the dereferencer seems to be called again from the top loop anyway when > dealing with "aliases of aliases". Great. I have checked it out, and validated your new tests all work. > Much more important, I was able to eliminate the global > semanticdb-searches by using the typecache, so we shouldn't have to > worry about performance anymore. :-) Even better! > I've also committed the new unit tests. Strangely, one of the tests is > working with 'make utest', but not interactively. I will have to > investigate that one further. The makefile starts up Emacs in a way that causes Semanticdb to ignore all saved databases, start from scratch, and to not save any databases. Your running Emacs may have left over cache files with incorrect data in them. Eric |