CEDET

Are you using the cedet beta, or semantic 1.4?  The search path you
mention in 1.4 is only useful if you want to jump to an include file.

In 1.4, it will search all files in the current project (directory
tree) for matches.  In 2.0beta, it explicitly chases search paths.
There were a couple bugs in it, so include files including files
don't work yet.  One level of include should work ok though.

Eric

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>I've been playing around with semantic (and the other bits), and
>they're potentially invaluable, I think.  
>
>But so far only (or mostly) within one file.  So presumably I'm doing
>something wrong.
>
>I have a file foo.h in ~/include, which declares one function, and a
>file bar.c in ~/temp which uses it.  Now, if bar.c has #include
>"~/include/foo.h", then semantic shows the prototype when point is
>over it, as I'd hope (I've visited foo.h, so semantic.cache has been
>created).
>
>Obviously, in reality I'd want to have #include "foo.h", and to
>declare include paths somehow.  But I can't seem to get that to work.
>
>It seems that setting semantic-default-c-path ought to work.
>Something like:
>
>(setq semantic-default-c-path (list 
>			       (expand-file-name "~/include")))
>
>That causes semantic-dependency-include-path to be set plausibly.  But
>it doesn't seem to work.  (Adding a file-name-as-directory doesn't
>help.)
>
>So how is this supposed to work?
  [ ... ]

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

> Are you using the cedet beta, or semantic 1.4?  The search path you
> mention in 1.4 is only useful if you want to jump to an include file.
>
> In 1.4, it will search all files in the current project (directory
> tree) for matches.  In 2.0beta, it explicitly chases search paths.
> There were a couple bugs in it, so include files including files
> don't work yet.  One level of include should work ok though.

It's whatever's in CVS, so presumably 2.0beta (that's what
semantic.cache says).  I'd like multi-level includes, but in this case
it's just one level.  ~/include/foo.h contains

extern void foo(int a, const char *b);

and ~/temp/bar.c contains

#include "foo.h"
int
main()
{
    int c = foo (

And in the buffer containing bar.c, semantic-default-c-path and
semantic-dependency-include-path both have the value
'("/home/bruce/include").  I'm expecting to see prototype information
for foo when point is over foo.  And when I have the full path in the
#include, then I do.

[...]

"Eric M. Ludlam" <eric@...> writes:

> Are you using the cedet beta, or semantic 1.4?  The search path you
> mention in 1.4 is only useful if you want to jump to an include file.
>
> In 1.4, it will search all files in the current project (directory
> tree) for matches.  In 2.0beta, it explicitly chases search paths.
> There were a couple bugs in it, so include files including files
> don't work yet.  One level of include should work ok though.

Any hints on how I can debug this?  I'm comfortable with elisp and
edebug, but eieio seems to push things just beyond where that'll work
easily, and I'm getting lost in a maze of define-overloads.

Or can someone describe a simple setup (say, a simple two-file one
like I described: a C file and a header file, in different
directories), together with settings of whatever variables are
necessary such that things definitely ought to work?  Perhaps from a
concrete example like that I could work out what I'm doing wrong.

[...]

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>> Are you using the cedet beta, or semantic 1.4?  The search path you
>> mention in 1.4 is only useful if you want to jump to an include file.
>>
>> In 1.4, it will search all files in the current project (directory
>> tree) for matches.  In 2.0beta, it explicitly chases search paths.
>> There were a couple bugs in it, so include files including files
>> don't work yet.  One level of include should work ok though.
>
>Any hints on how I can debug this?  I'm comfortable with elisp and
>edebug, but eieio seems to push things just beyond where that'll work
>easily, and I'm getting lost in a maze of define-overloads.
>
>Or can someone describe a simple setup (say, a simple two-file one
>like I described: a C file and a header file, in different
>directories), together with settings of whatever variables are
>necessary such that things definitely ought to work?  Perhaps from a
>concrete example like that I could work out what I'm doing wrong.
  [ ... ]

Hi,

  In the semantic 2.0 beta, if you are using eldoc, it will use
semanticdb and the new 'find' API.  The specific function you can
examine is: `semanticdb-find-table-for-include-default'.

  A typical "overload" function is pretty simple.  Any function that
does some processing with a tag that may need to be specific to a
language is defined as an overload function.  All this does is go
through a redirection table.  An implementation can be defined for a
given language.  If there is no implementation, a `-default' is
used.  A default is a function with the same name as the overload
function with -default in the name.  This makes calls similar to
"super.foo()" possible in this circumstance.  The trick is knowing
which functions have been overloaded.  All such functions are usually
in the language specific file, like `semantic-c.el'.  They are created
with a special macro which does have edebug support.

  Anyway, in the function `semanticdb-find-table-for-include-default',
you will see a comment that says: ;;;; THIS NEEDS WORK!
That particular area needs some work in that it caused some serious
speed issues during casual use.

  The point of the function is to create a list of semanticdb tables
which represents all files a given location has access to when looking
for symbols.  Doing this:

M-: (semanticdb-find-translate-path nil nil)

  should then report back a list of tables, with one representing each
header file in a given sample.

Good Luck
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

>   In the semantic 2.0 beta, if you are using eldoc, it will use
> semanticdb and the new 'find' API.  The specific function you can
> examine is: `semanticdb-find-table-for-include-default'.

Thanks, that what I needed to get started.

[...]

> Anyway, in the function `semanticdb-find-table-for-include-default',
> you will see a comment that says: ;;;; THIS NEEDS WORK!  That
> particular area needs some work in that it caused some serious speed
> issues during casual use.

Right.  A bit of a giveaway!  Without that code, there seems nothing
that would cause the "foo.h" #included in ~/test/c/bar.c to be
associated with the file ~/test/include/foo.h, even if the
semantic.cache associated with the latter happened to have been loaded
(which it hasn't).

So people who claim to have this working must be talking about a
different version (one before this section was commented-out), right,
or perhaps a case where there's no need to follow an include path?
Because surely there's nothing that could cause it to work with the
code as it is right now, even for single-level includes?

Things will work for #include files in the same directory (or
generally where the #include name is giving a path relative to the
buffer file name), but I don't have much use for that---most uses of
#include in the code I'm dealing with are in separate directories,
which gcc finds using -I flags.

So I think my best bet is to try uncommenting that code, and see how
it works out for me.

[...]

Hi Bruce,

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
  [ ... ]
>> Anyway, in the function `semanticdb-find-table-for-include-default',
>> you will see a comment that says: ;;;; THIS NEEDS WORK!  That
>> particular area needs some work in that it caused some serious speed
>> issues during casual use.
>
>Right.  A bit of a giveaway!  Without that code, there seems nothing
>that would cause the "foo.h" #included in ~/test/c/bar.c to be
>associated with the file ~/test/include/foo.h, even if the
>semantic.cache associated with the latter happened to have been loaded
>(which it hasn't).
>
>So people who claim to have this working must be talking about a
>different version (one before this section was commented-out), right,
>or perhaps a case where there's no need to follow an include path?
>Because surely there's nothing that could cause it to work with the
>code as it is right now, even for single-level includes?

Aha, therein lay my own confusion.  I remember it working with
include files in the same directory as the file I was working on.

I also set up a project root, so I get header files from different
parts of my tree, but nothing from areas like /usr/include.

>Things will work for #include files in the same directory (or
>generally where the #include name is giving a path relative to the
>buffer file name), but I don't have much use for that---most uses of
>#include in the code I'm dealing with are in separate directories,
>which gcc finds using -I flags.
>
>So I think my best bet is to try uncommenting that code, and see how
>it works out for me.
  [ ... ]

Good Luck.  I found that it "worked", which made me happy.  Then I
tried to use it in a real situation, and the setup for the different
queries took a long time which I found irritating.  Basically, I need
a way to automate first-time setup in a smart way so as to minimize
the pain.

Good Luck
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> Aha, therein lay my own confusion.  I remember it working with
> include files in the same directory as the file I was working on.
>
> I also set up a project root, so I get header files from different
> parts of my tree, but nothing from areas like /usr/include.

I've set up a project root, too, but I don't see how that makes any
difference.  

Project roots will surely make sense in the longer term, because I'll
want the include path to be per-project.  In theory, per directory or
even per file, but for practical purposes, per-project will work fine.

There seem to be two (connected) aspects to this: one is whether the
relevant semantic.cache file has been loaded (and it makes sense for a
project to help with that); the second is relating a tag "foo.h" in
some file with the file "foo.h" somewhere else in the filesystem.  

I'm not sure that a project root helps with the latter.  It seems to
me that the latter has to be done either by some kind of ad-hoc
searching---look for any foo.h, or perhaps look for the closest foo.h
or something---or, more practically, look in the include path.

[...]

>>So I think my best bet is to try uncommenting that code, and see how
>>it works out for me.
>
> Good Luck.  I found that it "worked", which made me happy.  Then I
> tried to use it in a real situation, and the setup for the different
> queries took a long time which I found irritating.  Basically, I
> need a way to automate first-time setup in a smart way so as to
> minimize the pain.

I see what you mean.  It's much too slow for me to use.

There are several things involved.  One is parsing the various files
(191 according to gcc -M, for the file I tried with).  That's going to
take time, but I can live with it---it's something that can be done
overnight.  (It would be unacceptable for someone just starting to use
cedet, so some of the automated aspects need to be disabled by
default, as you say.)

Another is the backend storage---one file per directory is probably
going to be OK, except that some of these files are rather big.  So
when I'm trying to get information about some non-SSL related function
in a file which #includes openssl/ssl.h or something, I need to wait
while the 1.8M semantic.cache file relating to include/openssl gets
read.  Yuck.  Even worse, the commented out section seems to load the
databases by visiting the files, so not only do the tables get read, I
also get Emacs buffers (font-locked, imenu, etc.), for files that I'm
mostly not interested in.

I guess I need a proper database backend, so that the code can ask for
the specific information it wants without having to read in irrelevant
stuff that happens to be in the same directory.  That won't help
creation of the information, but I think I can live with that (as
mentioned above, mostly that can happen in batch mode).  

Has anyone written an alternative backend for semanticdb?

[...]

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>[...]
>
>> Aha, therein lay my own confusion.  I remember it working with
>> include files in the same directory as the file I was working on.
>>
>> I also set up a project root, so I get header files from different
>> parts of my tree, but nothing from areas like /usr/include.
>
>I've set up a project root, too, but I don't see how that makes any
>difference.  
>
>Project roots will surely make sense in the longer term, because I'll
>want the include path to be per-project.  In theory, per directory or
>even per file, but for practical purposes, per-project will work fine.
>
>There seem to be two (connected) aspects to this: one is whether the
>relevant semantic.cache file has been loaded (and it makes sense for a
>project to help with that); the second is relating a tag "foo.h" in
>some file with the file "foo.h" somewhere else in the filesystem.  
>
>I'm not sure that a project root helps with the latter.  It seems to
>me that the latter has to be done either by some kind of ad-hoc
>searching---look for any foo.h, or perhaps look for the closest foo.h
>or something---or, more practically, look in the include path.

I think you are right.  There are two ways to search through
databases.  There is the "go through the include files" which is
supposed to use the include path, and there is the "brutish"
mechanism wich scans all files within the project.  I think I've
confused myself somewhere in my previous description.

>[...]
>
>>>So I think my best bet is to try uncommenting that code, and see how
>>>it works out for me.
>>
>> Good Luck.  I found that it "worked", which made me happy.  Then I
>> tried to use it in a real situation, and the setup for the different
>> queries took a long time which I found irritating.  Basically, I
>> need a way to automate first-time setup in a smart way so as to
>> minimize the pain.
>
>I see what you mean.  It's much too slow for me to use.
>
>There are several things involved.  One is parsing the various files
>(191 according to gcc -M, for the file I tried with).  That's going to
>take time, but I can live with it---it's something that can be done
>overnight.  (It would be unacceptable for someone just starting to use
>cedet, so some of the automated aspects need to be disabled by
>default, as you say.)

There is a shell script "semanticdb.sh" which will let you process
the files from a Makefile, cron, or whatever.

>Another is the backend storage---one file per directory is probably
>going to be OK, except that some of these files are rather big.  So
>when I'm trying to get information about some non-SSL related function
>in a file which #includes openssl/ssl.h or something, I need to wait
>while the 1.8M semantic.cache file relating to include/openssl gets
>read.  Yuck.  Even worse, the commented out section seems to load the
>databases by visiting the files, so not only do the tables get read, I
>also get Emacs buffers (font-locked, imenu, etc.), for files that I'm
>mostly not interested in.

There is a "system" database back end.  You can create a system
database via the command:

M-x semanticdb-create-system-database RET

The way it is "supposed" to work is that if you want tags from some
system header file, you can find it from one of these system
databases.  Unfortunately, I think the logic is a bit screwy right
now where "brutish" searches scan systems, but header file jumping
does not.

Ideally, the code you are looking at should "fetch the tokens from"
the header file (instead of jumping to the file as you identified).
For system includes (marked as such in the semantic tag) it could
look for in this database.

System databases are loaded at Emacs start time when cedet is loaded
and semanticdb is activated, so a large systemdb would burn cycles
then.

This interaction is something I just haven't gotten to yet.

>I guess I need a proper database backend, so that the code can ask for
>the specific information it wants without having to read in irrelevant
>stuff that happens to be in the same directory.  That won't help
>creation of the information, but I think I can live with that (as
>mentioned above, mostly that can happen in batch mode).  
>
>Has anyone written an alternative backend for semanticdb?
  [ ... ]

There are different types of back ends. The semanticdb-el.el library,
for example, is a back end that directly queries Emacs' table of
loaded symbols.  I believe there is a similar library being made for
java .class or .jar files.

I thought it would be cool to have an alternative to semanticdb-file
which would save tag info in postgres or mysql.  This sort of
database would be ideal for system databases that you are hoping to
use.  For projects, file's may be more useful, but for the whole
system, a read DB might be best.

There is an accommodation for this occurring inside the core
semanticdb classes, but of course it has not been tested.  There are
just oh-so-many things to do.

If you contemplate doing this, you will be pleased to not that the
core methods to override are quite simple.  If you examine
semanticdb-file.el, you will see that you just need your new eieio
classes, and to override the methods:

semanticdb-create-database
semanticdb-save-db - which may not be needed for a real db
semanticdb-full-filename - perhaps, not sure

In your case, you would also need to implement a new class for
semanticdb-table, where the table implements semanticdb-get-tags,
which may or may not choose to cache tags out of the database.  I
only recently started putting more restrictions on access to a
table's tags.  For a DB backend to be useful, blindly fetching all
the tags should be pushed down to 0 for application use.  That could
take some time and thought, but could be worked around short-term for
an alternate back end.

That may be all for basic manipulation.  After that, of course, you
need to implement all the query functions which is where the real
benefit to an SQL backend would lie.

Upon examination of the semanticdb table, I see I wasn't very clean
about where some slots are.  That semanticdb table implementation
could prove challenging. Hmmm.

Good Luck
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

"Eric M. Ludlam" <eric@...> writes:

[...]

> I think you are right.  There are two ways to search through
> databases.  There is the "go through the include files" which is
> supposed to use the include path, and there is the "brutish"
> mechanism wich scans all files within the project.  I think I've
> confused myself somewhere in my previous description.

Both can work.  Source Navigator <http://sourcenav.sourceforge.net/>
uses the latter (it has databases by symbol name, for various classes
of object), and personally I find it *really* annoying: I point at a
function name in a file, and want to jump to its definition, and
Source Navigator will ask which of the functions with that name I
want.  i.e., it seems to have no way of using the context from the
fact that I'm pointing at a particular function name.  However, that
kind of thing can work sometimes.

[...]

> There is a shell script "semanticdb.sh" which will let you process
> the files from a Makefile, cron, or whatever.

That only works if you install semantic in some system directory.  I
put the paths in my .emacs, so that shellscript doesn't work for me.
However, that's a trivial point.

[...]

> There is a "system" database back end.  You can create a system
> database via the command:
>
> M-x semanticdb-create-system-database RET
>
> The way it is "supposed" to work is that if you want tags from some
> system header file, you can find it from one of these system
> databases.  Unfortunately, I think the logic is a bit screwy right
> now where "brutish" searches scan systems, but header file jumping
> does not.
>
> Ideally, the code you are looking at should "fetch the tokens from"
> the header file (instead of jumping to the file as you identified).
> For system includes (marked as such in the semantic tag) it could
> look for in this database.
>
> System databases are loaded at Emacs start time when cedet is loaded
> and semanticdb is activated, so a large systemdb would burn cycles
> then.

I think I'd find that too annoying.  Although mostly I use long-lived
Emacses, now and again I like to fire up a new one for a quick edit,
so an expensive startup would get in my way, unless I changed my
habits, of course.

[...]

> I thought it would be cool to have an alternative to semanticdb-file
> which would save tag info in postgres or mysql.  This sort of
> database would be ideal for system databases that you are hoping to
> use.

I'm not necessarily thinking of system databases, although that would
be an obvious application, and I guess it might make sense in a large
project to regard slow-changing parts to be a system database.  I was
thinking about significant sized projects.

And I think you're right that an appropriate backend would be
postgresql (with pg.el, perhaps) or mysql, or perhaps even LDAP (LDAP
might be good because it's designed for hierarchical databases where
read/search performance is much more important than modify
performance).

> For projects, file's may be more useful, but for the whole
> system, a read DB might be best.
>
> There is an accommodation for this occurring inside the core
> semanticdb classes, but of course it has not been tested.  There are
> just oh-so-many things to do.
>
> If you contemplate doing this, you will be pleased to not that the
> core methods to override are quite simple.  If you examine
> semanticdb-file.el, you will see that you just need your new eieio
> classes, and to override the methods:
>
> semanticdb-create-database
> semanticdb-save-db - which may not be needed for a real db
> semanticdb-full-filename - perhaps, not sure
>
> In your case, you would also need to implement a new class for
> semanticdb-table, where the table implements semanticdb-get-tags,
> which may or may not choose to cache tags out of the database.  I
> only recently started putting more restrictions on access to a
> table's tags.  For a DB backend to be useful, blindly fetching all
> the tags should be pushed down to 0 for application use.  That could
> take some time and thought, but could be worked around short-term for
> an alternate back end.
>
> That may be all for basic manipulation.  After that, of course, you
> need to implement all the query functions which is where the real
> benefit to an SQL backend would lie.
>
> Upon examination of the semanticdb table, I see I wasn't very clean
> about where some slots are.  That semanticdb table implementation
> could prove challenging. Hmmm.

Thanks for the outline of what's necessary.

I was a bit worried that the querying probably wasn't abstract enough.

However, for some simple uses I was thinking I could just hack
something.  For example, when loading a file, I could construct a list
of relevant files (the file itself and all files included by it), and
for showing function prototypes I could just do a sequence of queries
using that list.  Now, that's not *really* want I want---I want to be
able to do completion and things---but it would be useful.

>>> Bruce Stephens <bruce+cedet@...> seems to think that:
>"Eric M. Ludlam" <eric@...> writes:
>
>[...]
>
>> I think you are right.  There are two ways to search through
>> databases.  There is the "go through the include files" which is
>> supposed to use the include path, and there is the "brutish"
>> mechanism wich scans all files within the project.  I think I've
>> confused myself somewhere in my previous description.
>
>Both can work.  Source Navigator <http://sourcenav.sourceforge.net/>
>uses the latter (it has databases by symbol name, for various classes
>of object), and personally I find it *really* annoying: I point at a
>function name in a file, and want to jump to its definition, and
>Source Navigator will ask which of the functions with that name I
>want.  i.e., it seems to have no way of using the context from the
>fact that I'm pointing at a particular function name.  However, that
>kind of thing can work sometimes.

That is the sort of thing that the analyzer can do.  That particular
interface isn't written yet though.  In theory, it might look like
this:

(let* ((a (semantic-analyze-current-context))
       (tag (car (reverse (oref a prefix)))))
   ( jump to TAG ))

You can test just the analyzer with:

M-x semantic-analyze-current-context

>[...]
>
>> There is a shell script "semanticdb.sh" which will let you process
>> the files from a Makefile, cron, or whatever.
>
>That only works if you install semantic in some system directory.  I
>put the paths in my .emacs, so that shellscript doesn't work for me.
>However, that's a trivial point.

You can hack the script, or set the environment variable $SEMANTIC_PATH.

>[...]
>
>> There is a "system" database back end.  You can create a system
>> database via the command:
>>
>> M-x semanticdb-create-system-database RET
>>
>> The way it is "supposed" to work is that if you want tags from some
>> system header file, you can find it from one of these system
>> databases.  Unfortunately, I think the logic is a bit screwy right
>> now where "brutish" searches scan systems, but header file jumping
>> does not.
>>
>> Ideally, the code you are looking at should "fetch the tokens from"
>> the header file (instead of jumping to the file as you identified).
>> For system includes (marked as such in the semantic tag) it could
>> look for in this database.
>>
>> System databases are loaded at Emacs start time when cedet is loaded
>> and semanticdb is activated, so a large systemdb would burn cycles
>> then.
>
>I think I'd find that too annoying.  Although mostly I use long-lived
>Emacses, now and again I like to fire up a new one for a quick edit,
>so an expensive startup would get in my way, unless I changed my
>habits, of course.

Some sort of on-demand loading may be needed then.

  [ ... ]
>Thanks for the outline of what's necessary.
>
>I was a bit worried that the querying probably wasn't abstract enough.
>
>However, for some simple uses I was thinking I could just hack
>something.  For example, when loading a file, I could construct a list
>of relevant files (the file itself and all files included by it), and
>for showing function prototypes I could just do a sequence of queries
>using that list.  Now, that's not *really* want I want---I want to be
>able to do completion and things---but it would be useful.
  [ ... ]

The function semanticdb-find-translate-path is an override method.  If
you override this method, you could have it return any list of
semanticdb tables you want using your simplified logic.  It is, for
normal searches, supposed to scan all the header files so such a
change would not be too obscure.

Good Luck
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ludlam.net            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org