Thank you for these detailed explanations. Forgive me when I cannot
answer in a similarly detailed fashion, but time is sparse at the
moment. Eric could probably say more about EDE than me, but it's
pumpkin-throwing season. :-)
Alastair Rankine writes:
> This information is very useful for tools like CEDET, as it enables the tool to
> unambiguously determine the include paths and preprocessor definitions for C
> and C++ source files. This information is otherwise quite difficult to
> determine automatically, and most current tools typically require it to be
> provided redundantly (eg once in the build tool input file and again in an EDE
Yes. I agree such a database would be very useful. I'm not particularly
fond of tools which wrap themselves around 'make' or similar and
override CC etc. to get this information, since they often don't play
well with complicated builds, stuff like distcc and similar.
Therefore, if this compilation database is trying to establish a
standard for this ti get support from compilers, so that we don't need
this hacks anymore, I'm all for it.
> So how is the compilation database generated? I know of two ways, and there may
> be others. For CMake-based projects using the GNU Make build tool, there is an
> option (CMAKE_EXPORT_COMPILE_COMMANDS) which tells CMake to write out a
> compile_commands.json file along with the generated Makefiles in the build
> directory. This file contains the entire compilation database for the project.
> For projects using the Ninja build tool (http://martine.github.io/ninja/), the
> compilation database can be generated on-demand using the "-t compdb" command,
> either for the entire project, or for a specific source file.
I just tried out the tool 'Bear' which does this "wrap around
Make"-Hack, and it actually worked nicely with gcc:
As you might know from recent discussion, it would be problematic if
this feature would only work with clang, so this is good news.
> Firstly, I'm thinking that a new EDE project type would be needed. This is
> because the include paths and preprocessor symbols would be sourced from the
> compilation database on-demand, unlike any of the existing project types.
> Also, the current semantic/bovine/clang support isn't really suitable for reuse
> here. In fact it seems to be going the opposite direction - using the include
> paths and defines already in the EDE project, to build a clang command line for
> performing semantic completion.
Yes, this is outside of the scope of semantic-clang, which is only there
to provide completions.
> 1. Maintain a build directory per build configuration. The build directory is
> required to access the compilation database either as a file, or by invoking
> the build tool. A desirable feature would be a custom hook function which would
> be invoked to locate the build directory for a given configuration. Also it
> should be possible for the initial build configuration to be determined by
> searching possible build directory locations. For example, you should be able
> to say "if there exists a build.dbg directory, use that as the build directory
> for debug builds, and set the current configuration to debug". I think this
> type of discovery is going to be necessary so that the compilation database is
> available for parsing when a file is first loaded.
My advice would be to start simple: Create an EDE project type which
accepts a filename which points to an existing compilation
database. Things like automatic creation of the database and
configuration detection can come later. Doing things automatically is
great and all, but the complexity that comes with it is painful.
> 2. Determine how to access the compilation database, and detect when it
> changes. This would typically involve watching a file in the current build
> directory which is updated whenever the compilation database is. For example,
> it could be the compile_commands.json file, or the rules.ninja file, or
> something else entirely. This should be customizable with a user option to
> specify how this is done, and also fall back to simple heuristics (eg if the
> compile_commands.json file is there, then use it).
Again: nice to have, can come later. :-)
> I don't want to use a JSON library if it is just going to build an AST
> in memory. Intsead I'd rather parse this in a streaming fashion. The
> compilation database entries can be stored in a hash table, keyed on
> the source file name.
Take a look at json.el that ships with Emacs. I've never used it, but it
seems to use a hash table for storage, so maybe it does what you need.
> The compilation database can in theory support more than one
> compilation command per file, but I think this is unlikely to be an
> issue when we have a build directory per configuration. If needed in
> future, we could defer to user hooks to disambiguate cases such as
I think configurations should be supported through an EDE
'configurations' slot, in accordance with how other EDE projects work.
> 4. Implement the various methods required to provide the include paths and
> preprocessor definitions, eg ede-system-include-path. These can probably be
> parsed as-needed from the compiler command line found in the hash table, and
> possibly cached. Initially I think a simple regular expression-based match
> would be sufficient to parse the compiler command line. These regular
> expressions would be customizable for different compilers, for example.
> However, given the compilation database is an LLVM standard, I would expect
> Clang to be the most commonly-used compiler.
Clang follows gcc, so you can support both easily. But anyway, things
like '-I', '-L' and '-D' are pretty much standard, at least on Unix, and
those are the most important information we need.
> 5. System include paths automatically added by the compiler (for standard
> library headers, for example) can be discovered by querying the compiler
> directly, as we do currently for clang and gcc. These can be stored in a
> global-scope variable, as they won't vary per project.
Yes, although getting this information from the compiler is so fast that
it really doesn't matter if every project does it for itself.
> If I am correct, it should be possible for this new project type to 'bootstrap'
> itself without requiring significant manual configuration. Hopefully, given a
> build directory, and some customizable heuristics, we can discover all we need
> to be able to useful things with C and C++ source files.
> The above items are a minimal useful subset of functionality, and also a
> suitable base onto which more features can be built.
As I've written, I think the minimal useful subset is way smaller, more
(ede-compdb-project "MY-PROJECT" :file "~/project/compilationdb.json")
which covers all files under "~/project". If I load a file from there,
the project would look into compilationdb.json and parse things like the
used compiler, include paths and preprocessor symbols, so that Semantic
can use them. At first, the user should take care of keeping this
database up-to-date. If EDE parses the file on-demand, it doesn't even
have to watch it.