#118 depslib adds unnecessary dependencies which cause unnecessary recompilation in some cases

[Riot]

This is quite an obscure issue so i'll explain the background - this has been causing me a headache for a while but I haven't until now been able to pin down anything concrete to report as a bug. Unmodified source files were being rebuilt repeatedly, when they should not have been, considerably lengthening recompile times of some projects.

I eventually tracked it down to a case where via a boost include, depslib was adding a dependency to myproject.depends for "boost/config/platform/vxworks.hpp", which in turn included a dependency on the non-existant <version.h>. However, since I was using the code::blocks autoversioning plugin, I had a local version.h and the directory it was in was part of the search path, so every time this changed (every recompile), every file that included those boost algorithms were also tagged by depslib as needing to be recompiled, incorrectly.

The trouble is that in boost, "boost/config/platform/vxworks.hpp" is included inside an #if 0 block, so it is never actually included at compile time. But depslib picks it up and processes it recursively, anyway.

The problem can be summarised thus:
Depslib flags potential dependencies even when they are not actually included due to #if choices, potentially finding files that have changed but do not really affect the project, and causing unnecessary repeated recompiles.

Apologies that this is long-winded but I hope I've been clear enough; I can provide a minimal project that illustrates this problem. It is quite a serious issue, as I discovered it when recompiles that should have taken only a few seconds were taking several minutes each time.

[Riot]

Note for reference: http://www.boost.org/doc/libs/1_57_0/boost/config/select_platform_config.hpp is where "boost/config/platform/vxworks.hpp" is included inside a #if 0 block (at the bottom of the file).

[Teodor Petrov]

Heh, another reason why boost should not be used:)

Posting an example project might help, but the fix is pretty hard to do right and reliably.

[Riot]

Arguably that's a fault with wxwidgets more than boost, but in any case there are any number of libraries that selectively include files that may not exist in some situations, in which case those includes will be omitted by #ifdefs. A good dependency scanner must be able to omit includes that would not be included at compile time.

If you'd like me to provide an example I can do, but it would be about five lines long and just look something like

#if 0
#include <non_existent_file.h>
#endif

and the problem would be shown by examining myproject.depends and seeing non_existent_file.h listed as a dependency. For a more colourful example, replace #include <non_existent_file.h> with #include "/dev/urandom" or #include "/proc/kcore" or whatever it takes to convince you that processing omitted includes is a terrible idea ;)

If you think this is an intractable problem for depslib then you could simply use gcc -M where gcc is available, which deals with this situation perfectly and accurately - and seemingly faster than the built-in depslib implementation.

[Teodor Petrov]

I've no idea how depslib works, so I can't comment much if this is fixable or not. I just guess that it is hard to handle. Also I don't know why cb is not using gcc -M, but I've never heard of complains that depslibs is slow.

[ollydbg]

Depslib just blindly scan each line of the file, and catch #include xxxx as its dependencies. It don't handle preprocessor directives.

Which IDE use dependencies generated by GCC?

[ollydbg]

It looks like Codelite use the dependencies from GCC. See:
Re: Compile doesn't show print out of hello world?
Also: CodeLite, an open source, cross platform C/C++ IDE | LiteEditor / ProjectSettingsSummary

Since CodeLite uses a makefile based system, you can pass complex expressions like $(shell wx-config --cflags)

[Riot]

Just a thought - code::blocks already tracks enabled #defines, since it has a default-on option to selectively grey out unused preprocessor code:

Since CB already has this info, how hard would it be to pass this on to depslib?

[Teodor Petrov]

In fact it is the scintilla component that tracks this info and not codeblocks itself.
Also it is know that scintilla's parser is not fully complete or bug free.

[Riot]

Fair enough; but i suspect it be less of a bug than having depslib rebuild based on changes to non-existent files, or running dependency scans over entire trees of files that are never actually included.

I really think the best solution is simply to use gcc -M's capability, which is already perfect for the job and parses defines correctly without having to manually reinvent the wheel here. It's the right tool for the job, and depslib's naive #include search is clearly behaving contrary to expectations here.

[Teodor Petrov]

At the moment I can only say patches welcome.

[ollydbg]

As I know, currently, CodeCompletion's parser can handle all the #include directive when parsing, but I don't know how to create a full file include hierarchy, so, let's wait for some one who can help to improve this feature.