Hi,
I would like to know if there is any way to add dependencies to the dependency list of a target during the execution of the rule.
In many cases, we know the list of dependency only after the command (i.e. on compilation), and i would line to update the dependency list after the command(s).
I know we have scanners for this, but I don't want to use it because:
- In most cases it's not accurate enough.
- Sometimes its just not possible to know in advance the dependencies (e.g. link command).
I thought of two possible solutions:
- Write a custom scanner an run it at the after the rule, not before.
- use a Perl wrapper to the command and use makepp internal functions to update the list during the rule execution.
I am trying to solve it myself, but it would be great if someone could provide the solution and spare me hours of searching in makepp code.
Thanks,
Yoni.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
the way you put your problem is a bit absurd, because a dependency is something that has to be built before starting to execute the rule that depends on it! As for the link command, with inexistent libraries and multiple -L options, it is a bit vague, so the solution can only be heuristic. Makepp gets around that by looking where it knows how to build the libraries, and doing that. Likewise for includes with multiple -I options.
If you don't want to write a scanner, you can do it ad hoc:
sub extract_dependencies {
$_ = /nice line/ ? "stuff: foo bar\n" : ''; # maybe a bit cleverer ;-)
}
include mystuff # this learned the rule stuff: foo bar
this has some disadvantages, because it means performing a rule right there when finding the include statement, but it's often much easier than writing a scanner.
best regards —
Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Daniel,
Thanks for the reply!
I beg to differ with you on this issue.
IMHO, in many cases the only accurate way to know (not guess) a target dependencies is after the command, and there is no reason not to update the dependency list after the command.
I Parse in my makefiles the output of gcc and cl.exe and I generate a file with a rule that add those dependencies to the target.
Then I "-include" this file to add the dependencies if it exist.
Because I cant add those dependencies after the command, when I run makepp for the second time, the list of dependencies change (since I included the the file I generated in the first time) and makepp decide to run the command again (and everything depends on this object get linked again).
Its true also if the I add an include in the c/h file, run makepp, compile because the stamp to the c/h file change, run makepp again, and makepp run the command again because the dependencies changed.
I don't (yet :-)) familiar enough with makepp code, but I think maybe adding a post_scanner will do the job.
Yoni.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> because a dependency is something that has to be built before starting to execute the rule
I would like to better explain the root of this, since its a very basic concept in build-systems in general (and specifically in Makefiles):
A build-system never knows BEFORE a command is executed the full dependencies. It will never know. The only way to ever know the full dependencies of a rule is to actually EXECUTE the rule's command.
Any attempt to try and understand the dependencies without executing the command will slowly lead you into building an full emulator of the command, which will take as much computation power (or more!) than actually executing the command, and will give you less accurate results.
I will give an example of C code compilation, but this is true for many other common building commands:
To build foo.o we obviously need foo.c.
But what about /usr/include/string.h? and all the other headers?
One way is to write an emulator for the command. Like the 'C scanner' of Makepp.
But this:
1. will require a lot of computational power, just as gcc's actual pre-preprocessing, or even more!
2. will never be accurate as gcc's -MD
3. in order to improve accuracy, you need to pass to the Makepp C-scanner more and more of gcc's options: the search direcroties (-I) etc. And even that is not enough to make it really correct!
In our system, we have sometimes our own implementation of certain headers, such as stdio.h, which gcc always knows the right file, whereas makepp thinks /usr/include/stdio.h is the one (its not!).
The same problem happens in linkage: when linking with an archive, the linker optimization only links to the obj files inside the archive that are actually in use. You CANNOT know this beforehand (unless you write a linker emulator…).
So - before running a rule for the first time, the Makefile system only needs to know the 'root' dependencies (foo.c in the above example).
Once executing the commands of the build of foo.o, it learns more on the dependencies foo.c brings in to the system (due to '#include').
This assists it next time to decide whether there is a need to rebuild.
… and same goes for linking: first time it just executes the command. Second time, it already knows (from the first execution) the linker's dependency on *.a (and objs inside it), and can use this info to decide whether a re-build is needed.
In traditional GNUmake/BSDMake we solved this problem by generating a dependency file as a side affect of every compilation (using -MD) and linkage (using the linkage map file), and using "-include foo.o.dep" to include these dependencies the second time around.
In Makepp's concept of working, executing a command which 'learns' more accurate dependency info while executing should have an ability to update the rule's dependency at the post-rule level.
FYI: we already wrote wrappers also for MSVC compiler that generates dependency information of compilation and linkage, so this method is rather cross-platform.
Yoni.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ah, I see, you want a bootstrap run, from which to auto-configure your actual build.
That's a dilemma, because mpp guarantees reliable builds, so you can't change stuff and pretend it didn't change (except for last resort mechanisms like -dont-build). With your approach, I'd be much more comfortable with rerunning the rule after changed dependencies. It should change only in rare cases, where mpp guesses wrong, and even then only once, so what the heck…
In simple cases your approach might be fine, but in our build we use all of swig, corba-idl, sql-preprocessor and template generated sources and headers, as well as a dozen shared libraries we build. These would never be found by -MD, because they're not there before mpp discovers them as dependencies and builds them — which has to happen **before** running the rule that depends on them! This kind of thing was hard to do with gmake, but you get used to the comfort and don't want to miss it :-)
As for the correctness, mpp is quite good about general compiler options, as well as specifically about many of the crazy gcc-options, but you're right: it can't be 100% sure about it all. Because of this you have the :smart_scan rule option, which lets the precompiler (much cheaper than the actual compiling) do the work, based on your actual command.
Not sure what you mean about stdio.h. If you have your own, you should include it with "stdio.h", not <stdio.h>. But even if you don't, mpp would only not build it, because you say you wan't the system one. But if it is present in a -I directory or in the same one as the including source, and no -I-, it will of course be noticed as a dependency.
regards — Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've played around with the necessary post-processing a bit, since it might be interesting, even if it cuts you off from some of makepp's superior features. But someone who only has a traditional static setup, without any source generator or convenient linking of headers to one central include directory might actually profit from this, because it might be faster than having mpp do the scanning.
My proposed solution is a rule modifier :include. This would have the effect of the -include statement when reading the rule, and again of the include statement if the file changed after successfully executing the rule, but before writing the metadata. So as of the 2nd run mpp would rebuild only if dependencies change.
Not sure about % yet, but something along the lines of
Hi,
I would like to know if there is any way to add dependencies to the dependency list of a target during the execution of the rule.
In many cases, we know the list of dependency only after the command (i.e. on compilation), and i would line to update the dependency list after the command(s).
I know we have scanners for this, but I don't want to use it because:
- In most cases it's not accurate enough.
- Sometimes its just not possible to know in advance the dependencies (e.g. link command).
I thought of two possible solutions:
- Write a custom scanner an run it at the after the rule, not before.
- use a Perl wrapper to the command and use makepp internal functions to update the list during the rule execution.
I am trying to solve it myself, but it would be great if someone could provide the solution and spare me hours of searching in makepp code.
Thanks,
Yoni.
Dear Yoni,
the way you put your problem is a bit absurd, because a dependency is something that has to be built before starting to execute the rule that depends on it! As for the link command, with inexistent libraries and multiple -L options, it is a bit vague, so the solution can only be heuristic. Makepp gets around that by looking where it knows how to build the libraries, and doing that. Likewise for includes with multiple -I options.
If you don't want to write a scanner, you can do it ad hoc:
sub extract_dependencies {
$_ = /nice line/ ? "stuff: foo bar\n" : ''; # maybe a bit cleverer ;-)
}
mystuff.makepp: mystuff.input
# whatever commands to create dependendency statements
&grep &extract_dependencies $(input) -o $(output)
include mystuff # this learned the rule stuff: foo bar
this has some disadvantages, because it means performing a rule right there when finding the include statement, but it's often much easier than writing a scanner.
best regards —
Daniel
Hi Daniel,
Thanks for the reply!
I beg to differ with you on this issue.
IMHO, in many cases the only accurate way to know (not guess) a target dependencies is after the command, and there is no reason not to update the dependency list after the command.
I Parse in my makefiles the output of gcc and cl.exe and I generate a file with a rule that add those dependencies to the target.
Then I "-include" this file to add the dependencies if it exist.
Because I cant add those dependencies after the command, when I run makepp for the second time, the list of dependencies change (since I included the the file I generated in the first time) and makepp decide to run the command again (and everything depends on this object get linked again).
Its true also if the I add an include in the c/h file, run makepp, compile because the stamp to the c/h file change, run makepp again, and makepp run the command again because the dependencies changed.
I don't (yet :-)) familiar enough with makepp code, but I think maybe adding a post_scanner will do the job.
Yoni.
Hi Daniel,
> because a dependency is something that has to be built before starting to execute the rule
I would like to better explain the root of this, since its a very basic concept in build-systems in general (and specifically in Makefiles):
A build-system never knows BEFORE a command is executed the full dependencies. It will never know. The only way to ever know the full dependencies of a rule is to actually EXECUTE the rule's command.
Any attempt to try and understand the dependencies without executing the command will slowly lead you into building an full emulator of the command, which will take as much computation power (or more!) than actually executing the command, and will give you less accurate results.
I will give an example of C code compilation, but this is true for many other common building commands:
foo.o: foo.c
gcc -c -o foo.o foo.c
and foo.c is:
#include <string.h>
#include <stdio.h>
#ifdef __linux__
#include <sys/ip.h>
#else
#include <netinet/ip.h>
#end
…
To build foo.o we obviously need foo.c.
But what about /usr/include/string.h? and all the other headers?
One way is to write an emulator for the command. Like the 'C scanner' of Makepp.
But this:
1. will require a lot of computational power, just as gcc's actual pre-preprocessing, or even more!
2. will never be accurate as gcc's -MD
3. in order to improve accuracy, you need to pass to the Makepp C-scanner more and more of gcc's options: the search direcroties (-I) etc. And even that is not enough to make it really correct!
In our system, we have sometimes our own implementation of certain headers, such as stdio.h, which gcc always knows the right file, whereas makepp thinks /usr/include/stdio.h is the one (its not!).
The same problem happens in linkage: when linking with an archive, the linker optimization only links to the obj files inside the archive that are actually in use. You CANNOT know this beforehand (unless you write a linker emulator…).
So - before running a rule for the first time, the Makefile system only needs to know the 'root' dependencies (foo.c in the above example).
Once executing the commands of the build of foo.o, it learns more on the dependencies foo.c brings in to the system (due to '#include').
This assists it next time to decide whether there is a need to rebuild.
… and same goes for linking: first time it just executes the command. Second time, it already knows (from the first execution) the linker's dependency on *.a (and objs inside it), and can use this info to decide whether a re-build is needed.
In traditional GNUmake/BSDMake we solved this problem by generating a dependency file as a side affect of every compilation (using -MD) and linkage (using the linkage map file), and using "-include foo.o.dep" to include these dependencies the second time around.
In Makepp's concept of working, executing a command which 'learns' more accurate dependency info while executing should have an ability to update the rule's dependency at the post-rule level.
FYI: we already wrote wrappers also for MSVC compiler that generates dependency information of compilation and linkage, so this method is rather cross-platform.
Yoni.
Hi Yoni,
Ah, I see, you want a bootstrap run, from which to auto-configure your actual build.
That's a dilemma, because mpp guarantees reliable builds, so you can't change stuff and pretend it didn't change (except for last resort mechanisms like -dont-build). With your approach, I'd be much more comfortable with rerunning the rule after changed dependencies. It should change only in rare cases, where mpp guesses wrong, and even then only once, so what the heck…
In simple cases your approach might be fine, but in our build we use all of swig, corba-idl, sql-preprocessor and template generated sources and headers, as well as a dozen shared libraries we build. These would never be found by -MD, because they're not there before mpp discovers them as dependencies and builds them — which has to happen **before** running the rule that depends on them! This kind of thing was hard to do with gmake, but you get used to the comfort and don't want to miss it :-)
As for the correctness, mpp is quite good about general compiler options, as well as specifically about many of the crazy gcc-options, but you're right: it can't be 100% sure about it all. Because of this you have the :smart_scan rule option, which lets the precompiler (much cheaper than the actual compiling) do the work, based on your actual command.
Not sure what you mean about stdio.h. If you have your own, you should include it with "stdio.h", not <stdio.h>. But even if you don't, mpp would only not build it, because you say you wan't the system one. But if it is present in a -I directory or in the same one as the including source, and no -I-, it will of course be noticed as a dependency.
regards — Daniel
I've played around with the necessary post-processing a bit, since it might be interesting, even if it cuts you off from some of makepp's superior features. But someone who only has a traditional static setup, without any source generator or convenient linking of headers to one central include directory might actually profit from this, because it might be faster than having mpp do the scanning.
My proposed solution is a rule modifier :include. This would have the effect of the -include statement when reading the rule, and again of the include statement if the file changed after successfully executing the rule, but before writing the metadata. So as of the 2nd run mpp would rebuild only if dependencies change.
Not sure about % yet, but something along the lines of
%.o: %.cpp :include $(stem).d :scanner none
g++ -c -MD $(input)
or maybe even
%.o: %.cpp :include %.d :scanner none
g++ -c -MD $(input)
Does that sound like it might solve your problem?
Ok, the 2nd variant described above works with the new version in cvs.