We're in the progress if improving our build times since some of our machines take up to 1 hour building the current code. Since there is a lot of redundancy files being compiled then we could save up to 60% of the time if we would recompile all of them over and over again for incremental builds.
After spending several days with frustrations then we reached something that partially works and gives a lot of troubles.
Our buildtree is organised as follows:
/ <- also called top
/shared <- files shared between all projects
/project1/shared <- files shared within a single project
/project1/app1 <- actual application that uses files
Today we build in a temp directory located in /project1/app1:
While it works decently there is a fair amount of mystery going on:
1. sometimes the files in the shared cache are rebuilt without reason. Makepp says that this happen because the build directory has changed. So essentially since the path (likely the relative path to the top of the build tree) changed it will recompile it even though it should not. I tried to come around this by using absolute filenames instead but it didn't help so far.
2. I get a thousand warnings like:
makepp: warning: rule `/cygdrive/c/our_software/top.mk:188(shared/generated)' produces /cygdrive/c/our_software/shared/generated/temp/win32_x86/bn_s_mp_sub.obj in two different ways
It seems that makepp gets confused about itself. This warning comes when loading the other makefiles at demand. However I double-verified that there are no duplicate rules except for the one listed here. So it gets confused about the same line (perhaps again due to relative path changes).
Can anyone guide us forward or do we need to give up this project? I should note for the record that it's not something which is easy to debug as there are about 20 makefiles involved.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
absolute pathes are a bad thing, unless you will only build in exactly that constellation. Else repositories and build caches will perform below target, because things will be different, depending on where you are.
I'm not exactly sure I get what you're doing, but it seems to me that each subproject has the same rules to build the common part. Thus, when you build that from a diferent subproject, the directory changes, and makepp notices that something is different, so it had better rebuild.
When I suggested this split, I was thinking about an autonomous common part, that would contain the rules to build itself. And the build fo common would actually happen there. The other projects would simply use what is there, without trying to build it. This is achieved for example by each part having its own RootMakeppfile, so that everything outside it is automatically --dont-build (and present by the time it isd needed).
best regards
Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Actually I found that there is a buildcache keyword in makepp already... Wanting to try it out it turned out not to work though.
I followed it manual but it keeps saying:
set MAKEPPFLAGS=--build-cache=c:\\temp
c:\myapp>makepp
makepp: no update necessary
makepp: error: Build cache c:\temp does not have a valid format
c:\temp/build_cache_options.pl is missing or corrupted
It seems not to work properly under cygwin.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok I realized I was silly since I manually needed to create the cache. However it seems not to be possible to set a maximum size on it. That would be really handy as we plan to use it on a ramdisk which obviously is not infinite in size.
What happens if it runs out of space? Will it delete old items and try again or does it just fail?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There is no such thing as you ask for, though it might be interesting to add. I don't really know how to specify it as args, because the other args give the cleaning strategy. They're an absolute limit now, irrespective of the resulting size, but with the new option you suggest the'd become a target to get near to. So don't count on it soon. As a workaround you could call mppbcc clean repeatedly, with ever more drastic criteria, until the output of "du -s <bc-dir>" is small enough. In any case, this is just an approximation of your needs, because cleaning can only happen periodically, and in the meantime a full build with some new options can fill the cache fast.
There is not much of an advantage in putting the cache into ram, unless your build trees are also there, such that the built files just get linked in. See Unsafe disk under http://makepp.sourceforge.net/1.50/makepp_build_cache.html#build_cache_grouping -- grouping is very interesting for keeping the build cache persistent even while profiting from the ram disk. Everybody will read and write to the ramdisk, without loosing anything. You must repair the group, i.e. recreate it after every reboot of course, and preferably call clean immediately, as that performs the replication.
best regards
Daniel
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We're in the progress if improving our build times since some of our machines take up to 1 hour building the current code. Since there is a lot of redundancy files being compiled then we could save up to 60% of the time if we would recompile all of them over and over again for incremental builds.
After spending several days with frustrations then we reached something that partially works and gives a lot of troubles.
Our buildtree is organised as follows:
/ <- also called top
/shared <- files shared between all projects
/project1/shared <- files shared within a single project
/project1/app1 <- actual application that uses files
Today we build in a temp directory located in /project1/app1:
$(TMP)/%.$(O): %.cpp
$(COMP_CXX)
$(TMP)/%.$(O): $(SHAREDGENERICDIR)/%.cpp
$(COMP_CXX)
Here both local and shared files goes into the same dir.
We want however to put the shared files in a common dir. So I changes the rules to:
$(TOP_SHARED_TMP)/%.$(O): $(SHAREDGENERICDIR)/%.cpp
$(COMP_CXX)
While it works decently there is a fair amount of mystery going on:
1. sometimes the files in the shared cache are rebuilt without reason. Makepp says that this happen because the build directory has changed. So essentially since the path (likely the relative path to the top of the build tree) changed it will recompile it even though it should not. I tried to come around this by using absolute filenames instead but it didn't help so far.
2. I get a thousand warnings like:
makepp: warning: rule `/cygdrive/c/our_software/top.mk:188(shared/generated)' produces /cygdrive/c/our_software/shared/generated/temp/win32_x86/bn_s_mp_sub.obj in two different ways
It seems that makepp gets confused about itself. This warning comes when loading the other makefiles at demand. However I double-verified that there are no duplicate rules except for the one listed here. So it gets confused about the same line (perhaps again due to relative path changes).
Can anyone guide us forward or do we need to give up this project? I should note for the record that it's not something which is easy to debug as there are about 20 makefiles involved.
Hi Henrik,
absolute pathes are a bad thing, unless you will only build in exactly that constellation. Else repositories and build caches will perform below target, because things will be different, depending on where you are.
I'm not exactly sure I get what you're doing, but it seems to me that each subproject has the same rules to build the common part. Thus, when you build that from a diferent subproject, the directory changes, and makepp notices that something is different, so it had better rebuild.
When I suggested this split, I was thinking about an autonomous common part, that would contain the rules to build itself. And the build fo common would actually happen there. The other projects would simply use what is there, without trying to build it. This is achieved for example by each part having its own RootMakeppfile, so that everything outside it is automatically --dont-build (and present by the time it isd needed).
best regards
Daniel
Actually I found that there is a buildcache keyword in makepp already... Wanting to try it out it turned out not to work though.
I followed it manual but it keeps saying:
set MAKEPPFLAGS=--build-cache=c:\\temp
c:\myapp>makepp
makepp: no update necessary
makepp: error: Build cache c:\temp does not have a valid format
c:\temp/build_cache_options.pl is missing or corrupted
It seems not to work properly under cygwin.
Actually I get the same error under linux... so it seems to be broken.
Ok I realized I was silly since I manually needed to create the cache. However it seems not to be possible to set a maximum size on it. That would be really handy as we plan to use it on a ramdisk which obviously is not infinite in size.
What happens if it runs out of space? Will it delete old items and try again or does it just fail?
Hi Henrik,
read the part on the subcommand clean in http://makepp.sourceforge.net/1.50/makepp_build_cache.html#how_to_manage_a_build_cache
There is no such thing as you ask for, though it might be interesting to add. I don't really know how to specify it as args, because the other args give the cleaning strategy. They're an absolute limit now, irrespective of the resulting size, but with the new option you suggest the'd become a target to get near to. So don't count on it soon. As a workaround you could call mppbcc clean repeatedly, with ever more drastic criteria, until the output of "du -s <bc-dir>" is small enough. In any case, this is just an approximation of your needs, because cleaning can only happen periodically, and in the meantime a full build with some new options can fill the cache fast.
There is not much of an advantage in putting the cache into ram, unless your build trees are also there, such that the built files just get linked in. See Unsafe disk under http://makepp.sourceforge.net/1.50/makepp_build_cache.html#build_cache_grouping -- grouping is very interesting for keeping the build cache persistent even while profiting from the ram disk. Everybody will read and write to the ramdisk, without loosing anything. You must repair the group, i.e. recreate it after every reboot of course, and preferably call clean immediately, as that performs the replication.
best regards
Daniel