linuxsh-dev

[PATCH] race in parallel out-of-tree build

[Johansson Erik-E. <eri...@mo...>]

sh: fix race in parallel out-of-tree build

Depending on which of the three dependencies for archprepare (in
arch/sh/Makefile) get built first, the directory include/asm-sh may or
may not exist when the maketools target is built. If the directory does
not exist, awk will fail to generate machtypes.h. This patch fixes this
by creating the directory before awk is executed.

Signed-off-by: Erik Johansson <eri...@mo...>
---
Diff from linux-2.6.17, but the file hasn't changed in sh-2.6.git tree,
so it should apply there as well.

--- linux-2.6.17/arch/sh/tools/Makefile~	2006-06-18
03:49:35.000000000 +0200
+++ linux-2.6.17/arch/sh/tools/Makefile	2007-06-19 11:20:05.000000000
+0200
@@ -12,4 +12,5 @@
=20
 include/asm-sh/machtypes.h: $(src)/gen-mach-types $(src)/mach-types
 	@echo '  Generating $@'
+	$(Q)if [ ! -d include/asm-sh ]; then mkdir -p include/asm-sh; fi
 	$(Q)$(AWK) -f $^ > $@ || { rm -f $@; /bin/false; }

Re: [PATCH] race in parallel out-of-tree build

[Paul M. <le...@li...>]

On Tue, Jun 19, 2007 at 11:55:55AM +0100, Johansson Erik-EJO017 wrote:
> Depending on which of the three dependencies for archprepare (in
> arch/sh/Makefile) get built first, the directory include/asm-sh may or
> may not exist when the maketools target is built. If the directory does
> not exist, awk will fail to generate machtypes.h. This patch fixes this
> by creating the directory before awk is executed.
> 
[snip]

> Diff from linux-2.6.17, but the file hasn't changed in sh-2.6.git tree,
> so it should apply there as well.
> 
I'm not sure what sh-2.6.git tree you're looking at, but mine already has
this for the .cpu and .mach cookies:

include/asm-sh/.cpu: $(wildcard include/config/cpu/*.h) \
                     include/config/auto.conf FORCE
        @echo '  SYMLINK include/asm-sh/cpu -> include/asm-sh/$(cpuincdir-y)'
        $(Q)if [ ! -d include/asm-sh ]; then mkdir -p include/asm-sh; fi

...

include/asm-sh/.mach: $(wildcard include/config/sh/*.h) \
                      include/config/auto.conf FORCE
        @echo -n '  SYMLINK include/asm-sh/mach -> '
        $(Q)if [ ! -d include/asm-sh ]; then mkdir -p include/asm-sh; fi
...

git-annotate points at cad8244840d1a148f638925758afd1cdf81fc839 when this
particular change happened:

cad8244840d1a148f638925758afd1cdf81fc839        (Paul Mundt     2006-01-16 22:14:19 -0800       153)    $(Q)if [ ! -d include/asm-sh ]; then mkdir -p include/asm-sh; fi

which happens to be:

commit cad8244840d1a148f638925758afd1cdf81fc839
Author: Paul Mundt <le...@li...>
Date:   Mon Jan 16 22:14:19 2006 -0800

    [PATCH] sh: Move CPU subtype configuration to its own Kconfig

...

which went in during the 2.6.16-rc1 merge window. So even your 2.6.17 tree
should have this. Both of these are also FORCE'd (which is also in the phony
targets), so the race you describe should be impossible. Confused.

RE: [PATCH] race in parallel out-of-tree build

[Johansson Erik-E. <eri...@mo...>]

> -----Original Message-----
> From: Paul Mundt [mailto:le...@li...]
> which went in during the 2.6.16-rc1 merge window. So even your 2.6.17
tree
> should have this. Both of these are also FORCE'd (which is also in the
> phony
> targets), so the race you describe should be impossible. Confused.

Yes, we have it in our tree as well. The problem is that when building
in parallel, the maketools target may be built before .cpu and .mach,
resulting in a missing asm-sh directory. FORCE only forces the targets
to be built every time, it doesn't force the ordering.

/Erik

Re: [PATCH] race in parallel out-of-tree build

[Paul M. <le...@li...>]

On Wed, Jun 20, 2007 at 07:58:55AM +0100, Johansson Erik-EJO017 wrote:
> > -----Original Message-----
> > From: Paul Mundt [mailto:le...@li...]
> > which went in during the 2.6.16-rc1 merge window. So even your 2.6.17
> > tree should have this. Both of these are also FORCE'd (which is also
> > in the phony targets), so the race you describe should be impossible.
> > Confused.
> 
> Yes, we have it in our tree as well. The problem is that when building
> in parallel, the maketools target may be built before .cpu and .mach,
> resulting in a missing asm-sh directory. FORCE only forces the targets
> to be built every time, it doesn't force the ordering.
> 
Hmm.. Have you actually seen this as a problem? An easy way to reproduce,
or at least a log of the error would be nice. I don't seem to have any
luck reproducing this. maketools depends on version.h, which is generated
by prepare1 in the top-level, which comes after archprepare in the .PHONY
list. The dependency ordering is explicit, in that each stage depends on
the previous one, so I still don't see anything wrong here. On the other
hand, if this is a valid bug, ARM also has this problem.

RE: [PATCH] race in parallel out-of-tree build

[Johansson Erik-E. <eri...@mo...>]

> -----Original Message-----
> From: Paul Mundt [mailto:le...@li...]
> Hmm.. Have you actually seen this as a problem? An easy way to
reproduce,
> or at least a log of the error would be nice. I don't seem to have any
> luck reproducing this. maketools depends on version.h, which is
generated
> by prepare1 in the top-level, which comes after archprepare in the
.PHONY
> list. The dependency ordering is explicit, in that each stage depends
on
> the previous one, so I still don't see anything wrong here. On the
other
> hand, if this is a valid bug, ARM also has this problem.

Output from building with -j 16 without my patch:
  GEN     <long_path>/kernel-debug/Makefile
  SYMLINK include/asm -> include/asm-sh
  CHK     include/linux/version.h
  Using <long_path>/linux-2.6.17 as source for kernel
  SPLIT   include/linux/autoconf.h -> include/config/*
  UPD     include/linux/version.h
  SYMLINK include/asm-sh/cpu -> include/asm-sh/cpu-sh4
  Generating include/asm-sh/machtypes.h
/bin/sh: include/asm-sh/machtypes.h: No such file or directory
  SYMLINK include/asm-sh/mach -> include/asm-sh/stb7100ref
make[3]: *** [include/asm-sh/machtypes.h] Error 1
make[2]: *** [maketools] Error 2
make[2]: *** Waiting for unfinished jobs....

The error above is what you get if you run (without an include/asm-sh
dir):
/bin/sh -c 'awk -f gen-mach-types mach-types >
include/asm-sh/machtypes.h'

The same, but with my patch:
  GEN     <long_path>/kernel-debug/Makefile
  SYMLINK include/asm -> include/asm-sh
  SPLIT   include/linux/autoconf.h -> include/config/*
  CHK     include/linux/version.h
  Using <long_path>/linux-2.6.17 as source for kernel
  UPD     include/linux/version.h
  Generating include/asm-sh/machtypes.h
  SYMLINK include/asm-sh/cpu -> include/asm-sh/cpu-sh4
  SYMLINK include/asm-sh/mach -> include/asm-sh/stb7100ref

In both cases you see that Generating include/asm-sh/machtypes.h comes
before creating the two symlinks (cpu and mach).

/Erik

Re: [PATCH] race in parallel out-of-tree build

[Paul M. <le...@li...>]

On Wed, Jun 20, 2007 at 09:54:16AM +0100, Johansson Erik-EJO017 wrote:
> Output from building with -j 16 without my patch:
>   GEN     <long_path>/kernel-debug/Makefile
>   SYMLINK include/asm -> include/asm-sh
>   CHK     include/linux/version.h
>   Using <long_path>/linux-2.6.17 as source for kernel
>   SPLIT   include/linux/autoconf.h -> include/config/*
>   UPD     include/linux/version.h
>   SYMLINK include/asm-sh/cpu -> include/asm-sh/cpu-sh4
>   Generating include/asm-sh/machtypes.h
> /bin/sh: include/asm-sh/machtypes.h: No such file or directory
>   SYMLINK include/asm-sh/mach -> include/asm-sh/stb7100ref
> make[3]: *** [include/asm-sh/machtypes.h] Error 1
> make[2]: *** [maketools] Error 2
> make[2]: *** Waiting for unfinished jobs....
> 
> The error above is what you get if you run (without an include/asm-sh
> dir):
> /bin/sh -c 'awk -f gen-mach-types mach-types >
> include/asm-sh/machtypes.h'
> 
Can you try this with current git to see if it's still a problem? I'm
unfortunately unable to reproduce this, even with a -j.

RE: [PATCH] race in parallel out-of-tree build

[Johansson Erik-E. <eri...@mo...>]

> -----Original Message-----
> From: Paul Mundt [mailto:le...@li...]
> Can you try this with current git to see if it's still a problem? I'm
> unfortunately unable to reproduce this, even with a -j.

I tried 2.6.22-rc5 (git failed to clone sh-2.6.git) and was able to
reproduce the problem. If you add sleep 2 as the first command in the
two targets include/asm-sh/{.cpu,.mach} you should see it, without
having to rely on timing.

/Erik

Re: [PATCH] race in parallel out-of-tree build

[Paul M. <le...@li...>]

On Wed, Jun 20, 2007 at 01:14:06PM +0100, Johansson Erik-EJO017 wrote:
> 
> > -----Original Message-----
> > From: Paul Mundt [mailto:le...@li...]
> > Can you try this with current git to see if it's still a problem? I'm
> > unfortunately unable to reproduce this, even with a -j.
> 
> I tried 2.6.22-rc5 (git failed to clone sh-2.6.git) and was able to
> reproduce the problem. If you add sleep 2 as the first command in the
> two targets include/asm-sh/{.cpu,.mach} you should see it, without
> having to rely on timing.
> 
Indeed, that breaks it. Patch applied, thanks.