|
From: Paul M. <pa...@sa...> - 2004-08-20 01:10:09
|
I have a patch that starts the work of abstracting out the architecture-specific parts of Valgrind into their own files, in order to enable them to be replaced on other architectures (e.g. PowerPC :). The patch is at: http://ozlabs.org/~paulus/val-abstraction-040820.patch.gz The patch is against current CVS. I haven't included it inline since it is large (~56k compressed), but I would be happy to email to anyone on request. The major change introduced by the patch is to replace all the architecture-specific members of ThreadState with an `arch' member, whose type is defined in an arch-specific header. It also moves the handling of those fields into functions in an arch-specific source file. The patch doesn't add any support for other architectures. I have added a coregrind/arch/x86-common directory. I have not moved existing files in this patch except for moving vg_unistd.h into coregrind/arch/x86-linux. There are quite a few files in coregrind that would be candidates for moving into x86-linux or x86-common later. I would really appreciate it if some people could review the patch fairly soon. I know it's a large patch, but people keep making changes in CVS and therefore the patch will bitrot moderately quickly. This patch is a start but there is still more that needs to be abstracted, particularly in vg_translate.c. If this patch could be applied to CVS it will make subsequent patches easier. With this patch, Valgrind compiles and runs on x86 (on my P4 running Debian sid and a 2.6 kernel) and passes exactly the same set of tests as current CVS does. Paul. |
|
From: Nicholas N. <nj...@ca...> - 2004-08-20 11:47:31
Attachments:
automake.log
|
On Fri, 20 Aug 2004, Paul Mackerras wrote:
> I have a patch that starts the work of abstracting out the
> architecture-specific parts of Valgrind into their own files, in order
> to enable them to be replaced on other architectures (e.g. PowerPC :).
Good stuff.
I am extremely sympathetic to the reasoning behind this patch; having
been looking at the x86-64 port, I see how necessary the restructuring is.
Once it's in, assuming it's done well, it will make working on ports much
easier, because you're much less likely to have clashes. And on the
whole, at first look the patch looks to me, overall, very much on the
right track.
First thing: I applied the patch fine, but had automake troubles building
it. Automake output is attached. Any idea what is the problem? Perhaps
it would be easier to put the whole tarball up?
I have lots of comments, but I'll put the important things, which everyone
should read, first. This stuff is IMPORTANT, and should be read by anyone
at all interested in the various Valgrind ports.
First is about commit philosophy. For changes like this, is it better to
slip these things in gradually, or do it all in one big hit? I prefer the
former; in fact I've already started doing it for Opteron a bit (eg. my
committting of the 64-bit clean version of the allocator). I like this
because it's easier to catch mistakes when using small steps, and the code
tends to get scrutinised more. Anyone disagree?
Related to this; development is proceeding (and will continue to proceed
AFAICT) at such a rate that we're in danger of never putting out any
stable releases, which is bad. We keep getting bug reports for bugs in
2.0.0 that are fixed in 2.1.X, which shows people are still using 2.0.0.
Second and more important thing is that I think it is *crucial* that we
get the file/directory structure right, at the start, since this is
difficult to change once it's in place; and this is something that should
be done in one hit. The contents of the actual files aren't so hard to
change, and so can be done bit by bit.
There are two aspects to this -- the directory structure, and then the
interface between the generic bits and the platform-specific bits. And by
interface, I don't mean at the level of functions and variables, rather at
the level of files, ie. which header files get included from
platform-specific-land into generic-land, and how the platform-specific
code is compiled and linked with generic code.
For example, Paul has used coregrind/arch/x86-common/vg_archdefs.h, which
contains things like the arch-specific part of the ThreadState, which all
$ARCH-common directories will require. But it also contains various
things that I think will only be used within the x86-specific code, and so
the core doesn't need to see.
Also, I don't like the use of #ifdefs within the Makefile.am files.
Dealing with different platforms via #ifdefs leads to night-sweats,
feeble-mindedness and delirium.
So, we need to thrash this out a bit. To start the discussion, here's a
proposal, which shares some features of Paul's patch, but also has some
important differences.
----
Nick's proposal
---------------
Terminology: A "platform" is a specific $ARCH/$OS pair. For every arch
and OS, there will be 4 code parts, in order of genericity:
1. some generic code (aka common-common)
2a. some arch-specific, OS-independent code (aka $ARCH-common)
2b. some arch-independent, OS-specific code (aka common-$OS)
3. some platform-specific code (aka $ARCH-$OS)
2a, 2b and 3 are all "non-generic".
The keys ideas are:
- to very carefully identify which category all code belongs to
- to minimise the interface between generic and non-generic code
- to minimise code duplication by pushing code into the most generic code
part possible
Abstract directory structure:
valgrind/
coregrind/
arch/
arch_common_defs.h (contains arch-specific definitions)
common_os_defs.h (contains OS-specific definitions)
arch_os_defs.h (contains platform-specific definitions)
arch_common@ (symlink to $ARCH-common/, made at config-time)
common_os@ (symlink to common-$OS/, made at config-time)
arch_os@ (symlink to $ARCH-$OS/, made at config-time)
$ARCH-common/
arch_common_macros.h (contains arch-specific macros used by core)
arch_common.a (contains all arch-specific code)
common-$OS/
common_os_macros.h (contains OS-specific macros used by core)
common_os.a (contains all OS-specific code)
$ARCH-$OS/
arch_os_macros.h (contains platform-specific macros used by core)
arch_os.a (contains all platform-specific code)
$ARCH and $OS would, of course, be replaced with real arch and OS names,
eg. x86-linux.
Note that each part's .h is necessarily split into defs and macros; the
defs .h can be shared by all sub-dirs, whereas each sub-dir has to provide
its own version of every macro. This is because functions/variables can
be defined/declared in different places, but macros can't. This is the
"minimise duplication" principle at work.
The generic part would need only these six #includes:
#include "platform/arch_common_defs.h"
#include "platform/common_os_defs.h"
#include "platform/arch_os_defs.h"
These could be put in vg_include.h and thus be made visible to all core
code. Or not.
Each of the _defs.h files would have the following respective #includes:
#include "platform/arch_common/arch_common_macros.h"
#include "platform/common_os/common_os_macros.h"
#include "platform/arch_os/arch_os_macros.h"
in order to pull in the relevant _macros.h files -- the symlinks would
ensure the right ones are pulled in.
When building stage2, coregrind/Makefile.am would just link all the core
modules with:
- platform/arch_common/arch_common.a
- platform/common_os/common_os.a
- platform/arch_os/arch_os.a
The symlinks would again ensure this works. These .a files would be made
by linking together all the .o files in each part.
Each non-generic sub-dir can also include as many other .c, .h and .S
files as it needs (plus anything else, like ld scripts). But the generic
part wouldn't see or care about that.
Note that arch/ is a really bad name. I'm not convinced we even need that
subdirectory between coregrind/ and the non-generic subdirs -- it will
just create more typing, for little benefit.
I haven't considered skins at all here. Perhaps each of the the proposed
headers needs to be split into two halves: one for the skin-visible
stuff, and one for the core-visible stuff (which would #include the
skin-visible header).
I haven't considered asm code at all here; some .S files need some
definitions, but you can't have any macros included in .S files.
(Note for the above two paragraphs: this skin/non-skin and asm/non-asm
header split is why we currently have four main headers: vg_include.h,
vg_skin.h, vg_constants.h and vg_constants_skin.h.)
I haven't considered the UME stuff that is shared between stage1 and
stage2. Not sure about that, it would probably be kept separate at least
to some extent.
For tools that need to do non-generic things (which should be avoided as
much as possible, but may be inevitable in small amounts, esp. for
Memcheck) parts of the structure can be mirrored as necessary within the
tool/ (eg. memcheck/) directory.
The names might be a bit confusing, in particular I'm not sure about the
wisdom of reusing "common" in the names "arch-common" and "common-os".
Perhaps the following would be clearer (it would also be less typing):
- arch-common --> arch
- common-os --> os
- arch-os --> arch-os
Eg. x86/, ppc/, x86-linux/, linux/, instead of x86-common/, ppc-common/,
x86/linux, common-linux.
I haven't used the "vg_" prefix in filenames above -- it's not necessary
just creates more typing, and the more recently added files don't use it.
Does that make sense? It feels very idealistic, but then, it feels very
clean and generic too, and I can't see a better way to do it. The nice
thing is that although the actual code/definitions/macros might get moved
around between the four parts over time, I believe the file/dir structure,
#include technique, and build technique would not need to change.
My preference would be to create this structure (or whatever we decide on)
in one hit, and then start moving things around, ie. commit the contents
of Paul's patch small steps at a time.
Comments are welcome!
----
Ok, now for various lower-level comments about the patch, mostly aimed at
Paul.
- Have you tried fitting your PPC port into this structure? Does it fit?
That's a good test.
- You haven't touched the x86/ directory, I see. Any reason why not?
- The syscalls are going to need a whole lot more abstraction/factoring
than what you've done with mmap(), but as you said the patch is only a
start.
- You've used VG_(arch_...) for a lot of names. I think a new prefix, eg.
VGA_(...) would be good. We could even use different prefixes for
definitions in the arch-common, common-os and arch-os interface, and then
a different prefix for code within those non-generic parts that isn't seen
by the generic code. Or maybe that's going too far, but something like
that would be good.
- VG_(arch_deliver_signal)() is big -- is it possible to factor out the
generic/non-generic parts better?
- --pointer-check is not necessarily x86-specific. The current mechanism
(using segments) is x86-specific, but it could also be done with eg.
explicit pointer-masking.
- why are arch_set_arg_and_ret() and arch_thread_initial_stack() not
VG_() prefixed?
- There doesn't seem to be much consistency in the file-naming scheme, eg.
some have "arch" in them, some have "x86", some don't. My proposal above
partly addresses this.
- I'm not happy about vg_libpthread.c being in x86-$OS/ subdirs -- huge
parts will be shared with the x86-freebsd, etc. But maybe you copied that
from the current (misleading) coregrind/arch/x86-$OS/ directories, the
contents of which aren't actually used.
- If this patch was done in small steps, is there any order you'd
prefer/suggest?
- It would be great to know if this matches well with Doug's FreeBSD port.
That's probably enough from me for the moment :)
N |
|
From: Paul M. <pa...@sa...> - 2004-08-20 13:18:29
|
Nicholas Nethercote writes: > First thing: I applied the patch fine, but had automake troubles building > it. Automake output is attached. Any idea what is the problem? Perhaps > it would be easier to put the whole tarball up? Just to answer this bit quickly: I have been using aclocal-1.7 and automake-1.7, because I had problems with earlier versions. I'm a long way from being an automake guru. :-P > Also, I don't like the use of #ifdefs within the Makefile.am files. > Dealing with different platforms via #ifdefs leads to night-sweats, > feeble-mindedness and delirium. I looked at the automake manual and that was the only obvious way to do it (well the only way that was obvious to me :), but as I said, I know little about automake. Is anyone on the list an automake expert? (I much prefer the kernel's way of dealing with multiple platforms, actually.) Paul. |
|
From: Nicholas N. <nj...@ca...> - 2004-08-20 14:08:43
|
On Fri, 20 Aug 2004, Paul Mackerras wrote: > I'm a long way from being an automake guru. :-P We all are... :( >> Also, I don't like the use of #ifdefs within the Makefile.am files. >> Dealing with different platforms via #ifdefs leads to night-sweats, >> feeble-mindedness and delirium. > > I looked at the automake manual and that was the only obvious way to > do it (well the only way that was obvious to me :), but as I said, I > know little about automake. Is anyone on the list an automake expert? > (I much prefer the kernel's way of dealing with multiple platforms, > actually.) What is the kernel's way of dealing with it -- I thought it was with symlinks, which is what I suggested in my dir/file proposal. Does Valgrind differ in some way from the kernel in this respect? N |
|
From: Paul M. <pa...@sa...> - 2004-08-21 01:59:11
|
Nicholas Nethercote writes:
> What is the kernel's way of dealing with it -- I thought it was with
> symlinks, which is what I suggested in my dir/file proposal. Does
> Valgrind differ in some way from the kernel in this respect?
There is a symlink for include files: include/asm points to
asm-${ARCH}.
The kernel makefile system was completely reengineered for 2.6. I've
heard the new system described as a better make implemented in the
GNU-make language. :) It's pretty complex and I don't understand it
fully, but it works nicely. One of the big changes from 2.4 was that
we no longer recursively invoke make in subdirectories. Instead the
one instance of make does all the compiling and linking required.
The top-level makefile determines what architecture we are on and
assigns that to ${ARCH}. If it wasn't set by the user (for
cross-compiling) we get it from $(shell uname -m). It then includes
arch/$(ARCH)/Makefile. There's also a Makefile in each of the
top-level directories that get included. Those can add values to
obj-y and obj-m, which are either the names of .o files to build and
link in or directory names with a trailing /.
The reason for the obj-y and obj-m names is that you can then do
obj-$(CONFIG_FOO) += foo.o
where CONFIG_FOO can be n, m or y. Then there are other rules that
build in everything listed in obj-y and build modules for everything
listed in obj-m. Along the way it creates a .foo.o.cmd for every
foo.o that contains the dependencies for foo.o.
I don't think we want to use the kernel Makefile system directly but
we can possibly get some good ideas from it. One of the big
differences is that we are building a lot of separate things in
valgrind whereas the kernel makefiles are mainly aimed at building
vmlinux (though to be fair, quite a few other things get built along
the way).
Are people wedded to the GNU standards for Makefiles and/or to using
automake? If so then we absolutely need an automake expert.
Otherwise we could try to come up with something hopefully a bit
simpler that would do what we need.
The main challenges I see in building valgrind are:
* Setting up symlinks so that the headers for the architecture we're
building for get included. On the whole I prefer symlinks to using
multiple -I flags (but I don't know how to get automake to create
the symlinks).
* Building and linking in the .o files for the architecture. This
could be done by recursing into ${ARCH}-${OS} and ${ARCH}-common
directories, or it could be done just by including Makefile
fragments from those directories.
* Autogenerating dependencies.
* Having global CFLAGS that depend on the architecture and then having
extra flags for some objects.
Paul.
|
|
From: KJK::Hyperion <no...@li...> - 2004-08-27 01:33:40
|
At 03.59 21/08/2004, Paul Mackerras wrote: >* Setting up symlinks so that the headers for the architecture we're >building for get included. On the whole I prefer symlinks to using >multiple -I flags (but I don't know how to get automake to create the >symlinks). I know my weight is zero until I actually produce a Windows port, but I don't like the idea of (ab?)using symlinks to have "prettier" command lines (on Windows 2000 and later you can create symlinks to directories only and on NTFS only, though) |
|
From: Paul M. <pa...@sa...> - 2004-08-21 06:03:12
|
Nicholas Nethercote writes: > First is about commit philosophy. For changes like this, is it better to > slip these things in gradually, or do it all in one big hit? I prefer the > former; in fact I've already started doing it for Opteron a bit (eg. my > committting of the 64-bit clean version of the allocator). I like this > because it's easier to catch mistakes when using small steps, and the code > tends to get scrutinised more. Anyone disagree? I agree, I think that things should be checked in in the smallest reasonable chunks. (For some things that might still end up being moderately large though.) > Second and more important thing is that I think it is *crucial* that we > get the file/directory structure right, at the start, since this is > difficult to change once it's in place; and this is something that should > be done in one hit. The contents of the actual files aren't so hard to > change, and so can be done bit by bit. It's a pity that CVS's support for moving files is so poor, otherwise moving files around wouldn't be so much of a hassle. Intellectually I don't think that the location of a file is difficult to change, but I agree that it usually ends up being a hassle in practice. > Also, I don't like the use of #ifdefs within the Makefile.am files. > Dealing with different platforms via #ifdefs leads to night-sweats, > feeble-mindedness and delirium. /me anxiously checks himself for symptoms. :) > Nick's proposal > --------------- > > Terminology: A "platform" is a specific $ARCH/$OS pair. For every arch > and OS, there will be 4 code parts, in order of genericity: > > 1. some generic code (aka common-common) > 2a. some arch-specific, OS-independent code (aka $ARCH-common) > 2b. some arch-independent, OS-specific code (aka common-$OS) > 3. some platform-specific code (aka $ARCH-$OS) > > 2a, 2b and 3 are all "non-generic". Yes. There may also be some code that depends on the executable format which is shared across several platforms, for example, ELF is used on x86-linux, ppc-linux, x86-freebsd I presume, but not ppc-macosx (which AFAIK uses mach-o). Not that I know of anybody doing a MacOS X port at the moment, but it seems an obvious thing for somebody to attempt after the ppc-linux port is integrated. > The keys ideas are: > > - to very carefully identify which category all code belongs to > - to minimise the interface between generic and non-generic code > - to minimise code duplication by pushing code into the most generic code > part possible Yes, this is certainly the goal. I don't think we are going to get this right from the beginning though, so we will have to allow it to evolve. [description of a directory structure snipped...] Looked good to me, although you don't say what to do with the platform-specific definitions in the files in include/. Do we have a set of platform directories under include as well as the set under coregrind, or do we try to keep it down to one set of platform- specific directories? > The names might be a bit confusing, in particular I'm not sure about the > wisdom of reusing "common" in the names "arch-common" and "common-os". > Perhaps the following would be clearer (it would also be less typing): > > - arch-common --> arch > - common-os --> os > - arch-os --> arch-os > > Eg. x86/, ppc/, x86-linux/, linux/, instead of x86-common/, ppc-common/, > x86/linux, common-linux. Yes, that's nice, and I agree we can lose the arch/ level too. > Ok, now for various lower-level comments about the patch, mostly aimed at > Paul. > > - Have you tried fitting your PPC port into this structure? Does it fit? > That's a good test. Yes it does, although there are still some ifdefs left, particularly in vg_translate.c and in parts of memcheck. > - You haven't touched the x86/ directory, I see. Any reason why not? Not wanting to make too many changes at once, basically. > - The syscalls are going to need a whole lot more abstraction/factoring > than what you've done with mmap(), but as you said the patch is only a > start. Indeed, and so is vg_translate.c. > - You've used VG_(arch_...) for a lot of names. I think a new prefix, eg. > VGA_(...) would be good. We could even use different prefixes for > definitions in the arch-common, common-os and arch-os interface, and then > a different prefix for code within those non-generic parts that isn't seen > by the generic code. Or maybe that's going too far, but something like > that would be good. Sounds good. BTW, is the VG_() for external symbols in fashion or out of fashion at the moment? I know I haven't been consistent about using or not using that. > - VG_(arch_deliver_signal)() is big -- is it possible to factor out the > generic/non-generic parts better? Hmmm, the last dozen or so lines could be moved back into generic code, but anything dealing with the signal frame layout is non-generic. > - why are arch_set_arg_and_ret() and arch_thread_initial_stack() not > VG_() prefixed? I did this over an extended period of time, and at the time I was doing that it seemed that the VG_() prefix wasn't in fashion. :) I don't really mind either way very much, although simpler is always better. > - There doesn't seem to be much consistency in the file-naming scheme, eg. > some have "arch" in them, some have "x86", some don't. My proposal above > partly addresses this. Part of that was because automake got unhappy if I had two .c files with the same name in different subdirectories. That problem would be solved by having make recurse into the platform-specific directories and make .a files. The other point was that I was concentrating more about what functions the platform code needed to supply to the generic code, than on what went in what files. > - I'm not happy about vg_libpthread.c being in x86-$OS/ subdirs -- huge > parts will be shared with the x86-freebsd, etc. But maybe you copied that > from the current (misleading) coregrind/arch/x86-$OS/ directories, the > contents of which aren't actually used. I didn't touch the versions of vg_libpthread.c in x86-$OS. I moved some of coregrind/vg_libpthread.c into x86-common/vg_x86_thread.c. I agree that having a redundant version in x86-$OS is confusing. :) > - If this patch was done in small steps, is there any order you'd > prefer/suggest? Probably start with putting various #defines in platform-specific headers (e.g. what I had in vg_archconst.h). The REGPARM definition and the FIRST/LAST_ARCH_REG definitions could be added to vg_skin.h.base fairly early. Then move all the platform-specific VGOFF_() variables and their initialization and use into platform-specific files. Then tackle the platform-specific parts of ThreadState. We also need to pull out the x86-specific parts of vg_skin.h.base at some point. Then we can start on the hard parts. :) Regards, Paul. |
|
From: Nicholas N. <nj...@ca...> - 2004-08-21 17:17:26
|
On Sat, 21 Aug 2004, Paul Mackerras wrote: > Yes. There may also be some code that depends on the executable > format which is shared across several platforms, for example, ELF is > used on x86-linux, ppc-linux, x86-freebsd I presume, but not > ppc-macosx (which AFAIK uses mach-o). Hmm, I'll think about that. > Looked good to me, although you don't say what to do with the > platform-specific definitions in the files in include/. Do we have a > set of platform directories under include as well as the set under > coregrind, or do we try to keep it down to one set of platform- > specific directories? Yes, not sure about that. Currently we have tool-visible headers (eg. vg_skin.h) in include/ and others (eg. vg_include.h) in coregrind/. I'll think about that too. > BTW, is the VG_() for external symbols in fashion or out of fashion at > the moment? I know I haven't been consistent about using or not using > that. AFAIK all external symbols do have VG_() (or SK_() or VGP_()) prefixed. It's not as necessary now that we have full virtualisation and Valgrind is effectively in a separate namespace from the client, although we do still have internal versions of many glibc functions. Also, I like that it makes clear what's external. And it's consistent. N |
|
From: Nicholas N. <nj...@ca...> - 2004-08-21 17:23:53
|
>> The keys ideas are: >> >> - to very carefully identify which category all code belongs to >> - to minimise the interface between generic and non-generic code >> - to minimise code duplication by pushing code into the most generic code >> part possible > > Yes, this is certainly the goal. I don't think we are going to get > this right from the beginning though, so we will have to allow it to > evolve. I'll take another look and work out a more detailed proposal, since nobody seems to think the basic outline is drastically flawed or anything. N |
|
From: Bob F. <bfr...@si...> - 2004-08-21 14:34:24
|
On Sat, 21 Aug 2004, Paul Mackerras wrote: > > Are people wedded to the GNU standards for Makefiles and/or to using > automake? If so then we absolutely need an automake expert. > Otherwise we could try to come up with something hopefully a bit > simpler that would do what we need. Automake is really not very complicated and is sufficiently powerful to handle building Valgrind. Nicholas did set up a single Automake Makefile.am to build Valgrind but he didn't like the way options were applied so the work was shelved. It seems that no more than a day of effort was expended. > The main challenges I see in building valgrind are: > > * Setting up symlinks so that the headers for the architecture we're > building for get included. On the whole I prefer symlinks to using > multiple -I flags (but I don't know how to get automake to create > the symlinks). Automake will pass through normal `make' rules so it is easy to create symlinks. I believe that using symlinks is not the right approach though since it is not completely portable and the -I flag mechanism is very easy to tailor from the configure script and through make variables. Bob ====================================== Bob Friesenhahn bfr...@si... http://www.simplesystems.org/users/bfriesen |
|
From: Nicholas N. <nj...@ca...> - 2004-08-21 17:22:53
|
On Sat, 21 Aug 2004, Bob Friesenhahn wrote: >> Are people wedded to the GNU standards for Makefiles and/or to using >> automake? If so then we absolutely need an automake expert. >> Otherwise we could try to come up with something hopefully a bit >> simpler that would do what we need. > > Automake is really not very complicated and is sufficiently powerful to > handle building Valgrind. > > Nicholas did set up a single Automake Makefile.am to build Valgrind but he > didn't like the way options were applied so the work was shelved. It seems > that no more than a day of effort was expended. Maybe a bit more than a day. Yes, compiling different C files with different compiler options was a problem -- it's easy to set the flags for all files, or for one, but not for a subset. Also, I just couldn't get everything to work properly. It seemed like in theory it all made sense, but there were enough difficult details that I gave up, because the benefits were not very significant. It's a good question -- what does using automake give us? I'm sure it has some nice things (I think it works out dependencies automatically?) Perhaps someone can list the advantages of using it over hand-writing Makefile.in files. >> * Setting up symlinks so that the headers for the architecture we're >> building for get included. On the whole I prefer symlinks to using >> multiple -I flags (but I don't know how to get automake to create >> the symlinks). > > Automake will pass through normal `make' rules so it is easy to create > symlinks. I believe that using symlinks is not the right approach though > since it is not completely portable and the -I flag mechanism is very easy to > tailor from the configure script and through make variables. symlinks vs. -I does not seem like a big deal to me. We already use both in different circumstances. It's not hard to switch between them. N |
|
From: Bob F. <bfr...@si...> - 2004-08-21 18:42:37
|
On Sat, 21 Aug 2004, Nicholas Nethercote wrote: > > It's a good question -- what does using automake give us? I'm sure it has > some nice things (I think it works out dependencies automatically?) Perhaps > someone can list the advantages of using it over hand-writing > Makefile.in files. Provides: * Assures make target conformance with GNU standards. * Encapsulates GNU standard prefix install rules. * Provides a framework to encapsulate test suites. * Works with any `make' program, not just GNU make. * Includes platform portability tweaks. * Integrated with libtool. * Conditional statements in Makefile.am. * Automatic dependency generation (including proprietary compilers). * Automatically imports configure definitions as Makefile variables. * Provides a simple syntax to install files into any directory. * Provides a solid 'dist' target. * Provides a self-validating 'distcheck' target. * Works well with VPATH builds. Does not provide: * Makefile includes. * Wildcarded sources/targets. * GNU make style functions. * Run-time make conditionals (uses configure time conditionals). There was a time when I knew every aspect of GNU make and wrote super-intelligent Makefiles by hand. I would not choose to do that now. Bob ====================================== Bob Friesenhahn bfr...@si... http://www.simplesystems.org/users/bfriesen |
|
From: Jeremy F. <je...@go...> - 2004-08-22 00:50:52
|
On Sat, 2004-08-21 at 13:42 -0500, Bob Friesenhahn wrote:
> On Sat, 21 Aug 2004, Nicholas Nethercote wrote:
> >
> > It's a good question -- what does using automake give us? I'm sure it has
> > some nice things (I think it works out dependencies automatically?) Perhaps
> > someone can list the advantages of using it over hand-writing
> > Makefile.in files.
>
Of these:
> Provides:
>
> * Provides a solid 'dist' target.
> * Provides a self-validating 'distcheck' target.
> * Works well with VPATH builds.
I think these are the really useful ones.
> Does not provide:
>
> * Makefile includes.
> * Wildcarded sources/targets.
> * GNU make style functions.
> * Run-time make conditionals (uses configure time conditionals).
And these are all big minuses. I would add:
* huge complexity
* bad documentation
* inflexible (or perhaps that's only "apparently inflexible, given
the bad documentation)
* multistage build just to generate Makefiles (I hate the fact
that you can't just edit the Makefiles and use them without a
"compilation" stage)
* billions of subtly incompatible versions
> There was a time when I knew every aspect of GNU make and wrote
> super-intelligent Makefiles by hand. I would not choose to do that
> now.
GNU make is much better documented and more tractable than the teetering
pile of M4, perl, sh and Makefiles, and the reliance on their
overlapping syntax.
So, I'd happily drop both automake and autoconf and do something more
kernel-like.
J
|
|
From: Bob F. <bfr...@si...> - 2004-08-22 01:11:39
|
On Sat, 21 Aug 2004, Jeremy Fitzhardinge wrote: > >> There was a time when I knew every aspect of GNU make and wrote >> super-intelligent Makefiles by hand. I would not choose to do that >> now. > > GNU make is much better documented and more tractable than the teetering > pile of M4, perl, sh and Makefiles, and the reliance on their > overlapping syntax. Comparing GNU make to Automake is almost like comparing an assembly language with a compiled language. It is easy enough to get started with GNU make for small programs, but building serious projects with the expected bells and whistles takes considerable effort just to get the framework in place. Bob ====================================== Bob Friesenhahn bfr...@si... http://www.simplesystems.org/users/bfriesen |
|
From: Julian S. <js...@ac...> - 2004-08-30 11:32:11
|
> On Sunday 22 August 2004 01:51, Jeremy Fitzhardinge wrote: > [...] > GNU make is much better documented and more tractable than the teetering > pile of M4, perl, sh and Makefiles, and the reliance on their > overlapping syntax. > > So, I'd happily drop both automake and autoconf and do something more > kernel-like. Yes, I find the complexity of the current build system scary and a potential (actual?) maintenance problem. And so I'd also be inclined to dump automake/autoconf in a hole* if possible, and do something simpler. Or at least dump automake? Is it viable/useful to back off to just autoconf? > On Sunday 22 August 2004 02:11, Bob Friesenhahn wrote: > Comparing GNU make to Automake is almost like comparing an assembly > language with a compiled language. =C2=A0It is easy enough to get started > with GNU make for small programs, but building serious projects with > the expected bells and whistles takes considerable effort just to get > the framework in place. Bob, can you show some specifics of what we would be losing by moving to a GNU-make-only system? Currently I have no clear idea of what the tradeoff is. J * providing of course this didn't cause environmental damage. Are they biodegradable? |
|
From: Jeremy F. <je...@go...> - 2004-08-22 00:36:32
|
On Fri, 2004-08-20 at 12:47 +0100, Nicholas Nethercote wrote: > Nick's proposal > --------------- > > Terminology: A "platform" is a specific $ARCH/$OS pair. For every arch > and OS, there will be 4 code parts, in order of genericity: > > 1. some generic code (aka common-common) > 2a. some arch-specific, OS-independent code (aka $ARCH-common) > 2b. some arch-independent, OS-specific code (aka common-$OS) > 3. some platform-specific code (aka $ARCH-$OS) > > 2a, 2b and 3 are all "non-generic". > > The keys ideas are: > > - to very carefully identify which category all code belongs to > - to minimise the interface between generic and non-generic code > - to minimise code duplication by pushing code into the most generic code > part possible Yep. This all looks pretty sound. My thinking also included a generic "unix" directory, for things which are common to all unixes but not really inherent to the generic core of Valgrind (eg, everyone has open/close/read/write, so you may as well make them common). This would only really make sense if you start porting to non-Unix-like targets (which is always an option, though the most likely non-Unix target is Windows, which looks pretty formidable). > > Abstract directory structure: > > valgrind/ > > coregrind/ > > arch/ > > arch_common_defs.h (contains arch-specific definitions) > common_os_defs.h (contains OS-specific definitions) > arch_os_defs.h (contains platform-specific definitions) Are these common across all arch/os combinations? How can they be specific? Or do you mean $ARCH_common_defs.h? > arch_common@ (symlink to $ARCH-common/, made at config-time) > common_os@ (symlink to common-$OS/, made at config-time) > arch_os@ (symlink to $ARCH-$OS/, made at config-time) > > $ARCH-common/ > > arch_common_macros.h (contains arch-specific macros used by core) > arch_common.a (contains all arch-specific code) > > common-$OS/ > > common_os_macros.h (contains OS-specific macros used by core) > common_os.a (contains all OS-specific code) > > $ARCH-$OS/ > > arch_os_macros.h (contains platform-specific macros used by core) > arch_os.a (contains all platform-specific code) Is the source which generates the .a files also in these directories? > The generic part would need only these six #includes: > > #include "platform/arch_common_defs.h" > #include "platform/common_os_defs.h" > #include "platform/arch_os_defs.h" > > These could be put in vg_include.h and thus be made visible to all core > code. Or not. Why not make it that the generic code always includes arch_os_defs.h, which in turn can include whichever generic headers are appropriate. This allows for things which are common across some arch/os combinations but not all (ELF loader, for example). The general idea is to allow arch-os ports to use generic code if appropriate, but they can always provide their own versions if necessary (ie, generic code is an arch-os helper library rather than something they're forced to use). Obviously use of generic code should be strongly encouraged, and rather than using an arch-os specific version of a generic function, we would prefer to make the generic code more generic. > > Each of the _defs.h files would have the following respective #includes: > > #include "platform/arch_common/arch_common_macros.h" > #include "platform/common_os/common_os_macros.h" > #include "platform/arch_os/arch_os_macros.h" > > in order to pull in the relevant _macros.h files -- the symlinks would > ensure the right ones are pulled in. > > When building stage2, coregrind/Makefile.am would just link all the core > modules with: > > - platform/arch_common/arch_common.a > - platform/common_os/common_os.a > - platform/arch_os/arch_os.a > > The symlinks would again ensure this works. These .a files would be made > by linking together all the .o files in each part. In general I'm not terribly keen on symlinks; why couldn't you set the linker search path with make variables? > Each non-generic sub-dir can also include as many other .c, .h and .S > files as it needs (plus anything else, like ld scripts). But the generic > part wouldn't see or care about that. > > > Note that arch/ is a really bad name. I'm not convinced we even need that > subdirectory between coregrind/ and the non-generic subdirs -- it will > just create more typing, for little benefit. I don't think that's a big issue, and I'd prefer to be able to "ls arch" to see what we have there. > I haven't considered skins at all here. Perhaps each of the the proposed > headers needs to be split into two halves: one for the skin-visible > stuff, and one for the core-visible stuff (which would #include the > skin-visible header). Yep. > I haven't considered asm code at all here; some .S files need some > definitions, but you can't have any macros included in .S files. The opposite: you can't include anything *except* macros in .S files. They're passed through cpp, so macro expansion happens like in C. > I haven't considered the UME stuff that is shared between stage1 and > stage2. Not sure about that, it would probably be kept separate at least > to some extent. The only issue is whether it gets compiled once or twice. I think it should be possible to compile once and link the .so into both executables. > > For tools that need to do non-generic things (which should be avoided as > much as possible, but may be inevitable in small amounts, esp. for > Memcheck) parts of the structure can be mirrored as necessary within the > tool/ (eg. memcheck/) directory. > > The names might be a bit confusing, in particular I'm not sure about the > wisdom of reusing "common" in the names "arch-common" and "common-os". > Perhaps the following would be clearer (it would also be less typing): > > - arch-common --> arch > - common-os --> os > - arch-os --> arch-os > > Eg. x86/, ppc/, x86-linux/, linux/, instead of x86-common/, ppc-common/, > x86/linux, common-linux. I'd prefer the "common", or something like it, so you can match, say, all x86 files with x86-* or *-linux (in other words, consistency for automated name generation is more important). > - I'm not happy about vg_libpthread.c being in x86-$OS/ subdirs -- huge > parts will be shared with the x86-freebsd, etc. But maybe you copied that > from the current (misleading) coregrind/arch/x86-$OS/ directories, the > contents of which aren't actually used. Well, we're hoping to disappear that altogether... J |
|
From: Nicholas N. <nj...@ca...> - 2004-08-22 14:09:32
|
On Sat, 21 Aug 2004, Jeremy Fitzhardinge wrote: > My thinking also included a generic "unix" directory, for things which > are common to all unixes but not really inherent to the generic core of > Valgrind (eg, everyone has open/close/read/write, so you may as well > make them common). This would only really make sense if you start > porting to non-Unix-like targets (which is always an option, though the > most likely non-Unix target is Windows, which looks pretty formidable). I'd say let's do that if/when the need arises. >> Abstract directory structure: >> >> valgrind/ >> >> coregrind/ >> >> arch/ >> >> arch_common_defs.h (contains arch-specific definitions) >> common_os_defs.h (contains OS-specific definitions) >> arch_os_defs.h (contains platform-specific definitions) > > Are these common across all arch/os combinations? How can they be > specific? Or do you mean $ARCH_common_defs.h? They're meant to be common. Things like the function definition for VG_(load_thread_state)(). >> $ARCH-$OS/ >> >> arch_os_macros.h (contains platform-specific macros used by core) >> arch_os.a (contains all platform-specific code) > > Is the source which generates the .a files also in these directories? Yes. >> The generic part would need only these six #includes: >> >> #include "platform/arch_common_defs.h" >> #include "platform/common_os_defs.h" >> #include "platform/arch_os_defs.h" >> >> These could be put in vg_include.h and thus be made visible to all core >> code. Or not. > > Why not make it that the generic code always includes arch_os_defs.h, > which in turn can include whichever generic headers are appropriate. > This allows for things which are common across some arch/os combinations > but not all (ELF loader, for example). Sure, could do; that seems like a more minor point to me. > The general idea is to allow arch-os ports to use generic code if > appropriate, but they can always provide their own versions if necessary > (ie, generic code is an arch-os helper library rather than something > they're forced to use). Obviously use of generic code should be > strongly encouraged, and rather than using an arch-os specific version > of a generic function, we would prefer to make the generic code more > generic. We may have different views of what is meant by "generic". I view normal operation as mostly occurring within generic code, but occasionally having to do something arch/os/platform specific, eg. get the stack pointer, or run a syscall -- in which case the generic code calls code in the arch/os/platform part. It sounds like you're thinking of generic code as a kind of library that the arch/os/platform-specific code can call upon to help its needs. Is that right? Perhaps both views are necessary. > In general I'm not terribly keen on symlinks; why couldn't you set the > linker search path with make variables? You could. Symlinks vs. specifying-include-dirs seems to be a style issue; Paul prefers symlinks, you and Bob prefer includes. It doesn't matter that much. >> Note that arch/ is a really bad name. I'm not convinced we even need that >> subdirectory between coregrind/ and the non-generic subdirs -- it will >> just create more typing, for little benefit. > > I don't think that's a big issue, and I'd prefer to be able to "ls arch" > to see what we have there. Then we're in disagreement. >> I haven't considered asm code at all here; some .S files need some >> definitions, but you can't have any macros included in .S files. > > The opposite: you can't include anything *except* macros in .S files. > They're passed through cpp, so macro expansion happens like in C. Whoops, yes, my bad. >> The names might be a bit confusing, in particular I'm not sure about the >> wisdom of reusing "common" in the names "arch-common" and "common-os". >> Perhaps the following would be clearer (it would also be less typing): >> >> - arch-common --> arch >> - common-os --> os >> - arch-os --> arch-os >> >> Eg. x86/, ppc/, x86-linux/, linux/, instead of x86-common/, ppc-common/, >> x86/linux, common-linux. > > I'd prefer the "common", or something like it, so you can match, say, > all x86 files with x86-* or *-linux (in other words, consistency for > automated name generation is more important). Then we're in disagreement again. N |
|
From: Nicholas N. <nj...@ca...> - 2004-08-22 14:42:46
|
On Sun, 22 Aug 2004, Nicholas Nethercote wrote: >>> I haven't considered asm code at all here; some .S files need some >>> definitions, but you can't have any macros included in .S files. >> >> The opposite: you can't include anything *except* macros in .S files. >> They're passed through cpp, so macro expansion happens like in C. > > Whoops, yes, my bad. Also, in my original proposal, I said "definitions" repeatedly when I actually meant "declarations". Sorry for the confusion. I'll post a modified, more detailed proposal some time soon, in which I'll try to address the received comments. N |