|
From: <eyu...@ab...> - 2008-12-15 10:43:28
|
Hello,
I have read the FAQ and searched the mailing lists (or tried to, there
were 100s of related posts) about this but...
The FAQ says:
--------------------------------------------------
As for eager reporting of copies of uninitialised memory values, this
has been suggested multiple times. Unfortunately, almost all programs
legitimately copy uninitialised memory values around (because
compilers pad structs to preserve alignment) and eager checking leads
to hundreds of false positives. Therefore Memcheck does not support
eager checking at this time."
--------------------------------------------------
However it is possible to pack the structs without holes...
from gcc manual:
--------------------------------------------------
-fpack-struct[=n]
Without a value specified, pack all structure members together
without holes. When a value is specified (which must be a small power
of two), pack structure members according to this value, representing
the maximum alignment (that is, objects with default alignment
requirements larger than this will be output potentially unaligned at
the next fitting location.
--------------------------------------------------
Would it be possible to introduce an eager checking mechanism which is
enabled by a command line option while running valgrind? or are there
other reasons why this is not done than the one explained in the FAQ?
If there are, what are they?
BTW I also tried the '--track-origins=yes' from the trunk code. It is
quite useful as well, but it might be nicer to be able to see where
the uninitialized variable is used first time (in an assignment etc.)
other than showing errors for only printf etc. type statements. Is
there any practical difficulty for not implementing this feature other
than the one explained at FAQ? am I missing something here?
Thanks,
Evren
|
|
From: Dallman, J. <joh...@si...> - 2008-12-15 11:20:44
|
> However it is possible to pack the structs without holes... You will not like what this does to performance. It also means that you will be Valgrinding a different build of your software from the one that gets run in anger. -- John Dallman Parasolid Porting Engineer |
|
From: <eyu...@ab...> - 2008-12-15 12:41:58
|
Quoting "Dallman, John" <joh...@si...>: >> However it is possible to pack the structs without holes... > > You will not like what this does to performance. It also means > that you will be Valgrinding a different build of your software > from the one that gets run in anger. I understand the performance penalty. I also understand the binaries will not be exactly the same obviously. But, isnt it so that when we run a program with valgrind, it is already not the same as running it without valgrind in a similar sense? Valgrind changes the program execution, am I wrong? But, lets say slowness and using a different build of the software are acceptable risks for finding uninitialized values under certain cases where it is very difficult to manually track uninitialized values. I would gladly compile the software so structs will be without holes just to find uninitialized values easier. Perhaps, after that, I can compile to software normally again for other purposes. The real question is if the structs being padded by the compiler the only reason why there cant be uninitialized value checks or are there other reasons? In other words, if this compiler option was used, would it be possible for valgrind to do eager checking of uninitialized values without false positives? Thanks, Evren |
|
From: Tom H. <to...@co...> - 2008-12-15 12:31:48
|
eyu...@ab... wrote: > However it is possible to pack the structs without holes... > > from gcc manual: > -------------------------------------------------- > -fpack-struct[=n] > Without a value specified, pack all structure members together > without holes. When a value is specified (which must be a small power > of two), pack structure members according to this value, representing > the maximum alignment (that is, objects with default alignment > requirements larger than this will be output potentially unaligned at > the next fitting location. > -------------------------------------------------- > > Would it be possible to introduce an eager checking mechanism which is > enabled by a command line option while running valgrind? or are there > other reasons why this is not done than the one explained in the FAQ? > If there are, what are they? You're going to recompile glibc and all the other libraries that your program uses with that option are you? Tom -- Tom Hughes (to...@co...) http://www.compton.nu/ |
|
From: Dallman, J. <joh...@si...> - 2008-12-15 13:01:10
|
Tom Hughes [mailto:to...@co...] > eyu...@ab... wrote: > > However it is possible to pack the structs without holes... > You're going to recompile glibc and all the other libraries that your > program uses with that option are you? And, of course, if you don't, structs passed between packed and unpacked code aren't compatible. Members of the structs will be at different offsets as far the two sets of code are concerned. The effects of this are undefined in the most comprehensive sense of the term: your computer is quite unlikely to catch fire, but if it does, it was your fault. -- John Dallman Parasolid Porting Engineer |
|
From: <eyu...@ab...> - 2008-12-15 15:20:13
|
Quoting "Dallman, John" <joh...@si...>: > Tom Hughes [mailto:to...@co...] >> eyu...@ab... wrote: >> > However it is possible to pack the structs without holes... >> You're going to recompile glibc and all the other libraries that your >> program uses with that option are you? > > And, of course, if you don't, structs passed between packed and unpacked > code aren't compatible. Members of the structs will be at different > offsets as far the two sets of code are concerned. The effects of this > are > undefined in the most comprehensive sense of the term: your computer is > quite unlikely to catch fire, but if it does, it was your fault. Oh yes, I forgot about those :) You have a good point, but still, the effort for recompiling the libraries and other required code for this purpose might be much less than trying to find an uninitialized variable in some cumbersome code. For example I could download the whole source tree for FreeBSD OS and re-compile with this compiler option (including kernel and all the libraries shipped with the OS) in about an hour or two (in which case this system can be used several times for debugging code without repeating this action). In comparison, even finding the source of a single uninitialized variable can take much longer time. So, the difficulty of finding an uninitialized variable (or many of them) can easily outweigh the difficulty of recompiling the libraries once for this purpose. Lets say that all of the difficulties we discussed until now have been resolved. Is there any other reason for valgrind not to do eager checking for this? Thanks, Evren |
|
From: Dallman, J. <joh...@si...> - 2008-12-15 16:51:28
|
> Oh yes, I forgot about those :) You have a good point, but still, the > effort for recompiling the libraries and other required code for this > purpose might be much less than trying to find an uninitialized > variable in some cumbersome code. It might. But it seems unlikely. To the best of my knowledge, most open source packages are not routinely tested in builds with packed structures, and are very rarely built that way in the first place. You seem to be assuming that your entire OS will just work, correctly, when rebuilt with packed structures. Do you have any basis for that assumption? What odds do you reckon you're looking at, and how long to sort out the problems? Are you experienced at debugging every single part of your OS? And debugging Valgrind? I suggest you try it. That will likely be the easiest way of convincing yourself that this is a poor idea. > Lets say that all of the difficulties we discussed until now have been > resolved. You have not reached that situation. -- John Dallman Parasolid Porting Engineer Siemens PLM Software 46 Regent Street, Cambridge, CB2 1DP United Kingdom Tel: +44-1223-371554 joh...@si... www.siemens.com/plm |
|
From: <eyu...@ab...> - 2008-12-17 15:21:05
|
Quoting "Dallman, John" <joh...@si...>: >> Oh yes, I forgot about those :) You have a good point, but still, the >> effort for recompiling the libraries and other required code for this >> purpose might be much less than trying to find an uninitialized >> variable in some cumbersome code. > > It might. But it seems unlikely. To the best of my knowledge, most > open source packages are not routinely tested in builds with packed > structures, and are very rarely built that way in the first place. > > You seem to be assuming that your entire OS will just work, correctly, > when rebuilt with packed structures. Do you have any basis for that > assumption? What odds do you reckon you're looking at, and how long > to sort out the problems? Are you experienced at debugging every > single part of your OS? And debugging Valgrind? > > I suggest you try it. That will likely be the easiest way of convincing > yourself that this is a poor idea. > >> Lets say that all of the difficulties we discussed until now have been >> resolved. > > You have not reached that situation. > You are right, I tried to recompile everything and the compilation of standard c++ libraries etc. fail. Sorry for wasting your time I guess :( Thanks, Evren |
|
From: Dallman, J. <joh...@si...> - 2008-12-17 17:00:04
|
> You are right, I tried to recompile everything and the compilation of > standard c++ libraries etc. fail. Sorry for wasting your time I guess > :( If you want to persist with this idea, don't let me stop you from starting work on making everything capable of building with packed structures. It will be a great deal of work, but it will be very educational. However, most projects will not see this as a priority, and may be unwilling to accept changes for this purpose, unless they have other benefits. -- John Dallman Parasolid Porting Engineer |
|
From: Nicholas N. <nj...@cs...> - 2008-12-15 22:21:38
|
On Mon, 15 Dec 2008 eyu...@ab... wrote: > Lets say that all of the difficulties we discussed until now have been > resolved. Is there any other reason for valgrind not to do eager > checking for this? A big slow-down -- instead of having to check memory accesses, syscalls, conditional branches, and a few other places for undefined values, it would have to check just about every instruction. Nick |
Tom Hughes wrote: > eyu...@ab... wrote: > >> However it is possible to pack the structs without holes... >> -fpack-struct[=n] > You're going to recompile glibc and all the other libraries that your > program uses with that option are you? For glibc I've already removed all the holes (which were few) and cleaned up other annoying memcheck reports: http://bitwagon.com/glibc-audit/glibc-audit.html The problems with holes lie more in X11, Gnome, KDE, etc. -- |
|
From: Florian K. <br...@ac...> - 2008-12-15 16:02:51
|
On Monday 15 December 2008 05:43:20 eyu...@ab... wrote:
> Hello,
>
> I have read the FAQ and searched the mailing lists (or tried to, there
> were 100s of related posts) about this but...
>
> The FAQ says:
> --------------------------------------------------
> As for eager reporting of copies of uninitialised memory values, this
> has been suggested multiple times. Unfortunately, almost all programs
> legitimately copy uninitialised memory values around (because
> compilers pad structs to preserve alignment) and eager checking leads
> to hundreds of false positives. Therefore Memcheck does not support
> eager checking at this time."
> --------------------------------------------------
>
> However it is possible to pack the structs without holes...
Huh?
struct foo {
int x : 2;
};
GCC will allocate at least 1 byte of memory for this struct
independent of -fpack-struct.
Florian
|
Florian Krohm wrote:
> On Monday 15 December 2008 05:43:20 eyu...@ab... wrote:
>> However it is possible to pack the structs without holes...
>
> Huh?
>
> struct foo {
> int x : 2;
> };
>
> GCC will allocate at least 1 byte of memory for this struct
> independent of -fpack-struct.
Any conforming compiler must allocate at least sizeof(int) bytes.
However, for that specific struct, it is possible for a compiler
to widen all writes to 'x' so that sizeof(int) bytes are written.
This will avoid the hole as far as memcheck is concerned.
Obviously such widening is not possible in the next case:
struct foo {
int x : 2;
int y : 30;
};
--
|
Nicholas Nethercote wrote: > On Mon, 15 Dec 2008 eyu...@ab... wrote: > >> Lets say that all of the difficulties we discussed until now have been >> resolved. Is there any other reason for valgrind not to do eager >> checking for this? > > A big slow-down -- instead of having to check memory accesses, syscalls, > conditional branches, and a few other places for undefined values, it would > have to check just about every instruction. When I did eager checking for undef bits in my own tool [not a member of the valgrind suite], the slowdown was about the same as memcheck. Just examine the Valid bits after every fetch from memory, and do enough data flow analysis to recognize "over fetching" by the compiler. Especially on x86, GCC may fetch 4 bytes when the member (particularly a bitfield) occupies only 2, etc. Incidentally, detecting undef is somewhat fast. The slowness comes from identifying and suppressing the second and subsequent reports of the "same" error. Eager checking is prone to "false positive" complaints of undefined bits. One of memcheck's advantages is a low rate of false positive reports. I don't like false positive reports either. However, my goal was closer to "the software never fetched undefined bits", which is more strict than "the results did not depend on undefined bits." A few customers paid real money for the extra property. Pointing out the location of actual holes did identify a few important cases of wasteful layout, and of grossly suboptimal code to deal with adjacent bitfields. In my opinion, memcheck should have a runtime option for eager checking. I would use it at least sometimes on every project. The maintainers of memcheck disagree with me on the usefulness and costs of eager checking. I have considered implementing the option myself, but so far other work is more important to me. I did clean up glibc: http://bitwagon.com/glibc-audit/glibc-audit.html -- |
|
From: <eyu...@ab...> - 2008-12-16 09:15:45
|
Quoting John Reiser <jreiser@BitWagon.com>: > In my opinion, memcheck should have a runtime option for eager checking. > I would use it at least sometimes on every project. The maintainers of > memcheck disagree with me on the usefulness and costs of eager checking. > I have considered implementing the option myself, but so far other work > is more important to me. I did clean up glibc: > http://bitwagon.com/glibc-audit/glibc-audit.html > I would like a runtime option too if it does not require significant changes to memcheck. I am not telling that a runtime option is absolutely necessary. But as far as I an see, it can be quite useful. While some of the examples here have a point, it is a small nuisance to recompile the OS for this purpose and I believe it is rare for programs to have less than 1 byte variables (acceptable amount of false positives perhaps). Thanks, Evren |