|
From: Bob H. <rsh...@gm...> - 2011-03-31 13:03:38
|
Howdy,
Here's a followup on my problem. I think it is clearly a problem with
valgrind in large memory conditions. Perhaps I can build valgrind in a way
that allows for larger blocks? (if that is true, please point me in the
right direction).
Since my program has an alternative allocation-tracking technique built in,
I wrote a converter that converts that allocation-tracker's output to a test
program that does the same series of allocations. This test program is
appended below my signature. It does nothing but allocate, reallocate, and
free.
I then built this on a linux-x86_64 machine with these gcc commands:
cc -c -g -O0 -pg lz_memory_test.c -o lz_memory_test_valgrind.o
cc lz_memory_test_valgrind.o -o lz_memory_test_valgrind
Then I ran valgrind on it, like this:
valgrind --leak-check=full lz_memory_test_valgrind
valgrind version is 3.5.0
Output from valgrind is below the program. I get the same sort of "set
address range perms" I get on my real program (please, if somebody could
give me more info about those, it would be helpful). Then it tells me I
have 2,736,652,288 bytes in use at exit. This would seem to relate to lines
84 and 85 of the program (which I have marked with trailing comments). Line
84 allocates exactly that many bytes, as part of a realloc of m14. Line 85
realloc's again, increasing the size. m14 is realloc'd several more times
after that, then freed at line 184. So, it seems that either (a) there's a
bug in realloc, and it did not dispose of the incoming m14 at line 85, or
(b) valgrind has lost track.
I also notice that the last "address range perms" complaint has a size that
corresponds exactly to the allocation of m14 at line 83, i.e. the last
allocation before the allocation that appears to be "lost"
Thanks for any help,
Bob H
My test program, lz_memory_test.c, which (I claim) allocates, reallocates,
and frees the same as my target program does.
#include <stdlib.h>
#include <stdio.h>
int main (int argc, char** argv);
int main (int argc, char** argv)
{
void* m1;
void* m2;
void* m3;
void* m4;
void* m5;
void* m6;
void* m7;
void* m8;
void* m9;
void* m10;
void* m11;
void* m12;
void* m13;
void* m14;
void* m15;
void* m16;
void* m17;
void* m18;
void* m19;
void* m20;
void* m21;
void* m22;
void* m23;
void* m24;
void* m25;
void* m26;
void* m27;
void* m28;
void* m29;
void* m30;
void* m31;
void* m32;
void* m33;
void* m34;
m1 = malloc((size_t)218);
m2 = malloc((size_t)51);
m3 = malloc((size_t)32);
m4 = malloc((size_t)20);
m5 = malloc((size_t)16);
m6 = malloc((size_t)263984);
m7 = malloc((size_t)184);
m8 = malloc((size_t)263984);
free(m4);
m9 = malloc((size_t)632);
m10 = malloc((size_t)1024);
m11 = malloc((size_t)51);
m12 = realloc(NULL,(size_t)16384);
m13 = realloc(NULL,(size_t)16384);
m14 = realloc(NULL,(size_t)16384);
m15 = realloc(NULL,(size_t)5);
m14 = realloc(m14,(size_t)249266176);
m16 = realloc(NULL,(size_t)156);
m17 = realloc(NULL,(size_t)156);
m18 = realloc(NULL,(size_t)1300108);
m19 = realloc(NULL,(size_t)1300108);
m20 = realloc(NULL,(size_t)5);
m14 = realloc(m14,(size_t)492453888);
m14 = realloc(m14,(size_t)690487296);
m14 = realloc(m14,(size_t)881639424);
m14 = realloc(m14,(size_t)1062551552);
m14 = realloc(m14,(size_t)1233666048);
m14 = realloc(m14,(size_t)1392803840);
m14 = realloc(m14,(size_t)1548075008);
m14 = realloc(m14,(size_t)1694433280);
m14 = realloc(m14,(size_t)1835646976);
m16 = realloc(m16,(size_t)160);
m17 = realloc(m17,(size_t)160);
m15 = realloc(m15,(size_t)6);
m14 = realloc(m14,(size_t)1971191808);
m20 = realloc(m20,(size_t)6);
m14 = realloc(m14,(size_t)2106195968);
m14 = realloc(m14,(size_t)2240053248);
m14 = realloc(m14,(size_t)2355216384);
m14 = realloc(m14,(size_t)2462564352);
m14 = realloc(m14,(size_t)2565095424);
m14 = realloc(m14,(size_t)2655453184); /* line 83 */
m14 = realloc(m14,(size_t)2736652288); /* line 84 */
m14 = realloc(m14,(size_t)2814722048); /* line 85 */
m14 = realloc(m14,(size_t)2877751296);
m14 = realloc(m14,(size_t)2937126912);
m14 = realloc(m14,(size_t)2996256768);
m14 = realloc(m14,(size_t)3047555072);
m14 = realloc(m14,(size_t)3095691264);
m15 = realloc(m15,(size_t)15);
m14 = realloc(m14,(size_t)3100606464);
m20 = realloc(m20,(size_t)15);
m14 = realloc(m14,(size_t)3105439744);
m14 = realloc(m14,(size_t)3110240256);
m14 = realloc(m14,(size_t)3114926080);
m14 = realloc(m14,(size_t)3119546368);
m14 = realloc(m14,(size_t)3124166656);
m14 = realloc(m14,(size_t)3128770560);
m15 = realloc(m15,(size_t)16);
m14 = realloc(m14,(size_t)3130458112);
m20 = realloc(m20,(size_t)16);
m14 = realloc(m14,(size_t)3131047936);
m15 = realloc(m15,(size_t)21);
m14 = realloc(m14,(size_t)3131588608);
m20 = realloc(m20,(size_t)21);
m14 = realloc(m14,(size_t)3131801600);
m14 = realloc(m14,(size_t)3131998208);
m14 = realloc(m14,(size_t)3132178432);
m14 = realloc(m14,(size_t)3132375040);
m14 = realloc(m14,(size_t)3132555264);
m14 = realloc(m14,(size_t)3132735488);
m14 = realloc(m14,(size_t)3132932096);
m14 = realloc(m14,(size_t)3133112320);
m14 = realloc(m14,(size_t)3133292544);
m14 = realloc(m14,(size_t)3133472768);
m15 = realloc(m15,(size_t)22);
m14 = realloc(m14,(size_t)3133636608);
m20 = realloc(m20,(size_t)22);
m14 = realloc(m14,(size_t)3133816832);
m14 = realloc(m14,(size_t)3133980672);
m14 = realloc(m14,(size_t)3134160896);
m14 = realloc(m14,(size_t)3134324736);
m14 = realloc(m14,(size_t)3134488576);
m14 = realloc(m14,(size_t)3134652416);
m14 = realloc(m14,(size_t)3134816256);
m14 = realloc(m14,(size_t)3134980096);
m14 = realloc(m14,(size_t)3135143936);
m14 = realloc(m14,(size_t)3135291392);
m14 = realloc(m14,(size_t)3135438848);
m14 = realloc(m14,(size_t)3135553536);
m14 = realloc(m14,(size_t)3135684608);
m14 = realloc(m14,(size_t)3135799296);
m14 = realloc(m14,(size_t)3135881216);
m14 = realloc(m14,(size_t)3135979520);
m14 = realloc(m14,(size_t)3136061440);
m14 = realloc(m14,(size_t)3136110592);
m14 = realloc(m14,(size_t)3136159744);
m14 = realloc(m14,(size_t)3136192512);
m14 = realloc(m14,(size_t)3136241664);
m14 = realloc(m14,(size_t)3136290816);
m14 = realloc(m14,(size_t)3136323584);
m14 = realloc(m14,(size_t)3136372736);
m14 = realloc(m14,(size_t)3136405504);
m14 = realloc(m14,(size_t)3136454656);
m14 = realloc(m14,(size_t)3136487424);
m14 = realloc(m14,(size_t)3136536576);
m14 = realloc(m14,(size_t)3136569344);
m14 = realloc(m14,(size_t)3136618496);
m14 = realloc(m14,(size_t)3136651264);
m14 = realloc(m14,(size_t)3136684032);
m14 = realloc(m14,(size_t)3136733184);
m14 = realloc(m14,(size_t)3136765952);
m14 = realloc(m14,(size_t)3136798720);
m14 = realloc(m14,(size_t)3136847872);
m14 = realloc(m14,(size_t)3136880640);
m14 = realloc(m14,(size_t)3136913408);
m14 = realloc(m14,(size_t)3136946176);
m14 = realloc(m14,(size_t)3136995328);
m14 = realloc(m14,(size_t)3137028096);
m14 = realloc(m14,(size_t)3137060864);
m14 = realloc(m14,(size_t)3137093632);
m14 = realloc(m14,(size_t)3137110016);
m14 = realloc(m14,(size_t)3137126400);
m14 = realloc(m14,(size_t)3137142784);
m14 = realloc(m14,(size_t)3137159168);
m14 = realloc(m14,(size_t)3137175552);
m21 = malloc((size_t)632);
m22 = malloc((size_t)1024);
m23 = malloc((size_t)32);
m24 = realloc(NULL,(size_t)16384);
m25 = realloc(NULL,(size_t)16384);
m26 = malloc((size_t)12615754384);
m27 = malloc((size_t)160032);
m28 = realloc(NULL,(size_t)16384);
m29 = realloc(NULL,(size_t)30);
m30 = realloc(NULL,(size_t)8);
m30 = realloc(m30,(size_t)9);
m31 = malloc((size_t)16777216);
m32 = malloc((size_t)16777216);
m33 = malloc((size_t)33554432);
m34 = malloc((size_t)16777216);
free(m2);
free(m14); /* line 184 */
free(m10);
free(m11);
free(m15);
free(m20);
free(m16);
free(m17);
free(m18);
free(m19);
free(m12);
free(m13);
free(m9);
free(m3);
free(m28);
free(m22);
free(m23);
free(m29);
free(m30);
free(m24);
free(m25);
free(m21);
free(m1);
free(m6);
free(m8);
free(m26);
free(m27);
free(m7);
free(m5);
free(m31);
free(m32);
free(m33);
free(m34);
return 0;
}
==12544== Memcheck, a memory error detector
==12544== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==12544== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==12544== Command: ./lz_memory_test_valgrind
==12544==
==12544== Warning: set address range perms: large range [0x13dde030,
0x31382050) (noaccess)
==12544== Warning: set address range perms: large range [0x3901d030,
0x6229d050) (noaccess)
==12544== Warning: set address range perms: large range [0x6229e030,
0x96b6a050) (noaccess)
==12544== Warning: set address range perms: large range [0x96b6b030,
0xd60bf050) (noaccess)
==12544== Warning: set address range perms: large range [0xd60c0030,
0x11f944050) (noaccess)
==12544== Warning: set address range perms: large range [0x11f945030,
0x17298d050) (noaccess)
==12544== Warning: set address range perms: large range [0x17298e030,
0x1cedea050) (noaccess)
==12544== Warning: set address range perms: large range [0x1cedeb030,
0x233ddb050) (noaccess)
==12544== Warning: set address range perms: large range [0x233ddc030,
0x2a1478050) (noaccess)
==12544== Warning: set address range perms: large range [0x2a1479030,
0x316c59050) (noaccess)
==12544== Warning: set address range perms: large range [0x316c5a030,
0x3944fa050) (noaccess)
==12544== Warning: set address range perms: large range [0x43dd9b030,
0x4c35e3050) (noaccess)
==12544== Warning: set address range perms: large range [0x4c766c030,
0x553c88050) (noaccess)
==12544== Warning: set address range perms: large range [0x557cad030,
0x5ea929050) (noaccess)
==12544== Warning: set address range perms: large range [0x5ee492030,
0x6872d6050) (noaccess)
==12544== Warning: set address range perms: large range [0x68a7e7030,
0x728c57050) (noaccess)
==12544==
==12544== HEAP SUMMARY:
==12544== in use at exit: 2,736,652,288 bytes in 1 blocks
==12544== total heap usage: 140 allocs, 64 frees, 33,075,445,557 bytes
allocated
==12544==
==12544== 2,736,652,288 bytes in 1 blocks are definitely lost in loss record
1 of 1
==12544== at 0x4A05F1D: realloc (vg_replace_malloc.c:476)
==12544== by 0x40091E: main (lz_memory_test.c:84)
==12544==
==12544== LEAK SUMMARY:
==12544== definitely lost: 2,736,652,288 bytes in 1 blocks
==12544== indirectly lost: 0 bytes in 0 blocks
==12544== possibly lost: 0 bytes in 0 blocks
==12544== still reachable: 0 bytes in 0 blocks
==12544== suppressed: 0 bytes in 0 blocks
==12544==
==12544== For counts of detected and suppressed errors, rerun with: -v
==12544== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
On Mar 28, 2011, at 4:10 PM, I wrote:
> I'm trying to check for leaks in a program I've written, using "valgrind
> --leak-check=full ...". When my program stays within relatively small
memory
> usage, that is, less than 2G total, valgrind has no complaints. However,
one
> of my test cases involves my program allocating a total of about 16G with
> about 12.6G in a single block. In this case, valgrind gives me this
warning,
> repeatedly (maybe a dozen times):
==31171== Warning: set address range perms: large range [0x13dde030,
0x31382050) (noaccess)
Then complains about a dozen times about "Invalid write of size 1" in
memcpy
> called by fread, as well as one other complaint, "Syscall param read(buf)
> points to unaddressable byte(s)", also in something called by fread.
After
> this it tells me it has detected more than 10 million errors and tells me
to
> "Go fix [my] program!". Then after reporting a page or so of stats, I get
a
> segfault.
>
> The program doesn't segfault on this case when built without without
> valgrind, and to the extent that I can tell, is giving me the correct
result
> (when valgrind isn't used).
>
> The machine I'm running on is a 64-bit linux machine with 32G of RAM.
This
> occurred with valgrind version 3.5.0 and also with version 3.6.1.
>
> I have attempted to find info on how to "set address range perms", but the
> best I have been able to find so far is that this is a message intended
for
> valgrind developers, not users. Is this something I can change if I build
> the valgrind source code?
>
> So... my question is, should I expect valgrind to work correctly in such
an
> environment? Or, equivalently, what are valgrind's memory limitations?
>
> Also worth noting, possibly, is that the single 12.6G block is the result
of
> several reallocs. Since the target program does not know a priori how
large
> this needs to be, it grows the block on an as-needed basis. I believe
> valgrind is reporting a "set address range perms" message ever time
realloc
> runs out of space in the current block. The first report has
> large range [0x13dde030, 0x31382050)
> and the final one has
> large range [0x68c353030, 0x72a7c3050)
> But if that last one is indeed the memory allocated for my huge block, it
is
> only 2.7G.
>
> I'm trying to decide whether I truly have a problem in my program, or if
the
> problem that valgrind doesn't handle blocks of this size.
|
|
From: Bob H. <rsh...@gm...> - 2011-03-31 19:32:58
|
On Mar 31, 2011, at 2:25 PM, John Reiser wrote: >> I get the same sort of "set address range perms" I get on my real program >> (please, if somebody could give me more info about those, it would be >> helpful). >> >> ==12544== Warning: set address range perms: large range [0x13dde030, 0x31382050) (noaccess) > > RTFM: "Diagnostic message, mostly for benefit of the Valgrind > developers, to do with memory permissions." > > Valgrind gives such a message whenever the byte length [here length=0x31382050, > base=0x13dde030; notice the square bracket on the left and the round bracket > on the right] exceeds some lower bound such as 256MB. The reason is that > such a large range is most unusual, and may indicate an error. As far as reading the manual is concerned, I did, and in my initial post I stated: > I have attempted to find info on how to "set address range perms", but the best > I have been able to find so far is that this is a message intended for valgrind > developers, not users. The message you repeat above is the one I found in the manual prior to my initial post. Having found nothing else in the manual about this issue, and having found nothing else after some other searching, I thought I might find some help here. My interpretation of [0x13dde030, 0x31382050) was that it is start,end, closed on the left open on the right, not start and length as you suggest. If I interpret 0x31382050 as a size (decimal 825,761,872), I don't see my program ever allocating a block of that size. If I interpret the messages as [start,end) intervals, I do see that they agree with some of the sizes of blocks that my program allocates. I agree with you; I'm absolutely certain that the size of the allocation block is what triggers the message. In my case such a large allocation is not an error unless the system libraries can't handled it. As I've mentioned in both my previous posts, I am trying to determine whether the problem is (a) not really a problem (i.e. problem occurs only when using valgrind), (b) is a problem with the runtime library (the runtime library is leaking a block during realloc), (c) is a problem with valgrind not being able to deal with large allocations, or (d) a problem in my program. (d) seems unlikely because the simple test program exhibits a similar failure. If the problem is (c) I'd like to VERIFY that so I can quit searching, or if there's a way to build valgrind that will allow it to handle programs that allocate large memory blocks, I'd like to know how to do it. If it can't it can't, but I'd like to know I've exhausted all avenues before giving up on this. Bob H |
|
From: John R. <jr...@bi...> - 2011-03-31 20:22:31
|
> As far as reading the manual is concerned, I did, and in my initial
> post I stated:
You are right. I'm sorry I misinterpreted your post.
> My interpretation of [0x13dde030, 0x31382050) was that it is
> start,end, closed on the left open on the right, not start and length
> as you suggest.
You are right again. However, this situation actually re-enforces
a patch that I submitted a few years ago but which was not accepted.
When printing one interval the most helpful notation is "[<base>; +<length>)"
which directly gives the base and the length. The length is [strongly
related to] what the programmer specified to the allocation routine,
and the base is what the allocator returned. It is easier to find the
ceiling by mentally adding (base + length) than to find the length
by mentally subtracting (ceiling - base). [If you use a debugger then
it does not matter.]
When printing more than one interval at a time, then perhaps "[base, ceiling)"
can be justified because [when the list is sorted by base address] then
it is easier to see how close the intervals are to being a partition
of some larger interval.
> I am trying to determine whether the problem is
> (a) not really a problem (i.e.
> problem occurs only when using valgrind),
One definite problem is peculiar messages from valgrind,
as given by the smaller test case.
Whether your original program has a problem _without_ valgrind
is not particularly evident (it isn't easy to tell.)
> (b) is a problem with the
> runtime library (the runtime library is leaking a block during
> realloc),
This is unlikely, although the same problems of part (c)
might apply to the runtime library, too.
> (c) is a problem with valgrind not being able to deal with
> large allocations,
This is somewhat likely. Somewhere in valgrind a 'size_t' was truncated
to an 'unsigned', or a case analysis forgot one situation, etc.
> or (d) a problem in my program.
Perhaps.
Because you have a somewhat small synthetic test case which triggers
the problem (and it is easy to confirm that the problem exists),
then I recommend filing a bug report at:
http://bugs.kde.org/enter_valgrind_bug.cgi
--
|
|
From: Tom H. <to...@co...> - 2011-03-31 20:25:15
|
On 31/03/11 20:32, Bob Harris wrote: > I agree with you; I'm absolutely certain that the size of the > allocation block is what triggers the message. In my case such a > large allocation is not an error unless the system libraries can't > handled it. As I've mentioned in both my previous posts, I am trying > to determine whether the problem is (a) not really a problem (i.e. > problem occurs only when using valgrind), (b) is a problem with the > runtime library (the runtime library is leaking a block during > realloc), (c) is a problem with valgrind not being able to deal with > large allocations, or (d) a problem in my program. > > (d) seems unlikely because the simple test program exhibits a similar > failure. If the problem is (c) I'd like to VERIFY that so I can quit > searching, or if there's a way to build valgrind that will allow it to > handle programs that allocate large memory blocks, I'd like to know > how to do it. If it can't it can't, but I'd like to know I've > exhausted all avenues before giving up on this. I've looked back at your original post and the main problem with it is that you didn't really provide any information (like the text of the error reports from valgrind) that might let us help you. Instead you seemed to have jumped to a conclusion that valgrind is somehow broken and start speculating that maybe it can't handle large chunks of memory. The simple answer is that, assuming you don't actually run of of memory or address space, valgrind really couldn't care less how small or large the chunks of memory you use are. If you were to run of space then you would get an error. Valgrind has told you that you are reading more data into a buffer than you have allocated space for - it is almost certainly correct. Trust it and investigate the problem it is telling you about. Ignore the segfault for now - that is just a consequence of you overrunning the buffer. It not happening outside of valgrind is of no consequence, it just means you are getting lucky. Under valgrind the memory layout will be different and you are happening to overwrite something that then causes the crash. And yes, as John has told you, you can ignore those warnings about large address ranges. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Bob H. <rsh...@gm...> - 2011-03-31 21:08:30
|
On Mar 31, 2011, at 4:25 PM, Tom Hughes wrote: > I've looked back at your original post and the main problem with it is that > you didn't really provide any information (like the text of the error reports > from valgrind) that might let us help you. Indeed, though I did paraphrase the reports in an attempt to keep the message shorter. I originally wrote it much longer, decided the specific error text would be TMI, and shortened that. After getting no reponses, one of the things I did do in my second post was include the complete error text. How 'bout we forget the first post, which everyone ignored anyway, and concentrate on the second one. > Instead you seemed to have jumped to a conclusion that valgrind is somehow > broken and start speculating that maybe it can't handle large chunks of > memory. True, that is the conclusion I jumped to. Limited info in the manual regarding "address range perms" was strongly suggestive that something exceptional had happened inside valgrind, something that it wasn't written to handle. > The simple answer is that, assuming you don't actually run of of memory or > address space, valgrind really couldn't care less how small or large the > chunks of memory you use are. If you were to run of space then you would get > an error. Thanks, it is helpful to know valgrind will report such an error. I'm assuming you mean that valgrind would report an error in that case, that a call to malloc or realloc has failed. And it is helpful to know that valgrind doesn't care about the size of my memory blocks. Though I can imagine how valgrind is implemented, I don't know the specifics. One can imagine that valgrind itself must be allocating *some* memory for its own use. Theoretically this could push it past some allocation limit. One can also imagine that valgrind has some shortcoming like tracking block sizes in a 32-bit variable. I'm not saying this is true, I'm just saying that if one doesn't know the internals and sees the information I saw, those are valid possible sources of the problem. So one question I have though is what "you" means in your last sentence. Does it mean that my target program has run out of memory, or the combination of valgrind and my target, together, has run out of memory. > Valgrind has told you that you are reading more data into a buffer than you > have allocated space for - it is almost certainly correct. Trust it and > investigate the problem it is telling you about. Hmmm... that may be true in the original target (though, at the risk of stepping on toes, I am still not convinced that this isn't an artifact of a failure of large memory allocation when valgrind is present). But in the simple test case I don't do anything with the allocated memory but free it. And yet I get some of the same complaints from valgrind. > Ignore the segfault for now - that is just a consequence of you overrunning > the buffer. It not happening outside of valgrind is of no consequence, it > just means you are getting lucky. Under valgrind the memory layout will be > different and you are happening to overwrite something that then causes the > crash. Yep. I do understand all that. I've been a working programmer for 35 years. About a third of that experience was writing in assembly language, and in fact implementing malloc-type allocators in several architectures. And I'm not saying that to puff up my chest or anything, just to give you an idea what my level of knowledge is. Because of the behavior of valgrind on the test case, it seems the best avenue to pursue is why the test case causes valgrind to report that a block hasn't been freed, when clearly it *has* been freed. In the test case the only possibilities for where the error is are (a) runtime library, (b) compiler, or (c) valgrind. > And yes, as John has told you, you can ignore those warnings about large > address ranges. Hmmm... but looking at the failure of my test program, the large address warnings look like the only useful clues. Thanks, Bob H |
|
From: Tom H. <to...@co...> - 2011-03-31 21:51:15
|
On 31/03/11 22:08, Bob Harris wrote: > On Mar 31, 2011, at 4:25 PM, Tom Hughes wrote: > >> The simple answer is that, assuming you don't actually run of of memory or >> address space, valgrind really couldn't care less how small or large the >> chunks of memory you use are. If you were to run of space then you would get >> an error. > > Thanks, it is helpful to know valgrind will report such an error. I'm > assuming you > mean that valgrind would report an error in that case, that a call to > malloc or realloc > has failed. Well either valgrind will report an error, or the function will return a null pointer in the normal way - it probably depends on exactly what fails which happens. > So one question I have though is what "you" means in your last sentence. Does > it mean that my target program has run out of memory, or the combination of > valgrind and my target, together, has run out of memory. Either could happen - if the primary allocation fails then I believe you will get a null pointer back in the normal way If the shadow memory allocation fails then I think you will get an error from valgrind instead. >> Valgrind has told you that you are reading more data into a buffer than you >> have allocated space for - it is almost certainly correct. Trust it and >> investigate the problem it is telling you about. > > Hmmm... that may be true in the original target (though, at the risk > of stepping on > toes, I am still not convinced that this isn't an artifact of a > failure of large memory > allocation when valgrind is present). But in the simple test case I > don't do anything > with the allocated memory but free it. And yet I get some of the same > complaints > from valgrind. I hadn't looked at the test program before, but I think I know what is happening there... You aren't checking the realloc return value, and when realloc fails it returns null but leaves the block intact. My guess is that it is failing and then the following realloc calls are doing realloc on a null pointer (ie malloc) and probably failing as well. At the end you free a null pointer (defined as safe) and are then left with the orphaned block from the point when the first failure occurred. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Bob H. <rsh...@gm...> - 2011-03-31 23:43:09
|
On Mar 31, 2011, at 5:51 PM, Tom Hughes wrote:
> I hadn't looked at the test program before, but I think I know what is
> happening there...
>
> You aren't checking the realloc return value, and when realloc fails it
> returns null but leaves the block intact. My guess is that it is failing and
> then the following realloc calls are doing realloc on a null pointer (ie
> malloc) and probably failing as well.
>
> At the end you free a null pointer (defined as safe) and are then left with
> the orphaned block from the point when the first failure occurred.
Oh, rats. That is correct, and that is what is happening in the test
program. It's should not be what is happening in the original program
though (as I will explain).
I changed my test program generator to check for a NULL pointer after
every malloc or realloc, and print a message like this:
if (m14==NULL) printf("demo program failed re-allocating
2736652288 to 2814722048 bytes for m14 at point 43\n");
where the message reflects what the alloc or realloc was supposed to
do. The new test program is at the end of this message (but word wrap
may do strange things to it).
Now, if I run the test program without valgrind, it finishes without
printing anything. But with valgrind, it fails, as you predicted.
Output is shown below.
Thanks very much for figuring that out-- I should know better. I can
try to blame it on hastily writing the converter that generated the
test program. But we all know the would be just trying to rationalize
my own screw up.
OK, So, I conclude from that the presence of valgrind is contributing
to the failure of the test program. Possibly there is something about
my build that is different with valgrind than without that causes the
underlying heap allocator to behave differently. I note that the
failure occurs around 2.8G, which is reminiscent of where I ran into
allocation limitations on a different unix machine, a machine with 4G
of physical RAM and a 32-bit address, about a year ago. Could it be
that when I build for valgrind I'm somehow building in 32-bit mode
(recall that this is an x86_64 with 32G physical RAM). I've appended
my Makefile in case anyone can see something obvious from it (I'm not
using any IDE for this, just a bash shell and "make" from the command
line).
Now, as to why I don't believe that I made the same stupid mistake in
the real program...
In the real program all alloc, realloc and free calls go through
wrapper routines (but see caveat at the end of this paragraph). The
wrapper routines check for a NULL pointer from malloc or realloc. If
the pointer is NULL, the program kills itself (and tells the user
why). Admittedly, the reporting mechanism uses things like fflush and
fprintf ad vfprintf before it eventually calls exit(). So it is
conceivable that those routines try to do some allocation, fail and
then failt to report. Hmmm... (here's the caveat) if any system
routine calls alloc or malloc, my wrappers are avoided, regardless of
when they happen. So I can't completely rule out a NULL pointer
problem.
But I still think the possibility of a large memory issue in valgrind
is the most promising thing to pursue.
Bob H
~~~~ command output ~~~~
$ valgrind --leak-check=full ./lz_memory_test_valgrind
==12345== Memcheck, a memory error detector
==12345== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==12345== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==12345== Command: ./lz_memory_test_valgrind
==12345==
==12345== Warning: set address range perms: large range [0x13dde030,
0x31382050) (noaccess)
==12345== Warning: set address range perms: large range [0x3901d030,
0x6229d050) (noaccess)
==12345== Warning: set address range perms: large range [0x6229e030,
0x96b6a050) (noaccess)
==12345== Warning: set address range perms: large range [0x96b6b030,
0xd60bf050) (noaccess)
==12345== Warning: set address range perms: large range [0xd60c0030,
0x11f944050) (noaccess)
==12345== Warning: set address range perms: large range [0x11f945030,
0x17298d050) (noaccess)
==12345== Warning: set address range perms: large range [0x17298e030,
0x1cedea050) (noaccess)
==12345== Warning: set address range perms: large range [0x1cedeb030,
0x233ddb050) (noaccess)
==12345== Warning: set address range perms: large range [0x233ddc030,
0x2a1478050) (noaccess)
==12345== Warning: set address range perms: large range [0x2a1479030,
0x316c59050) (noaccess)
==12345== Warning: set address range perms: large range [0x316c5a030,
0x3944fa050) (noaccess)
==12345== Warning: set address range perms: large range [0x43dd9b030,
0x4c35e3050) (noaccess)
==12345== Warning: set address range perms: large range [0x4c766c030,
0x553c88050) (noaccess)
==12345== Warning: set address range perms: large range [0x557cad030,
0x5ea929050) (noaccess)
==12345== Warning: set address range perms: large range [0x5ee492030,
0x6872d6050) (noaccess)
==12345== Warning: set address range perms: large range [0x68a7e7030,
0x728c57050) (noaccess)
demo program failed re-allocating 2736652288 to 2814722048 bytes for
m14 at point 43
demo program failed re-allocating 2814722048 to 2877751296 bytes for
m14 at point 44
demo program failed re-allocating 2877751296 to 2937126912 bytes for
m14 at point 45
demo program failed re-allocating 2937126912 to 2996256768 bytes for
m14 at point 46
demo program failed re-allocating 2996256768 to 3047555072 bytes for
m14 at point 47
... snipped the rest -- nearly all of the rest fail, but not all
==12345==
==12345== HEAP SUMMARY:
==12345== in use at exit: 2,736,652,288 bytes in 1 blocks
==12345== total heap usage: 140 allocs, 64 frees, 33,075,445,557
bytes allocated
==12345==
==12345== 2,736,652,288 bytes in 1 blocks are definitely lost in loss
record 1 of 1
==12345== at 0x4A05F1D: realloc (vg_replace_malloc.c:476)
==12345== by 0x400C89: main (lz_memory_test.c:83)
==12345==
==12345== LEAK SUMMARY:
==12345== definitely lost: 2,736,652,288 bytes in 1 blocks
==12345== indirectly lost: 0 bytes in 0 blocks
==12345== possibly lost: 0 bytes in 0 blocks
==12345== still reachable: 0 bytes in 0 blocks
==12345== suppressed: 0 bytes in 0 blocks
==12345==
==12345== For counts of detected and suppressed errors, rerun with: -v
==12345== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
~~~~ new test program ~~~~
(same program as earlier post, but tests for NULL added)
#include <stdlib.h>
#include <stdio.h>
int main (int argc, char** argv);
int main (int argc, char** argv) {
void* m1;
void* m2;
void* m3;
void* m4;
void* m5;
void* m6;
void* m7;
void* m8;
void* m9;
void* m10;
void* m11;
void* m12;
void* m13;
void* m14;
void* m15;
void* m16;
void* m17;
void* m18;
void* m19;
void* m20;
void* m21;
void* m22;
void* m23;
void* m24;
void* m25;
void* m26;
void* m27;
void* m28;
void* m29;
void* m30;
void* m31;
void* m32;
void* m33;
void* m34;
m1=malloc((size_t)218); if (m1==NULL) printf("demo program failed
allocating 218 bytes for m1 at point 1\n");
m2=malloc((size_t)51); if (m2==NULL) printf("demo program failed
allocating 51 bytes for m2 at point 2\n");
m3=malloc((size_t)32); if (m3==NULL) printf("demo program failed
allocating 32 bytes for m3 at point 3\n");
m4=malloc((size_t)20); if (m4==NULL) printf("demo program failed
allocating 20 bytes for m4 at point 4\n");
m5=malloc((size_t)16); if (m5==NULL) printf("demo program failed
allocating 16 bytes for m5 at point 5\n");
m6=malloc((size_t)263984); if (m6==NULL) printf("demo program failed
allocating 263984 bytes for m6 at point 6\n");
m7=malloc((size_t)184); if (m7==NULL) printf("demo program failed
allocating 184 bytes for m7 at point 7\n");
m8=malloc((size_t)263984); if (m8==NULL) printf("demo program failed
allocating 263984 bytes for m8 at point 8\n");
free(m4);
m9=malloc((size_t)632); if (m9==NULL) printf("demo program failed
allocating 632 bytes for m9 at point 9\n");
m10=malloc((size_t)1024); if (m10==NULL) printf("demo program failed
allocating 1024 bytes for m10 at point 10\n");
m11=malloc((size_t)51); if (m11==NULL) printf("demo program failed
allocating 51 bytes for m11 at point 11\n");
m12=realloc(NULL,(size_t)16384); if (m12==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m12 at point 12\n");
m13=realloc(NULL,(size_t)16384); if (m13==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m13 at point 13\n");
m14=realloc(NULL,(size_t)16384); if (m14==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m14 at point 14\n");
m15=realloc(NULL,(size_t)5); if (m15==NULL) printf("demo program
failed re-allocating 0 to 5 bytes for m15 at point 15\n");
m14=realloc(m14,(size_t)249266176); if (m14==NULL) printf("demo
program failed re-allocating 16384 to 249266176 bytes for m14 at point
16\n");
m16=realloc(NULL,(size_t)156); if (m16==NULL) printf("demo program
failed re-allocating 0 to 156 bytes for m16 at point 17\n");
m17=realloc(NULL,(size_t)156); if (m17==NULL) printf("demo program
failed re-allocating 0 to 156 bytes for m17 at point 18\n");
m18=realloc(NULL,(size_t)1300108); if (m18==NULL) printf("demo
program failed re-allocating 0 to 1300108 bytes for m18 at point
19\n");
m19=realloc(NULL,(size_t)1300108); if (m19==NULL) printf("demo
program failed re-allocating 0 to 1300108 bytes for m19 at point
20\n");
m20=realloc(NULL,(size_t)5); if (m20==NULL) printf("demo program
failed re-allocating 0 to 5 bytes for m20 at point 21\n");
m14=realloc(m14,(size_t)492453888); if (m14==NULL) printf("demo
program failed re-allocating 249266176 to 492453888 bytes for m14 at
point 22\n");
m14=realloc(m14,(size_t)690487296); if (m14==NULL) printf("demo
program failed re-allocating 492453888 to 690487296 bytes for m14 at
point 23\n");
m14=realloc(m14,(size_t)881639424); if (m14==NULL) printf("demo
program failed re-allocating 690487296 to 881639424 bytes for m14 at
point 24\n");
m14=realloc(m14,(size_t)1062551552); if (m14==NULL) printf("demo
program failed re-allocating 881639424 to 1062551552 bytes for m14 at
point 25\n");
m14=realloc(m14,(size_t)1233666048); if (m14==NULL) printf("demo
program failed re-allocating 1062551552 to 1233666048 bytes for m14 at
point 26\n");
m14=realloc(m14,(size_t)1392803840); if (m14==NULL) printf("demo
program failed re-allocating 1233666048 to 1392803840 bytes for m14 at
point 27\n");
m14=realloc(m14,(size_t)1548075008); if (m14==NULL) printf("demo
program failed re-allocating 1392803840 to 1548075008 bytes for m14 at
point 28\n");
m14=realloc(m14,(size_t)1694433280); if (m14==NULL) printf("demo
program failed re-allocating 1548075008 to 1694433280 bytes for m14 at
point 29\n");
m14=realloc(m14,(size_t)1835646976); if (m14==NULL) printf("demo
program failed re-allocating 1694433280 to 1835646976 bytes for m14 at
point 30\n");
m16=realloc(m16,(size_t)160); if (m16==NULL) printf("demo program
failed re-allocating 156 to 160 bytes for m16 at point 31\n");
m17=realloc(m17,(size_t)160); if (m17==NULL) printf("demo program
failed re-allocating 156 to 160 bytes for m17 at point 32\n");
m15=realloc(m15,(size_t)6); if (m15==NULL) printf("demo program
failed re-allocating 5 to 6 bytes for m15 at point 33\n");
m14=realloc(m14,(size_t)1971191808); if (m14==NULL) printf("demo
program failed re-allocating 1835646976 to 1971191808 bytes for m14 at
point 34\n");
m20=realloc(m20,(size_t)6); if (m20==NULL) printf("demo program
failed re-allocating 5 to 6 bytes for m20 at point 35\n");
m14=realloc(m14,(size_t)2106195968); if (m14==NULL) printf("demo
program failed re-allocating 1971191808 to 2106195968 bytes for m14 at
point 36\n");
m14=realloc(m14,(size_t)2240053248); if (m14==NULL) printf("demo
program failed re-allocating 2106195968 to 2240053248 bytes for m14 at
point 37\n");
m14=realloc(m14,(size_t)2355216384); if (m14==NULL) printf("demo
program failed re-allocating 2240053248 to 2355216384 bytes for m14 at
point 38\n");
m14=realloc(m14,(size_t)2462564352); if (m14==NULL) printf("demo
program failed re-allocating 2355216384 to 2462564352 bytes for m14 at
point 39\n");
m14=realloc(m14,(size_t)2565095424); if (m14==NULL) printf("demo
program failed re-allocating 2462564352 to 2565095424 bytes for m14 at
point 40\n");
m14=realloc(m14,(size_t)2655453184); if (m14==NULL) printf("demo
program failed re-allocating 2565095424 to 2655453184 bytes for m14 at
point 41\n");
m14=realloc(m14,(size_t)2736652288); if (m14==NULL) printf("demo
program failed re-allocating 2655453184 to 2736652288 bytes for m14 at
point 42\n");
m14=realloc(m14,(size_t)2814722048); if (m14==NULL) printf("demo
program failed re-allocating 2736652288 to 2814722048 bytes for m14 at
point 43\n");
m14=realloc(m14,(size_t)2877751296); if (m14==NULL) printf("demo
program failed re-allocating 2814722048 to 2877751296 bytes for m14 at
point 44\n");
m14=realloc(m14,(size_t)2937126912); if (m14==NULL) printf("demo
program failed re-allocating 2877751296 to 2937126912 bytes for m14 at
point 45\n");
m14=realloc(m14,(size_t)2996256768); if (m14==NULL) printf("demo
program failed re-allocating 2937126912 to 2996256768 bytes for m14 at
point 46\n");
m14=realloc(m14,(size_t)3047555072); if (m14==NULL) printf("demo
program failed re-allocating 2996256768 to 3047555072 bytes for m14 at
point 47\n");
m14=realloc(m14,(size_t)3095691264); if (m14==NULL) printf("demo
program failed re-allocating 3047555072 to 3095691264 bytes for m14 at
point 48\n");
m15=realloc(m15,(size_t)15); if (m15==NULL) printf("demo program
failed re-allocating 6 to 15 bytes for m15 at point 49\n");
m14=realloc(m14,(size_t)3100606464); if (m14==NULL) printf("demo
program failed re-allocating 3095691264 to 3100606464 bytes for m14 at
point 50\n");
m20=realloc(m20,(size_t)15); if (m20==NULL) printf("demo program
failed re-allocating 6 to 15 bytes for m20 at point 51\n");
m14=realloc(m14,(size_t)3105439744); if (m14==NULL) printf("demo
program failed re-allocating 3100606464 to 3105439744 bytes for m14 at
point 52\n");
m14=realloc(m14,(size_t)3110240256); if (m14==NULL) printf("demo
program failed re-allocating 3105439744 to 3110240256 bytes for m14 at
point 53\n");
m14=realloc(m14,(size_t)3114926080); if (m14==NULL) printf("demo
program failed re-allocating 3110240256 to 3114926080 bytes for m14 at
point 54\n");
m14=realloc(m14,(size_t)3119546368); if (m14==NULL) printf("demo
program failed re-allocating 3114926080 to 3119546368 bytes for m14 at
point 55\n");
m14=realloc(m14,(size_t)3124166656); if (m14==NULL) printf("demo
program failed re-allocating 3119546368 to 3124166656 bytes for m14 at
point 56\n");
m14=realloc(m14,(size_t)3128770560); if (m14==NULL) printf("demo
program failed re-allocating 3124166656 to 3128770560 bytes for m14 at
point 57\n");
m15=realloc(m15,(size_t)16); if (m15==NULL) printf("demo program
failed re-allocating 15 to 16 bytes for m15 at point 58\n");
m14=realloc(m14,(size_t)3130458112); if (m14==NULL) printf("demo
program failed re-allocating 3128770560 to 3130458112 bytes for m14 at
point 59\n");
m20=realloc(m20,(size_t)16); if (m20==NULL) printf("demo program
failed re-allocating 15 to 16 bytes for m20 at point 60\n");
m14=realloc(m14,(size_t)3131047936); if (m14==NULL) printf("demo
program failed re-allocating 3130458112 to 3131047936 bytes for m14 at
point 61\n");
m15=realloc(m15,(size_t)21); if (m15==NULL) printf("demo program
failed re-allocating 16 to 21 bytes for m15 at point 62\n");
m14=realloc(m14,(size_t)3131588608); if (m14==NULL) printf("demo
program failed re-allocating 3131047936 to 3131588608 bytes for m14 at
point 63\n");
m20=realloc(m20,(size_t)21); if (m20==NULL) printf("demo program
failed re-allocating 16 to 21 bytes for m20 at point 64\n");
m14=realloc(m14,(size_t)3131801600); if (m14==NULL) printf("demo
program failed re-allocating 3131588608 to 3131801600 bytes for m14 at
point 65\n");
m14=realloc(m14,(size_t)3131998208); if (m14==NULL) printf("demo
program failed re-allocating 3131801600 to 3131998208 bytes for m14 at
point 66\n");
m14=realloc(m14,(size_t)3132178432); if (m14==NULL) printf("demo
program failed re-allocating 3131998208 to 3132178432 bytes for m14 at
point 67\n");
m14=realloc(m14,(size_t)3132375040); if (m14==NULL) printf("demo
program failed re-allocating 3132178432 to 3132375040 bytes for m14 at
point 68\n");
m14=realloc(m14,(size_t)3132555264); if (m14==NULL) printf("demo
program failed re-allocating 3132375040 to 3132555264 bytes for m14 at
point 69\n");
m14=realloc(m14,(size_t)3132735488); if (m14==NULL) printf("demo
program failed re-allocating 3132555264 to 3132735488 bytes for m14 at
point 70\n");
m14=realloc(m14,(size_t)3132932096); if (m14==NULL) printf("demo
program failed re-allocating 3132735488 to 3132932096 bytes for m14 at
point 71\n");
m14=realloc(m14,(size_t)3133112320); if (m14==NULL) printf("demo
program failed re-allocating 3132932096 to 3133112320 bytes for m14 at
point 72\n");
m14=realloc(m14,(size_t)3133292544); if (m14==NULL) printf("demo
program failed re-allocating 3133112320 to 3133292544 bytes for m14 at
point 73\n");
m14=realloc(m14,(size_t)3133472768); if (m14==NULL) printf("demo
program failed re-allocating 3133292544 to 3133472768 bytes for m14 at
point 74\n");
m15=realloc(m15,(size_t)22); if (m15==NULL) printf("demo program
failed re-allocating 21 to 22 bytes for m15 at point 75\n");
m14=realloc(m14,(size_t)3133636608); if (m14==NULL) printf("demo
program failed re-allocating 3133472768 to 3133636608 bytes for m14 at
point 76\n");
m20=realloc(m20,(size_t)22); if (m20==NULL) printf("demo program
failed re-allocating 21 to 22 bytes for m20 at point 77\n");
m14=realloc(m14,(size_t)3133816832); if (m14==NULL) printf("demo
program failed re-allocating 3133636608 to 3133816832 bytes for m14 at
point 78\n");
m14=realloc(m14,(size_t)3133980672); if (m14==NULL) printf("demo
program failed re-allocating 3133816832 to 3133980672 bytes for m14 at
point 79\n");
m14=realloc(m14,(size_t)3134160896); if (m14==NULL) printf("demo
program failed re-allocating 3133980672 to 3134160896 bytes for m14 at
point 80\n");
m14=realloc(m14,(size_t)3134324736); if (m14==NULL) printf("demo
program failed re-allocating 3134160896 to 3134324736 bytes for m14 at
point 81\n");
m14=realloc(m14,(size_t)3134488576); if (m14==NULL) printf("demo
program failed re-allocating 3134324736 to 3134488576 bytes for m14 at
point 82\n");
m14=realloc(m14,(size_t)3134652416); if (m14==NULL) printf("demo
program failed re-allocating 3134488576 to 3134652416 bytes for m14 at
point 83\n");
m14=realloc(m14,(size_t)3134816256); if (m14==NULL) printf("demo
program failed re-allocating 3134652416 to 3134816256 bytes for m14 at
point 84\n");
m14=realloc(m14,(size_t)3134980096); if (m14==NULL) printf("demo
program failed re-allocating 3134816256 to 3134980096 bytes for m14 at
point 85\n");
m14=realloc(m14,(size_t)3135143936); if (m14==NULL) printf("demo
program failed re-allocating 3134980096 to 3135143936 bytes for m14 at
point 86\n");
m14=realloc(m14,(size_t)3135291392); if (m14==NULL) printf("demo
program failed re-allocating 3135143936 to 3135291392 bytes for m14 at
point 87\n");
m14=realloc(m14,(size_t)3135438848); if (m14==NULL) printf("demo
program failed re-allocating 3135291392 to 3135438848 bytes for m14 at
point 88\n");
m14=realloc(m14,(size_t)3135553536); if (m14==NULL) printf("demo
program failed re-allocating 3135438848 to 3135553536 bytes for m14 at
point 89\n");
m14=realloc(m14,(size_t)3135684608); if (m14==NULL) printf("demo
program failed re-allocating 3135553536 to 3135684608 bytes for m14 at
point 90\n");
m14=realloc(m14,(size_t)3135799296); if (m14==NULL) printf("demo
program failed re-allocating 3135684608 to 3135799296 bytes for m14 at
point 91\n");
m14=realloc(m14,(size_t)3135881216); if (m14==NULL) printf("demo
program failed re-allocating 3135799296 to 3135881216 bytes for m14 at
point 92\n");
m14=realloc(m14,(size_t)3135979520); if (m14==NULL) printf("demo
program failed re-allocating 3135881216 to 3135979520 bytes for m14 at
point 93\n");
m14=realloc(m14,(size_t)3136061440); if (m14==NULL) printf("demo
program failed re-allocating 3135979520 to 3136061440 bytes for m14 at
point 94\n");
m14=realloc(m14,(size_t)3136110592); if (m14==NULL) printf("demo
program failed re-allocating 3136061440 to 3136110592 bytes for m14 at
point 95\n");
m14=realloc(m14,(size_t)3136159744); if (m14==NULL) printf("demo
program failed re-allocating 3136110592 to 3136159744 bytes for m14 at
point 96\n");
m14=realloc(m14,(size_t)3136192512); if (m14==NULL) printf("demo
program failed re-allocating 3136159744 to 3136192512 bytes for m14 at
point 97\n");
m14=realloc(m14,(size_t)3136241664); if (m14==NULL) printf("demo
program failed re-allocating 3136192512 to 3136241664 bytes for m14 at
point 98\n");
m14=realloc(m14,(size_t)3136290816); if (m14==NULL) printf("demo
program failed re-allocating 3136241664 to 3136290816 bytes for m14 at
point 99\n");
m14=realloc(m14,(size_t)3136323584); if (m14==NULL) printf("demo
program failed re-allocating 3136290816 to 3136323584 bytes for m14 at
point 100\n");
m14=realloc(m14,(size_t)3136372736); if (m14==NULL) printf("demo
program failed re-allocating 3136323584 to 3136372736 bytes for m14 at
point 101\n");
m14=realloc(m14,(size_t)3136405504); if (m14==NULL) printf("demo
program failed re-allocating 3136372736 to 3136405504 bytes for m14 at
point 102\n");
m14=realloc(m14,(size_t)3136454656); if (m14==NULL) printf("demo
program failed re-allocating 3136405504 to 3136454656 bytes for m14 at
point 103\n");
m14=realloc(m14,(size_t)3136487424); if (m14==NULL) printf("demo
program failed re-allocating 3136454656 to 3136487424 bytes for m14 at
point 104\n");
m14=realloc(m14,(size_t)3136536576); if (m14==NULL) printf("demo
program failed re-allocating 3136487424 to 3136536576 bytes for m14 at
point 105\n");
m14=realloc(m14,(size_t)3136569344); if (m14==NULL) printf("demo
program failed re-allocating 3136536576 to 3136569344 bytes for m14 at
point 106\n");
m14=realloc(m14,(size_t)3136618496); if (m14==NULL) printf("demo
program failed re-allocating 3136569344 to 3136618496 bytes for m14 at
point 107\n");
m14=realloc(m14,(size_t)3136651264); if (m14==NULL) printf("demo
program failed re-allocating 3136618496 to 3136651264 bytes for m14 at
point 108\n");
m14=realloc(m14,(size_t)3136684032); if (m14==NULL) printf("demo
program failed re-allocating 3136651264 to 3136684032 bytes for m14 at
point 109\n");
m14=realloc(m14,(size_t)3136733184); if (m14==NULL) printf("demo
program failed re-allocating 3136684032 to 3136733184 bytes for m14 at
point 110\n");
m14=realloc(m14,(size_t)3136765952); if (m14==NULL) printf("demo
program failed re-allocating 3136733184 to 3136765952 bytes for m14 at
point 111\n");
m14=realloc(m14,(size_t)3136798720); if (m14==NULL) printf("demo
program failed re-allocating 3136765952 to 3136798720 bytes for m14 at
point 112\n");
m14=realloc(m14,(size_t)3136847872); if (m14==NULL) printf("demo
program failed re-allocating 3136798720 to 3136847872 bytes for m14 at
point 113\n");
m14=realloc(m14,(size_t)3136880640); if (m14==NULL) printf("demo
program failed re-allocating 3136847872 to 3136880640 bytes for m14 at
point 114\n");
m14=realloc(m14,(size_t)3136913408); if (m14==NULL) printf("demo
program failed re-allocating 3136880640 to 3136913408 bytes for m14 at
point 115\n");
m14=realloc(m14,(size_t)3136946176); if (m14==NULL) printf("demo
program failed re-allocating 3136913408 to 3136946176 bytes for m14 at
point 116\n");
m14=realloc(m14,(size_t)3136995328); if (m14==NULL) printf("demo
program failed re-allocating 3136946176 to 3136995328 bytes for m14 at
point 117\n");
m14=realloc(m14,(size_t)3137028096); if (m14==NULL) printf("demo
program failed re-allocating 3136995328 to 3137028096 bytes for m14 at
point 118\n");
m14=realloc(m14,(size_t)3137060864); if (m14==NULL) printf("demo
program failed re-allocating 3137028096 to 3137060864 bytes for m14 at
point 119\n");
m14=realloc(m14,(size_t)3137093632); if (m14==NULL) printf("demo
program failed re-allocating 3137060864 to 3137093632 bytes for m14 at
point 120\n");
m14=realloc(m14,(size_t)3137110016); if (m14==NULL) printf("demo
program failed re-allocating 3137093632 to 3137110016 bytes for m14 at
point 121\n");
m14=realloc(m14,(size_t)3137126400); if (m14==NULL) printf("demo
program failed re-allocating 3137110016 to 3137126400 bytes for m14 at
point 122\n");
m14=realloc(m14,(size_t)3137142784); if (m14==NULL) printf("demo
program failed re-allocating 3137126400 to 3137142784 bytes for m14 at
point 123\n");
m14=realloc(m14,(size_t)3137159168); if (m14==NULL) printf("demo
program failed re-allocating 3137142784 to 3137159168 bytes for m14 at
point 124\n");
m14=realloc(m14,(size_t)3137175552); if (m14==NULL) printf("demo
program failed re-allocating 3137159168 to 3137175552 bytes for m14 at
point 125\n");
m21=malloc((size_t)632); if (m21==NULL) printf("demo program failed
allocating 632 bytes for m21 at point 126\n");
m22=malloc((size_t)1024); if (m22==NULL) printf("demo program failed
allocating 1024 bytes for m22 at point 127\n");
m23=malloc((size_t)32); if (m23==NULL) printf("demo program failed
allocating 32 bytes for m23 at point 128\n");
m24=realloc(NULL,(size_t)16384); if (m24==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m24 at point 129\n");
m25=realloc(NULL,(size_t)16384); if (m25==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m25 at point 130\n");
m26=malloc((size_t)12615754384); if (m26==NULL) printf("demo program
failed allocating 12615754384 bytes for m26 at point 131\n");
m27=malloc((size_t)160032); if (m27==NULL) printf("demo program
failed allocating 160032 bytes for m27 at point 132\n");
m28=realloc(NULL,(size_t)16384); if (m28==NULL) printf("demo program
failed re-allocating 0 to 16384 bytes for m28 at point 133\n");
m29=realloc(NULL,(size_t)30); if (m29==NULL) printf("demo program
failed re-allocating 0 to 30 bytes for m29 at point 134\n");
m30=realloc(NULL,(size_t)8); if (m30==NULL) printf("demo program
failed re-allocating 0 to 8 bytes for m30 at point 135\n");
m30=realloc(m30,(size_t)9); if (m30==NULL) printf("demo program
failed re-allocating 8 to 9 bytes for m30 at point 136\n");
m31=malloc((size_t)16777216); if (m31==NULL) printf("demo program
failed allocating 16777216 bytes for m31 at point 137\n");
m32=malloc((size_t)16777216); if (m32==NULL) printf("demo program
failed allocating 16777216 bytes for m32 at point 138\n");
m33=malloc((size_t)33554432); if (m33==NULL) printf("demo program
failed allocating 33554432 bytes for m33 at point 139\n");
m34=malloc((size_t)16777216); if (m34==NULL) printf("demo program
failed allocating 16777216 bytes for m34 at point 140\n");
free(m2);
free(m14);
free(m10);
free(m11);
free(m15);
free(m20);
free(m16);
free(m17);
free(m18);
free(m19);
free(m12);
free(m13);
free(m9);
free(m3);
free(m28);
free(m22);
free(m23);
free(m29);
free(m30);
free(m24);
free(m25);
free(m21);
free(m1);
free(m6);
free(m8);
free(m26);
free(m27);
free(m7);
free(m5);
free(m31);
free(m32);
free(m33);
free(m34);
return 0;
}
~~~~ my Makefile ~~~~
CFLAGS=-g -O0 -Wall
PROGS = lz_memory_test lz_memory_valgrind
srcFiles = lz_memory_test \
all: ${PROGS}
lz_memory_test.c: lz_memory_log lz_memory_to_allocation_demo.py
cat lz_memory_log | ./lz_memory_to_allocation_demo.py > lz_memory_test.c
lz_memory_test: lz_memory_test.o
lz_memory_valgrind: lz_memory_test_valgrind.o
lz_memory_test_valgrind.o: lz_memory_test.c
${CC} -c -g -O0 -pg $< -o $@
clean:
rm -f ${PROGS}
rm -f *.o
|
|
From: Tom H. <to...@co...> - 2011-03-31 23:55:57
|
On 01/04/11 00:43, Bob Harris wrote: > OK, So, I conclude from that the presence of valgrind is contributing > to the failure of the test program. Possibly there is something about > my build that is different with valgrind than without that causes the > underlying heap allocator to behave differently. I note that the > failure occurs around 2.8G, which is reminiscent of where I ran into > allocation limitations on a different unix machine, a machine with 4G > of physical RAM and a 32-bit address, about a year ago. Could it be > that when I build for valgrind I'm somehow building in 32-bit mode > (recall that this is an x86_64 with 32G physical RAM). I've appended > my Makefile in case anyone can see something obvious from it (I'm not > using any IDE for this, just a bash shell and "make" from the command > line). It's entirely expected that it may fail under valgrind when it doesn't normally because the memory layout is different plus every byte of memory you allocate has a little over a byte (nine bits) of overhead in the form of the shadow memory valgrind is using to track it, and so on. You will run out of memory more quickly under valgrind that you do normally is the basic message. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Bob H. <rsh...@gm...> - 2011-04-01 02:03:53
|
On Mar 31, 2011, at 7:55 PM, Tom Hughes wrote:
> It's entirely expected that it may fail under valgrind when it doesn't
> normally because the memory layout is different plus every byte of memory
> you allocate has a little over a byte (nine bits) of overhead in the form
> of the shadow memory valgrind is using to track it, and so on.
>
> You will run out of memory more quickly under valgrind that you do normally
> is the basic message.
Sure, understandable. But to go from a program that is able to
allocate nearly 16G without valgrind to one that fails at 2.8G with
it?
The test program has this line:
m26=malloc((size_t)12615754384);
12.6G in a single block and that works fine without valgrind. With
valgrind, memory allocation starts to fail way earlier, and the 12G
attempt also fails.
And in fact, if I comment out that huge 12.6G block and compile on a
32bit machine, and run (without valgrind), I get allocation failures
starting at around 2.4G. This I expect on a 32-bit machine, from past
experience. All this, together with valgrind giving me a message that
seems to be telling me "my, grandmother, but that's a big block you
are trying to allocate", is what makes me suspicious that the valgrind
is doing something that is limiting me to the equivalent of a 32-bit
realm.
Anyway, Tom, I do really appreciate your help on this. My guess is
you probably feel like you are beating a dead horse at this point.
Thanks again,
Bob H
|
|
From: Dan K. <da...@ke...> - 2011-04-01 04:16:15
|
On Fri, Apr 1, 2011 at 2:03 AM, Bob Harris <rsh...@gm...> wrote: >> You will run out of memory more quickly under valgrind that you do normally >> is the basic message. > > Sure, understandable. But to go from a program that is able to > allocate nearly 16G without valgrind to one that fails at 2.8G with > it? That does seem a bit much, I thought Valgrind's memory overhead was an extra bit per byte. - Dan |
|
From: John R. <jr...@bi...> - 2011-04-01 05:28:00
|
First, let's repair the program to detect failure of realloc.
Use a text editor to append the statement:
if (NULL==m14) printf("FAIL line=%d\n", __LINE__);
to every line that does a realloc of m14. For example:
m14 = realloc(m14,(size_t)492453888); if (NULL==m14) printf("FAIL line=%d\n", __LINE__);
When I run it on a machine with 8GB RAM and 12.9 GB swap, it fails on line 82
and every subsequent realloc of m14. So Tom's guess of the behavior is correct.
The test case does not handle a failure of realloc(). [This failure is
determined by glibc and the operating system, and is the _expected_ way
to say, "the system is out of space". Memcheck handles this correctly.]
> ... But to go from a program that is able to
> allocate nearly 16G without valgrind to one that fails at 2.8G with
> it?
There is an interaction between realloc and memcheck that probably accounts
for this. When the test program runs without valgrind, probably m14
soon resides at an "end" of the address space, such that each subsequent
realloc which increases the size of m14 can succeed by placing new pages
adjacent to the existing end, where the size of the new pages is about
the _difference_ between the new size and the old size. The old data bytes
do not move; the return value of the realloc() equals the value of the
first argument.
When the test program is run under valgrind, then valgrind must allocate some
space to hold the accounting data for each block. The accounting data is
9 bits per byte, so if a block is large then the accounting data is large.
In particular, it is plausible that the realloc(m14,...) and the accounting
data "checkerboard" the address space so that each realloc must get enough
pages for the entire new size, copy the old bytes from *m14 to the new pages,
free() the old m14, and return a pointer to the new pages.
The checkerboarding is exacerbated because memcheck does not return free()d
areas immediately to the pool of available space. Instead, free()d areas
are "aged" for a while: put on a queue and free()d later [one at a time].
The aging allows memcheck to detect the error of access to a free()d block,
as long as the access happens while the free()d block is in the queue.
If a free()d block was re-allocated immediately by malloc(), then the bad
access to a recently-free()d block could not be detected, because the old
block was in fact [carved up and] returned by malloc().
So, memcheck causes a lag between free() and "available for use by malloc()."
The effect of this is that usage "balloons" for a while.
--
|
|
From: Tom H. <to...@co...> - 2011-04-01 08:44:58
|
On 01/04/11 06:27, John Reiser wrote: > There is an interaction between realloc and memcheck that probably accounts > for this. When the test program runs without valgrind, probably m14 > soon resides at an "end" of the address space, such that each subsequent > realloc which increases the size of m14 can succeed by placing new pages > adjacent to the existing end, where the size of the new pages is about > the _difference_ between the new size and the old size. The old data bytes > do not move; the return value of the realloc() equals the value of the > first argument. > > When the test program is run under valgrind, then valgrind must allocate some > space to hold the accounting data for each block. The accounting data is > 9 bits per byte, so if a block is large then the accounting data is large. > In particular, it is plausible that the realloc(m14,...) and the accounting > data "checkerboard" the address space so that each realloc must get enough > pages for the entire new size, copy the old bytes from *m14 to the new pages, > free() the old m14, and return a pointer to the new pages. Actually that probably won't happen because valgrind tries to use separate memory areas for client data and it's own data. It doesn't matter though, because valgrind's realloc deliberately forces all calls to do malloc+copy+free rather than resize in place in order to better detect bugs. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Julian S. <js...@ac...> - 2011-04-01 09:56:48
|
[... mucho discussion ...] I'm now comprehensively confused. Is there any bug/failure in Memcheck resulting from this, that needs to be looked at? J |
|
From: Tom H. <to...@co...> - 2011-04-01 10:00:39
|
On 01/04/11 10:50, Julian Seward wrote: > > [... mucho discussion ...] > > I'm now comprehensively confused. Is there any bug/failure in > Memcheck resulting from this, that needs to be looked at? There is no clear evidence of one at the moment. The current situation is that we have proved that the test program really did have a leak that was correctly reported. The only outstanding question there was about why memory allocation was failing under valgrind when it worked normally but that is expected. As to the original program, who knows... It was apparently reporting a read(2) into memory that was beyond an allocated block but we don't really know any more than that. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: John R. <jr...@bi...> - 2011-04-01 14:41:48
|
On 04/01/2011 03:00 AM, Tom Hughes wrote:
> On 01/04/11 10:50, Julian Seward wrote:
>>
>> [... mucho discussion ...]
>>
>> I'm now comprehensively confused. Is there any bug/failure in
>> Memcheck resulting from this, that needs to be looked at?
>
> There is no clear evidence of one at the moment.
>
> The current situation is that we have proved that the test program
> really did have a leak that was correctly reported. The only outstanding
> question there was about why memory allocation was failing under
> valgrind when it worked normally but that is expected.
There is a failure, at least from the user's perspective. When run under
memcheck, then the [corrected] test program runs out of space much sooner
than expected. The space overhead appears to be greater than 20 to 1,
when something like 3:1 might be "normal". When the size of individual
blocks is huge (exceeds (RAM + SWAP)/400, for example) then the default
settings tend to exhaust space very quickly, with symptoms that are
mysterious to non-experts. This is a Usability bug.
Bart Van Assche suggests --freelist-vol=0, which is an excellent suggestion.
For better usability, then memcheck should take more care with huge blocks.
Put a bound of (RAM + SWAP)/8 on the volume of the freelist; or, keep a
separate freelist for huge blocks, and age them faster. For a growing
realloc() of a huge block, then relax the strategy which forces malloc_new +
copy + free_old. Give up some local error detection for the chance to
live long enough to detect more errors later.
If nothing else, then when the first huge block is detected give a warning:
huge block detected (%lu bytes); consider adjusting --freelist-vol
[The values of MemTotal and SwapTotal are in /proc/meminfo. Evaluating
the threshold once is enough even though /sbin/swapon or pluggable RAM
might change it. Disregarding other processes is OK because even novices
know to minimize external contention.]
--
|
|
From: John R. <jr...@bi...> - 2011-04-01 15:38:47
|
> There is a failure, at least from the user's perspective. When run under > memcheck, then the [corrected] test program runs out of space much sooner > than expected. Recorded for tracking purposes at: https://bugs.kde.org/show_bug.cgi?id=269884 -- |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-04-01 15:44:59
|
>Recorded for tracking purposes at: >https://bugs.kde.org/show_bug.cgi?id=269884 I believe this one should be marked as a duplicate of https://bugs.kde.org/show_bug.cgi?id=250101 (which also contains a patch fixing the problem). Philippe ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-04-01 15:42:56
|
>If nothing else, then when the first huge block is detected >give a warning: > huge block detected (%lu bytes); consider adjusting --freelist-vol Note that in this case, the problem is not the free list. The default value of the free list is quite small (20Mb). See my other mail(s) for a description of why this "realloc" program is failing under Valgrind (and the patch improving Valgrind). However, there are real problems with the free list when huge blocks are allocated and freed by the client: there is in fact no (good) protection anymore by memcheck against dangling pointer pointing to big memory blocks, as these have a lot of chance to be re-used directly after free (as they do not go to the free list). I am even suspecting that free-ing a huge block (> freelist-vol) will flush the whole of freelist-vol, so in effect removing all the memcheck dangling pointer protection for all the recently allocated and freed small blocks. I was intending to work on improving the free-list behaviour, but I think it is first needed to have https://bugs.kde.org/show_bug.cgi?id=250101 fixed. (and as always, my current priority is gdbserver patch :). Philippe ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-04-01 15:32:39
|
Hello Julian, >[... mucho discussion ...] > >I'm now comprehensively confused. Is there any bug/failure in >Memcheck resulting from this, that needs to be looked at? I suggest to look at two "fixes/improvements": 1. integrate the patch https://bugs.kde.org/show_bug.cgi?id=250101 (I just tried a 3.7.0 SVN with this (updated) patch, and the program of Bob Harris works now perfectly till the end, without leaks/without out of memory). 2. add a new kind of error in memcheck: "failed (re-)allocation". It is quite frequent to see code which forgets to check the return of malloc/realloc. Memcheck can then at least detect and report specifically when it has really failed. Now, a failed client (re-)alloc is invisible, and makes a lot of people lose a lot of time to re-discover the problem. (and of course, I am still working hard to integrate the gdbserver patch :). I have done the port on ARM, PPC32 and PPC64 last week-end. ARM tests are ok. Hoping to have all the tests working on PPC this week-end. (problems are not bugs which appears, but rather that the tests are extremely difficult to make "insensitive" to the gdb/the libc/the specific OS/...) Philippe ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: John R. <jr...@bi...> - 2011-04-01 16:16:09
|
On 04/01/2011 08:32 AM, WAROQUIERS Philippe wrote: > I suggest to look at two "fixes/improvements": > 1. integrate the patch https://bugs.kde.org/show_bug.cgi?id=250101 > (I just tried a 3.7.0 SVN with this (updated) patch, and > the program of Bob Harris works now perfectly till the end, > without leaks/without out of memory). I have voted +10 for this patch. [Why did it not appear in 3.6.x?] > > 2. add a new kind of error in memcheck: "failed (re-)allocation". > It is quite frequent to see code which forgets to check the > return of malloc/realloc. Memcheck can then at least detect > and report specifically when it has really failed. > Now, a failed client (re-)alloc is invisible, and makes a lot of > people lose a lot of time to re-discover the problem. I support such a warning the first time (each) for malloc and realloc. For realloc because memcheck tries to increase the chances of such an error; for malloc because of general cleanliness. I'm wary of giving a warning every time: too much noise for good programs which correctly check and recover, but that happen to run out of space. -- |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-04-01 17:03:35
|
>> 1. integrate the patch https://bugs.kde.org/show_bug.cgi?id=250101 >I have voted +10 for this patch. [Why did it not appear in 3.6.x?] It was a candidate for 3.6.0, but I understand there was too much work, and too little effort available. For 3.6.1, this patch was not taken, as this is not fixing a real bug, but just improving the amount of resources needed. But I am sure it is again a candidate for 3.7.0 :). >> 2. add a new kind of error in memcheck: "failed (re-)allocation". >I support such a warning the first time (each) for malloc and realloc. >For realloc because memcheck tries to increase the chances of >such an error; >for malloc because of general cleanliness. >I'm wary of giving a warning every time: too much noise for >good programs >which correctly check and recover, but that happen to run out of space. I understand the worry, but at least in my experience, I have not seen much programs which effectively go to an out of memory situation and then check and recover. And in practice, if the "failed (re-)alloc" is handled like all the other errors of Valgrind, the re-occurrence of the same error will be "collapsed" with the previous occurences (giving things such as "10 errors from one context"). So, I believe the best is to not do a specialized kind of warning for that, but just a normal memcheck error which will be handled as usual, can be suppressed as usual with the stack trace, etc. Philippe ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: John R. <jr...@bi...> - 2011-04-02 14:33:19
|
On 04/01/2011 10:03 AM, WAROQUIERS Philippe wrote: >>> 2. add a new kind of error in memcheck: "failed (re-)allocation". >> I support such a warning the first time (each) for malloc and realloc. >> For realloc because memcheck tries to increase the chances of >> such an error; >> for malloc because of general cleanliness. >> I'm wary of giving a warning every time: too much noise for >> good programs >> which correctly check and recover, but that happen to run out of space. > I understand the worry, but at least in my experience, I have not seen > much programs which effectively go to an out of memory situation and > then check and recover. I'd say it's the other way around: most programs already check. C++ implementations of 'new' and 'new[]' always check their results from malloc(), and often those are the only calls to malloc except for internal calls from libc itself, which of course checks. Most non-small projects already allocate indirectly, often using a symbol such as 'xmalloc'. Calls to malloc from third-party libraries are covered by interposing using LD_PRELOAD and RTLD_NEXT. (I admit that effective _recovery_ from 0==malloc() is almost non-existent.) -- |
|
From: WAROQUIERS P. <phi...@eu...> - 2011-04-04 07:51:51
|
>> I understand the worry, but at least in my experience, I have not seen >> much programs which effectively go to an out of memory situation and >> then check and recover. > >I'd say it's the other way around: most programs already check. ... >Calls to malloc from third-party libraries are covered by interposing >using LD_PRELOAD and RTLD_NEXT. (I admit that effective _recovery_ >from 0==malloc() is almost non-existent.) I think we both say the same, i.e. "programs that check **and** recover" is very unfrequent. Taking that into account, I think that handling a failed (re-)alloc like any other Valgrind error (i.e. reporting each occurences) will not cause much disturbances. Philippe ____ This message and any files transmitted with it are legally privileged and intended for the sole use of the individual(s) or entity to whom they are addressed. If you are not the intended recipient, please notify the sender by reply and delete the message and any attachments from your system. Any unauthorised use or disclosure of the content of this message is strictly prohibited and may be unlawful. Nothing in this e-mail message amounts to a contractual or legal commitment on the part of EUROCONTROL, unless it is confirmed by appropriately signed hard copy. Any views expressed in this message are those of the sender. |
|
From: Bart V. A. <bva...@ac...> - 2011-04-01 10:14:40
|
On Fri, Apr 1, 2011 at 4:03 AM, Bob Harris <rsh...@gm...> wrote: > On Mar 31, 2011, at 7:55 PM, Tom Hughes wrote: >> It's entirely expected that it may fail under valgrind when it doesn't >> normally because the memory layout is different plus every byte of memory >> you allocate has a little over a byte (nine bits) of overhead in the form >> of the shadow memory valgrind is using to track it, and so on. >> >> You will run out of memory more quickly under valgrind that you do normally >> is the basic message. > > Sure, understandable. But to go from a program that is able to > allocate nearly 16G without valgrind to one that fails at 2.8G with > it? > > The test program has this line: > m26=malloc((size_t)12615754384); > 12.6G in a single block and that works fine without valgrind. With > valgrind, memory allocation starts to fail way earlier, and the 12G > attempt also fails. > > And in fact, if I comment out that huge 12.6G block and compile on a > 32bit machine, and run (without valgrind), I get allocation failures > starting at around 2.4G. This I expect on a 32-bit machine, from past > experience. All this, together with valgrind giving me a message that > seems to be telling me "my, grandmother, but that's a big block you > are trying to allocate", is what makes me suspicious that the valgrind > is doing something that is limiting me to the equivalent of a 32-bit > realm. Have you already tried to add --freelist-vol=0 to the memcheck command-line options ? See also http://www.valgrind.org/docs/manual/mc-manual.html for more information. Bart. |
|
From: Bob H. <rsh...@gm...> - 2011-08-25 16:47:59
|
Back in March I posted a problem (see a piece of the thread below), and I finally got back to that problem yesterday. I've figured out what was happening. As I'm about to describe, there was no problem with valgrind and there was no problem with my target program, only the combination, and then primarily due to a mistake a made in my build for valgrind. I'm not reporting this to point out any kind of problem with valgrind. I just posting this as a recap in case other users run into a similar problem. I'll try to keep this brief. In the target program, calls to malloc and realloc go through wrapper routines that make the allocation call, and abort on allocation failure. None of the calls from the rest of the program check for allocation failure. The target program has a large data structure which it grows by repeated calls to realloc. Without valgrind, the allocation doesn't fail. With valgrind, the memory footprint is increased, and the allocation does eventually fail. The footprint increase is caused, as I understand it, by valgrind keeping alive blocks that would normally be freed during realloc, so that it can check for accesses to those blocks. In any case, the only relevant fact is that the footprint increases for the grow-by-realloc usage model, and can increase drastically. The mistake that I made was that when I build for valgrind, I was also turning off my allocation wrapper routines. I'm not sure why I did that initially, perhaps because the wrappers also have an option to track allocation/deallocation and report on usage (sort of a built-in poor man's version of valgrind). What this means, though, is that since the wrapper is the only place that checks for allocation failure, the program that valgrind was checking was no longer checking for allocation failures. So, when reallocation failed, the target program kept happily running, and valgrind happily reported the rogue accesses that resulted. So all of this was me chasing my tail. But it did get me to look at my allocation pattern in more detail, and I did see that in some cases my memory footprint might be more than I thought it was (even in the absence of valgrind). I've made a few changes to the program to account for that. Thanks again to all the people who responded/helped. Bob H On Mar 31, 2011, at 7:55 PM, Tom Hughes wrote: It's entirely expected that it may fail under valgrind when it doesn't normally because the memory layout is different plus every byte of memory you allocate has a little over a byte (nine bits) of overhead in the form of the shadow memory valgrind is using to track it, and so on. You will run out of memory more quickly under valgrind that you do normally is the basic message. On Mar 31, 2011, at 10:03 PM, Bob Harris replied: Sure, understandable. But to go from a program that is able to allocate nearly 16G without valgrind to one that fails at 2.8G with it? The test program has this line: m26=malloc((size_t)12615754384); 12.6G in a single block and that works fine without valgrind. With valgrind, memory allocation starts to fail way earlier, and the 12G attempt also fails. And in fact, if I comment out that huge 12.6G block and compile on a 32bit machine, and run (without valgrind), I get allocation failures starting at around 2.4G. This I expect on a 32-bit machine, from past experience. All this, together with valgrind giving me a message that seems to be telling me "my, grandmother, but that's a big block you are trying to allocate", is what makes me suspicious that the valgrind is doing something that is limiting me to the equivalent of a 32-bit realm. Anyway, Tom, I do really appreciate your help on this. My guess is you probably feel like you are beating a dead horse at this point. Thanks again, Bob H |