From: Bob H. <rsh...@gm...> - 2011-03-31 13:03:38
|
Howdy, Here's a followup on my problem. I think it is clearly a problem with valgrind in large memory conditions. Perhaps I can build valgrind in a way that allows for larger blocks? (if that is true, please point me in the right direction). Since my program has an alternative allocation-tracking technique built in, I wrote a converter that converts that allocation-tracker's output to a test program that does the same series of allocations. This test program is appended below my signature. It does nothing but allocate, reallocate, and free. I then built this on a linux-x86_64 machine with these gcc commands: cc -c -g -O0 -pg lz_memory_test.c -o lz_memory_test_valgrind.o cc lz_memory_test_valgrind.o -o lz_memory_test_valgrind Then I ran valgrind on it, like this: valgrind --leak-check=full lz_memory_test_valgrind valgrind version is 3.5.0 Output from valgrind is below the program. I get the same sort of "set address range perms" I get on my real program (please, if somebody could give me more info about those, it would be helpful). Then it tells me I have 2,736,652,288 bytes in use at exit. This would seem to relate to lines 84 and 85 of the program (which I have marked with trailing comments). Line 84 allocates exactly that many bytes, as part of a realloc of m14. Line 85 realloc's again, increasing the size. m14 is realloc'd several more times after that, then freed at line 184. So, it seems that either (a) there's a bug in realloc, and it did not dispose of the incoming m14 at line 85, or (b) valgrind has lost track. I also notice that the last "address range perms" complaint has a size that corresponds exactly to the allocation of m14 at line 83, i.e. the last allocation before the allocation that appears to be "lost" Thanks for any help, Bob H My test program, lz_memory_test.c, which (I claim) allocates, reallocates, and frees the same as my target program does. #include <stdlib.h> #include <stdio.h> int main (int argc, char** argv); int main (int argc, char** argv) { void* m1; void* m2; void* m3; void* m4; void* m5; void* m6; void* m7; void* m8; void* m9; void* m10; void* m11; void* m12; void* m13; void* m14; void* m15; void* m16; void* m17; void* m18; void* m19; void* m20; void* m21; void* m22; void* m23; void* m24; void* m25; void* m26; void* m27; void* m28; void* m29; void* m30; void* m31; void* m32; void* m33; void* m34; m1 = malloc((size_t)218); m2 = malloc((size_t)51); m3 = malloc((size_t)32); m4 = malloc((size_t)20); m5 = malloc((size_t)16); m6 = malloc((size_t)263984); m7 = malloc((size_t)184); m8 = malloc((size_t)263984); free(m4); m9 = malloc((size_t)632); m10 = malloc((size_t)1024); m11 = malloc((size_t)51); m12 = realloc(NULL,(size_t)16384); m13 = realloc(NULL,(size_t)16384); m14 = realloc(NULL,(size_t)16384); m15 = realloc(NULL,(size_t)5); m14 = realloc(m14,(size_t)249266176); m16 = realloc(NULL,(size_t)156); m17 = realloc(NULL,(size_t)156); m18 = realloc(NULL,(size_t)1300108); m19 = realloc(NULL,(size_t)1300108); m20 = realloc(NULL,(size_t)5); m14 = realloc(m14,(size_t)492453888); m14 = realloc(m14,(size_t)690487296); m14 = realloc(m14,(size_t)881639424); m14 = realloc(m14,(size_t)1062551552); m14 = realloc(m14,(size_t)1233666048); m14 = realloc(m14,(size_t)1392803840); m14 = realloc(m14,(size_t)1548075008); m14 = realloc(m14,(size_t)1694433280); m14 = realloc(m14,(size_t)1835646976); m16 = realloc(m16,(size_t)160); m17 = realloc(m17,(size_t)160); m15 = realloc(m15,(size_t)6); m14 = realloc(m14,(size_t)1971191808); m20 = realloc(m20,(size_t)6); m14 = realloc(m14,(size_t)2106195968); m14 = realloc(m14,(size_t)2240053248); m14 = realloc(m14,(size_t)2355216384); m14 = realloc(m14,(size_t)2462564352); m14 = realloc(m14,(size_t)2565095424); m14 = realloc(m14,(size_t)2655453184); /* line 83 */ m14 = realloc(m14,(size_t)2736652288); /* line 84 */ m14 = realloc(m14,(size_t)2814722048); /* line 85 */ m14 = realloc(m14,(size_t)2877751296); m14 = realloc(m14,(size_t)2937126912); m14 = realloc(m14,(size_t)2996256768); m14 = realloc(m14,(size_t)3047555072); m14 = realloc(m14,(size_t)3095691264); m15 = realloc(m15,(size_t)15); m14 = realloc(m14,(size_t)3100606464); m20 = realloc(m20,(size_t)15); m14 = realloc(m14,(size_t)3105439744); m14 = realloc(m14,(size_t)3110240256); m14 = realloc(m14,(size_t)3114926080); m14 = realloc(m14,(size_t)3119546368); m14 = realloc(m14,(size_t)3124166656); m14 = realloc(m14,(size_t)3128770560); m15 = realloc(m15,(size_t)16); m14 = realloc(m14,(size_t)3130458112); m20 = realloc(m20,(size_t)16); m14 = realloc(m14,(size_t)3131047936); m15 = realloc(m15,(size_t)21); m14 = realloc(m14,(size_t)3131588608); m20 = realloc(m20,(size_t)21); m14 = realloc(m14,(size_t)3131801600); m14 = realloc(m14,(size_t)3131998208); m14 = realloc(m14,(size_t)3132178432); m14 = realloc(m14,(size_t)3132375040); m14 = realloc(m14,(size_t)3132555264); m14 = realloc(m14,(size_t)3132735488); m14 = realloc(m14,(size_t)3132932096); m14 = realloc(m14,(size_t)3133112320); m14 = realloc(m14,(size_t)3133292544); m14 = realloc(m14,(size_t)3133472768); m15 = realloc(m15,(size_t)22); m14 = realloc(m14,(size_t)3133636608); m20 = realloc(m20,(size_t)22); m14 = realloc(m14,(size_t)3133816832); m14 = realloc(m14,(size_t)3133980672); m14 = realloc(m14,(size_t)3134160896); m14 = realloc(m14,(size_t)3134324736); m14 = realloc(m14,(size_t)3134488576); m14 = realloc(m14,(size_t)3134652416); m14 = realloc(m14,(size_t)3134816256); m14 = realloc(m14,(size_t)3134980096); m14 = realloc(m14,(size_t)3135143936); m14 = realloc(m14,(size_t)3135291392); m14 = realloc(m14,(size_t)3135438848); m14 = realloc(m14,(size_t)3135553536); m14 = realloc(m14,(size_t)3135684608); m14 = realloc(m14,(size_t)3135799296); m14 = realloc(m14,(size_t)3135881216); m14 = realloc(m14,(size_t)3135979520); m14 = realloc(m14,(size_t)3136061440); m14 = realloc(m14,(size_t)3136110592); m14 = realloc(m14,(size_t)3136159744); m14 = realloc(m14,(size_t)3136192512); m14 = realloc(m14,(size_t)3136241664); m14 = realloc(m14,(size_t)3136290816); m14 = realloc(m14,(size_t)3136323584); m14 = realloc(m14,(size_t)3136372736); m14 = realloc(m14,(size_t)3136405504); m14 = realloc(m14,(size_t)3136454656); m14 = realloc(m14,(size_t)3136487424); m14 = realloc(m14,(size_t)3136536576); m14 = realloc(m14,(size_t)3136569344); m14 = realloc(m14,(size_t)3136618496); m14 = realloc(m14,(size_t)3136651264); m14 = realloc(m14,(size_t)3136684032); m14 = realloc(m14,(size_t)3136733184); m14 = realloc(m14,(size_t)3136765952); m14 = realloc(m14,(size_t)3136798720); m14 = realloc(m14,(size_t)3136847872); m14 = realloc(m14,(size_t)3136880640); m14 = realloc(m14,(size_t)3136913408); m14 = realloc(m14,(size_t)3136946176); m14 = realloc(m14,(size_t)3136995328); m14 = realloc(m14,(size_t)3137028096); m14 = realloc(m14,(size_t)3137060864); m14 = realloc(m14,(size_t)3137093632); m14 = realloc(m14,(size_t)3137110016); m14 = realloc(m14,(size_t)3137126400); m14 = realloc(m14,(size_t)3137142784); m14 = realloc(m14,(size_t)3137159168); m14 = realloc(m14,(size_t)3137175552); m21 = malloc((size_t)632); m22 = malloc((size_t)1024); m23 = malloc((size_t)32); m24 = realloc(NULL,(size_t)16384); m25 = realloc(NULL,(size_t)16384); m26 = malloc((size_t)12615754384); m27 = malloc((size_t)160032); m28 = realloc(NULL,(size_t)16384); m29 = realloc(NULL,(size_t)30); m30 = realloc(NULL,(size_t)8); m30 = realloc(m30,(size_t)9); m31 = malloc((size_t)16777216); m32 = malloc((size_t)16777216); m33 = malloc((size_t)33554432); m34 = malloc((size_t)16777216); free(m2); free(m14); /* line 184 */ free(m10); free(m11); free(m15); free(m20); free(m16); free(m17); free(m18); free(m19); free(m12); free(m13); free(m9); free(m3); free(m28); free(m22); free(m23); free(m29); free(m30); free(m24); free(m25); free(m21); free(m1); free(m6); free(m8); free(m26); free(m27); free(m7); free(m5); free(m31); free(m32); free(m33); free(m34); return 0; } ==12544== Memcheck, a memory error detector ==12544== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al. ==12544== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info ==12544== Command: ./lz_memory_test_valgrind ==12544== ==12544== Warning: set address range perms: large range [0x13dde030, 0x31382050) (noaccess) ==12544== Warning: set address range perms: large range [0x3901d030, 0x6229d050) (noaccess) ==12544== Warning: set address range perms: large range [0x6229e030, 0x96b6a050) (noaccess) ==12544== Warning: set address range perms: large range [0x96b6b030, 0xd60bf050) (noaccess) ==12544== Warning: set address range perms: large range [0xd60c0030, 0x11f944050) (noaccess) ==12544== Warning: set address range perms: large range [0x11f945030, 0x17298d050) (noaccess) ==12544== Warning: set address range perms: large range [0x17298e030, 0x1cedea050) (noaccess) ==12544== Warning: set address range perms: large range [0x1cedeb030, 0x233ddb050) (noaccess) ==12544== Warning: set address range perms: large range [0x233ddc030, 0x2a1478050) (noaccess) ==12544== Warning: set address range perms: large range [0x2a1479030, 0x316c59050) (noaccess) ==12544== Warning: set address range perms: large range [0x316c5a030, 0x3944fa050) (noaccess) ==12544== Warning: set address range perms: large range [0x43dd9b030, 0x4c35e3050) (noaccess) ==12544== Warning: set address range perms: large range [0x4c766c030, 0x553c88050) (noaccess) ==12544== Warning: set address range perms: large range [0x557cad030, 0x5ea929050) (noaccess) ==12544== Warning: set address range perms: large range [0x5ee492030, 0x6872d6050) (noaccess) ==12544== Warning: set address range perms: large range [0x68a7e7030, 0x728c57050) (noaccess) ==12544== ==12544== HEAP SUMMARY: ==12544== in use at exit: 2,736,652,288 bytes in 1 blocks ==12544== total heap usage: 140 allocs, 64 frees, 33,075,445,557 bytes allocated ==12544== ==12544== 2,736,652,288 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==12544== at 0x4A05F1D: realloc (vg_replace_malloc.c:476) ==12544== by 0x40091E: main (lz_memory_test.c:84) ==12544== ==12544== LEAK SUMMARY: ==12544== definitely lost: 2,736,652,288 bytes in 1 blocks ==12544== indirectly lost: 0 bytes in 0 blocks ==12544== possibly lost: 0 bytes in 0 blocks ==12544== still reachable: 0 bytes in 0 blocks ==12544== suppressed: 0 bytes in 0 blocks ==12544== ==12544== For counts of detected and suppressed errors, rerun with: -v ==12544== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4) On Mar 28, 2011, at 4:10 PM, I wrote: > I'm trying to check for leaks in a program I've written, using "valgrind > --leak-check=full ...". When my program stays within relatively small memory > usage, that is, less than 2G total, valgrind has no complaints. However, one > of my test cases involves my program allocating a total of about 16G with > about 12.6G in a single block. In this case, valgrind gives me this warning, > repeatedly (maybe a dozen times): ==31171== Warning: set address range perms: large range [0x13dde030, 0x31382050) (noaccess) Then complains about a dozen times about "Invalid write of size 1" in memcpy > called by fread, as well as one other complaint, "Syscall param read(buf) > points to unaddressable byte(s)", also in something called by fread. After > this it tells me it has detected more than 10 million errors and tells me to > "Go fix [my] program!". Then after reporting a page or so of stats, I get a > segfault. > > The program doesn't segfault on this case when built without without > valgrind, and to the extent that I can tell, is giving me the correct result > (when valgrind isn't used). > > The machine I'm running on is a 64-bit linux machine with 32G of RAM. This > occurred with valgrind version 3.5.0 and also with version 3.6.1. > > I have attempted to find info on how to "set address range perms", but the > best I have been able to find so far is that this is a message intended for > valgrind developers, not users. Is this something I can change if I build > the valgrind source code? > > So... my question is, should I expect valgrind to work correctly in such an > environment? Or, equivalently, what are valgrind's memory limitations? > > Also worth noting, possibly, is that the single 12.6G block is the result of > several reallocs. Since the target program does not know a priori how large > this needs to be, it grows the block on an as-needed basis. I believe > valgrind is reporting a "set address range perms" message ever time realloc > runs out of space in the current block. The first report has > large range [0x13dde030, 0x31382050) > and the final one has > large range [0x68c353030, 0x72a7c3050) > But if that last one is indeed the memory allocated for my huge block, it is > only 2.7G. > > I'm trying to decide whether I truly have a problem in my program, or if the > problem that valgrind doesn't handle blocks of this size. |