|
From: Peter T. <pt...@li...> - 2012-01-20 15:47:40
|
Hi Valgrinders I work from time to time with code which spans ~500.000 lines of code. When running I use 500-1000 MB on e.g. Red Hat Linux. When prefixing my binary with valgrind I normally have good sucess, but occationally (~1% of all cases) I see the memory growing towards infinity. Typically I set a general memory limit of 3 GB, where I kill the binary . I reach that level ~1% of my runs. For the remaining 99% of my valgrind tests I have a quite limited memory overhead due to valgrind. Have any of you seen similar behavior? -- Peter Toft, PhD http://petertoft.dk |
|
From: WAROQUIERS P. <phi...@eu...> - 2012-01-20 16:31:30
|
There has been some improvements in the way some client malloc patterns are managed by Valgrind in 3.7.0. (see bug 250101). In SVN 3.8.0, there has been a recent commit to further improve the memory usage (see bug 282230). To have a better idea about what is happening, it would be good to at least run with -v -v --stats=yes and optionally with --profile-heap=yes (this somewhat increases the memory usage). This will very probably give some indications about what consumes that memory. You can then see the difference between a succesful run and a failing run and/or post the stats of the succesful/failing runs here. Philippe ________________________________ From: Peter Toft [mailto:pt...@li...] Sent: Friday 20 January 2012 16:13 To: Valgrind users Subject: [Valgrind-users] Extreme memory growth with Valgrind 3.6.0 and 3.7.0 Hi Valgrinders I work from time to time with code which spans ~500.000 lines of code. When running I use 500-1000 MB on e.g. Red Hat Linux. When prefixing my binary with valgrind I normally have good sucess, but occationally (~1% of all cases) I see the memory growing towards infinity. Typically I set a general memory limit of 3 GB, where I kill the binary . I reach that level ~1% of my runs. For the remaining 99% of my valgrind tests I have a quite limited memory overhead due to valgrind. Have any of you seen similar behavior? -- Peter Toft, PhD http://petertoft.dk |
|
From: Peter T. <pt...@li...> - 2012-01-22 11:42:40
|
Thanx Philippe I re-ran valgrind 3.6.0 with "-v -v --stats=yes --profile-heap=yes" and just before the memory crash I do see new output, but I cannot understand what I should see from this. --19677-- client : 23 sbs, 573568 bs, 3297/3297 free bs, 100470784 mmap, 53280112 loan-------- Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 --19677-- tool : 15 sbs, 745570 bs, 7/7 free bs, 62914560 mmap, 26228624 loan-------- Arena "tool": 62914560 mmap'd, 26228624/26228624 max/curr -------- 128 in 2: hashtable.Hc.1 128 in 2: commandline.sua.2 128 in 2: commandline.sua.3 256 in 1: initimg-linux.sce.5 528 in 2: mc.cSVT.1 (sec VBit table) 704 in 30: mc.resi.1 1,264 in 7: errormgr.mre.2 6,160 in 1: hashtable.Hc.2 8,368 in 166: mc.iaLL.1 2,359,456 in 2: hashtable.resize.1 23,851,504 in 745,348: mc.cMC.1 (a MC_Chunk) --19677-- client : 29 sbs, 733108 bs, 9946/9946 free bs, 131190784 mmap, 68070720 loan-------- Arena "client": 131190784 mmap'd, 68070720/68070720 max/curr -------- 68,070,720 in 723,162: replacemalloc.cm.1 --19677-- client : 30 sbs, 678741 bs, 7692/7692 free bs, 135385088 mmap, 76847008 loan-------- Arena "client": 135385088 mmap'd, 76847008/76847008 max/curr -------- 76,847,008 in 671,049: replacemalloc.cm.1 --19677-- client : 33 sbs, 679112 bs, 7710/7710 free bs, 176037888 mmap, 115553744 loan-------- Arena "client": 176037888 mmap'd, 115553744/115553744 max/curr -------- 115,553,744 in 671,402: replacemalloc.cm.1 --19677-- TT/TC: initialise sector 1--19677-- client : 35 sbs, 679573 bs, 7752/7752 free bs, 186687488 mmap, 128247232 loan-------- Arena "client": 186687488 mmap'd, 128247232/128247232 max/curr -------- 128,247,232 in 671,821: replacemalloc.cm.1 --19677-- REDIR: 0x31706796f0 (strcspn) redirected to 0x4908e00 (strcspn)--19677-- client : 38 sbs, 627674 bs, 18400/18400 free bs, 201531392 mmap, 141531824 loan-------- Arena "client": 201531392 mmap'd, 141531824/141531824 max/curr -------- 141,531,824 in 609,274: replacemalloc.cm.1 --19677-- client : 39 sbs, 626844 bs, 18316/18316 free bs, 216150016 mmap, 157200624 loan-------- Arena "client": 216150016 mmap'd, 157200624/157200624 max/curr -------- 157,200,624 in 608,528: replacemalloc.cm.1 --19677-- client : 43 sbs, 621501 bs, 17684/17684 free bs, 232927232 mmap, 173009392 loan-------- Arena "client": 232927232 mmap'd, 173009392/173009392 max/curr -------- 173,009,392 in 603,817: replacemalloc.cm.1 --19677-- client : 47 sbs, 610298 bs, 23535/23535 free bs, 250429440 mmap, 190310464 loan-------- Arena "client": 250429440 mmap'd, 190310464/190310464 max/curr -------- 190,310,464 in 586,763: replacemalloc.cm.1 --19677-- client : 50 sbs, 587330 bs, 34531/34531 free bs, 495476736 mmap, 437869472 loan-------- Arena "client": 495476736 mmap'd, 437869472/437869472 max/curr -------- 437,869,472 in 552,799: replacemalloc.cm.1 --19677-- client : 51 sbs, 587332 bs, 34531/34531 free bs, 613445632 mmap, 555834320 loan-------- Arena "client": 613445632 mmap'd, 555834320/555834320 max/curr -------- 555,834,320 in 552,801: replacemalloc.cm.1 --19677-- TT/TC: initialise sector 2--19677-- client : 63 sbs, 591866 bs, 69362/69362 free bs, 676491264 mmap, 611707216 loan-------- Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 Clues? Best Peter On Fri, 20 Jan 2012 17:30:38 +0100, WAROQUIERS Philippe wrote: > There has been some improvements in the way some client malloc patterns are > managed by Valgrind in 3.7.0. (see bug 250101). > > In SVN 3.8.0, there has been a recent commit to further improve the memory usage > (see bug 282230). > > To have a better idea about what is happening, it would be good to > at least run with -v -v --stats=yes > and optionally with --profile-heap=yes (this somewhat increases the memory usage). > This will very probably give some indications about what consumes that memory. > You can then see the difference between a succesful run and a failing run and/or > post the stats of the succesful/failing runs here. > > Philippe > >> ------------------------- >> FROM: Peter Toft [mailto:pt...@li...] >> SENT: Friday 20 January 2012 16:13 >> TO: Valgrind users >> SUBJECT: [Valgrind-users] Extreme memory growth with Valgrind 3.6.0 and 3.7.0 >> >> Hi Valgrinders >> >> I work from time to time with code which spans ~500.000 lines of code. >> >> When running I use 500-1000 MB on e.g. Red Hat Linux. >> >> When prefixing my binary with valgrind I normally have good sucess, but occationally (~1% of all cases) I see the memory growing towards infinity. >> >> Typically I set a general memory limit of 3 GB, where I kill the binary . I reach that level ~1% of my runs. >> >> For the remaining 99% of my valgrind tests I have a quite limited memory overhead due to valgrind. >> >> Have any of you seen similar behavior? >> >> -- >> Peter Toft, PhD >> http://petertoft.dk -- Peter Toft, PhD http://petertoft.dk |
|
From: Peter T. <pt...@li...> - 2012-01-23 08:45:39
|
More comments; Next test; valgrind runs fine if I add "--show-reachable=no --show-possibly-lost=no" Best /pto On Sun, 22 Jan 2012 12:42:25 +0100, Peter Toft wrote: > Thanx Philippe > > I re-ran valgrind 3.6.0 with "-v -v --stats=yes --profile-heap=yes" and just before the memory crash I do see new output, but I cannot understand what I should see from this. > > --19677-- client : 23 sbs, 573568 bs, 3297/3297 free bs, 100470784 mmap, 53280112 loan-------- Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 > --19677-- tool : 15 sbs, 745570 bs, 7/7 free bs, 62914560 mmap, 26228624 loan-------- Arena "tool": 62914560 mmap'd, 26228624/26228624 max/curr -------- 128 in 2: hashtable.Hc.1 128 in 2: commandline.sua.2 128 in 2: commandline.sua.3 256 in 1: initimg-linux.sce.5 528 in 2: mc.cSVT.1 (sec VBit table) 704 in 30: mc.resi.1 1,264 in 7: errormgr.mre.2 6,160 in 1: hashtable.Hc.2 8,368 in 166: mc.iaLL.1 2,359,456 in 2: hashtable.resize.1 23,851,504 in 745,348: mc.cMC.1 (a MC_Chunk) > --19677-- client : 29 sbs, 733108 bs, 9946/9946 free bs, 131190784 mmap, 68070720 loan-------- Arena "client": 131190784 mmap'd, 68070720/68070720 max/curr -------- 68,070,720 in 723,162: replacemalloc.cm.1 > --19677-- client : 30 sbs, 678741 bs, 7692/7692 free bs, 135385088 mmap, 76847008 loan-------- Arena "client": 135385088 mmap'd, 76847008/76847008 max/curr -------- 76,847,008 in 671,049: replacemalloc.cm.1 > --19677-- client : 33 sbs, 679112 bs, 7710/7710 free bs, 176037888 mmap, 115553744 loan-------- Arena "client": 176037888 mmap'd, 115553744/115553744 max/curr -------- 115,553,744 in 671,402: replacemalloc.cm.1 > --19677-- TT/TC: initialise sector 1--19677-- client : 35 sbs, 679573 bs, 7752/7752 free bs, 186687488 mmap, 128247232 loan-------- Arena "client": 186687488 mmap'd, 128247232/128247232 max/curr -------- 128,247,232 in 671,821: replacemalloc.cm.1 > --19677-- REDIR: 0x31706796f0 (strcspn) redirected to 0x4908e00 (strcspn)--19677-- client : 38 sbs, 627674 bs, 18400/18400 free bs, 201531392 mmap, 141531824 loan-------- Arena "client": 201531392 mmap'd, 141531824/141531824 max/curr -------- 141,531,824 in 609,274: replacemalloc.cm.1 > --19677-- client : 39 sbs, 626844 bs, 18316/18316 free bs, 216150016 mmap, 157200624 loan-------- Arena "client": 216150016 mmap'd, 157200624/157200624 max/curr -------- 157,200,624 in 608,528: replacemalloc.cm.1 > --19677-- client : 43 sbs, 621501 bs, 17684/17684 free bs, 232927232 mmap, 173009392 loan-------- Arena "client": 232927232 mmap'd, 173009392/173009392 max/curr -------- 173,009,392 in 603,817: replacemalloc.cm.1 > --19677-- client : 47 sbs, 610298 bs, 23535/23535 free bs, 250429440 mmap, 190310464 loan-------- Arena "client": 250429440 mmap'd, 190310464/190310464 max/curr -------- 190,310,464 in 586,763: replacemalloc.cm.1 > --19677-- client : 50 sbs, 587330 bs, 34531/34531 free bs, 495476736 mmap, 437869472 loan-------- Arena "client": 495476736 mmap'd, 437869472/437869472 max/curr -------- 437,869,472 in 552,799: replacemalloc.cm.1 > --19677-- client : 51 sbs, 587332 bs, 34531/34531 free bs, 613445632 mmap, 555834320 loan-------- Arena "client": 613445632 mmap'd, 555834320/555834320 max/curr -------- 555,834,320 in 552,801: replacemalloc.cm.1 > --19677-- TT/TC: initialise sector 2--19677-- client : 63 sbs, 591866 bs, 69362/69362 free bs, 676491264 mmap, 611707216 loan-------- Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 > > Clues? > > Best > > Peter > > On Fri, 20 Jan 2012 17:30:38 +0100, WAROQUIERS Philippe wrote: > >> There has been some improvements in the way some client malloc patterns are >> managed by Valgrind in 3.7.0. (see bug 250101). >> >> In SVN 3.8.0, there has been a recent commit to further improve the memory usage >> (see bug 282230). >> >> To have a better idea about what is happening, it would be good to >> at least run with -v -v --stats=yes >> and optionally with --profile-heap=yes (this somewhat increases the memory usage). >> This will very probably give some indications about what consumes that memory. >> You can then see the difference between a succesful run and a failing run and/or >> post the stats of the succesful/failing runs here. >> >> Philippe >> >>> ------------------------- >>> FROM: Peter Toft [mailto:pt...@li...] >>> SENT: Friday 20 January 2012 16:13 >>> TO: Valgrind users >>> SUBJECT: [Valgrind-users] Extreme memory growth with Valgrind 3.6.0 and 3.7.0 >>> >>> Hi Valgrinders >>> >>> I work from time to time with code which spans ~500.000 lines of code. >>> >>> When running I use 500-1000 MB on e.g. Red Hat Linux. >>> >>> When prefixing my binary with valgrind I normally have good sucess, but occationally (~1% of all cases) I see the memory growing towards infinity. >>> >>> Typically I set a general memory limit of 3 GB, where I kill the binary . I reach that level ~1% of my runs. >>> >>> For the remaining 99% of my valgrind tests I have a quite limited memory overhead due to valgrind. >>> >>> Have any of you seen similar behavior? >>> >>> -- >>> Peter Toft, PhD >>> http://petertoft.dk > > -- > Peter Toft, PhD > http://petertoft.dk -- Peter Toft, PhD http://petertoft.dk |
|
From: WAROQUIERS P. <phi...@eu...> - 2012-01-23 09:38:01
|
> Next test; valgrind runs fine if I add "--show-reachable=no --show-possibly-lost=no" Not clear why this changes the behaviour. To my knowledge, the above options have no effect on the memory/no effect on the "run-time" unless you are doing a leak search. Is there any error msg reported by memcheck during the run ? (use of uninitialized memory or similar) Philippe |
|
From: Peter T. <pt...@li...> - 2012-01-23 10:16:43
|
On Mon, 23 Jan 2012 10:37:38 +0100, WAROQUIERS Philippe wrote: >> Next test; valgrind runs fine if I add "--show-reachable=no > --show-possibly-lost=no" Not clear why this changes the behaviour. To my knowledge, the above options have no effect on the memory/no effect on the "run-time" unless you are doing a leak search. Is there any error msg reported by memcheck during the run ? (use of uninitialized memory or similar) Philippe There are no other errors seen, albeit I have a binary only lib, which I have to link in :/ Best -- Peter Toft, PhD http://petertoft.dk |
|
From: WAROQUIERS P. <phi...@eu...> - 2012-01-23 09:34:41
|
Valgrind is managing the memory in multiple arena. We see in the below some information about the client arena and tool arena. Details about an arena is shown each time the memory usage of this arena increases by 5%. (the client is the application running under valgrind, while the tool arena is in this case the memory needed by memcheck tool). There are other arenas (not shown in the extract below) for things such as debug info, stack traces, ... In the below, we see mostly the client arena growing. The client arena first stat shows about 100 Mb of mmap'd memory, for 570,271 blocks allocated by the client (i.e,. nr of malloc-ed blocks). We then see the client arena growing in memory to about 676 Mb, for 522,504 blocks allocated. (the difference between the mmap-d memory and the details about replacemalloc is the overhead for each malloc-ed block). Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 .... Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 So, it looks like the application is freeing small blocks and re-allocating bigger blocks. If the below trace is the moment at which Valgrind crashes out of memory, it is strange as the tool arena is about 63 Mb, and the last client state is 676 Mb. So, this is far to be 3 Gb (the other arenas are usually not very big). What is the last error msg produced by Valgrind at the moment of the crash ? You might get more information about what the client application is doing by running with 3.7.0 and using Valgrind gdbserver+gdb to insert breakpoints in your application at relevant places, and then use the leak_check monitor command (using the "delta leak search") to see which small blocks are released and which bigger blocks are allocated. Philippe ________________________________ From: Peter Toft [mailto:pt...@li...] Sent: Sunday 22 January 2012 12:42 To: WAROQUIERS Philippe Cc: Valgrind users Subject: RE: [Valgrind-users] Extreme memory growth with Valgrind 3.6.0 and 3.7.0 Thanx Philippe I re-ran valgrind 3.6.0 with "-v -v --stats=yes --profile-heap=yes" and just before the memory crash I do see new output, but I cannot understand what I should see from this. --19677-- client : 23 sbs, 573568 bs, 3297/3297 free bs, 100470784 mmap, 53280112 loan-------- Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 --19677-- tool : 15 sbs, 745570 bs, 7/7 free bs, 62914560 mmap, 26228624 loan-------- Arena "tool": 62914560 mmap'd, 26228624/26228624 max/curr -------- 128 in 2: hashtable.Hc.1 128 in 2: commandline.sua.2 128 in 2: commandline.sua.3 256 in 1: initimg-linux.sce.5 528 in 2: mc.cSVT.1 (sec VBit table) 704 in 30: mc.resi.1 1,264 in 7: errormgr.mre.2 6,160 in 1: hashtable.Hc.2 8,368 in 166: mc.iaLL.1 2,359,456 in 2: hashtable.resize.1 23,851,504 in 745,348: mc.cMC.1 (a MC_Chunk) --19677-- client : 29 sbs, 733108 bs, 9946/9946 free bs, 131190784 mmap, 68070720 loan-------- Arena "client": 131190784 mmap'd, 68070720/68070720 max/curr -------- 68,070,720 in 723,162: replacemalloc.cm.1 --19677-- client : 30 sbs, 678741 bs, 7692/7692 free bs, 135385088 mmap, 76847008 loan-------- Arena "client": 135385088 mmap'd, 76847008/76847008 max/curr -------- 76,847,008 in 671,049: replacemalloc.cm.1 --19677-- client : 33 sbs, 679112 bs, 7710/7710 free bs, 176037888 mmap, 115553744 loan-------- Arena "client": 176037888 mmap'd, 115553744/115553744 max/curr -------- 115,553,744 in 671,402: replacemalloc.cm.1 --19677-- TT/TC: initialise sector 1--19677-- client : 35 sbs, 679573 bs, 7752/7752 free bs, 186687488 mmap, 128247232 loan-------- Arena "client": 186687488 mmap'd, 128247232/128247232 max/curr -------- 128,247,232 in 671,821: replacemalloc.cm.1 --19677-- REDIR: 0x31706796f0 (strcspn) redirected to 0x4908e00 (strcspn)--19677-- client : 38 sbs, 627674 bs, 18400/18400 free bs, 201531392 mmap, 141531824 loan-------- Arena "client": 201531392 mmap'd, 141531824/141531824 max/curr -------- 141,531,824 in 609,274: replacemalloc.cm.1 --19677-- client : 39 sbs, 626844 bs, 18316/18316 free bs, 216150016 mmap, 157200624 loan-------- Arena "client": 216150016 mmap'd, 157200624/157200624 max/curr -------- 157,200,624 in 608,528: replacemalloc.cm.1 --19677-- client : 43 sbs, 621501 bs, 17684/17684 free bs, 232927232 mmap, 173009392 loan-------- Arena "client": 232927232 mmap'd, 173009392/173009392 max/curr -------- 173,009,392 in 603,817: replacemalloc.cm.1 --19677-- client : 47 sbs, 610298 bs, 23535/23535 free bs, 250429440 mmap, 190310464 loan-------- Arena "client": 250429440 mmap'd, 190310464/190310464 max/curr -------- 190,310,464 in 586,763: replacemalloc.cm.1 --19677-- client : 50 sbs, 587330 bs, 34531/34531 free bs, 495476736 mmap, 437869472 loan-------- Arena "client": 495476736 mmap'd, 437869472/437869472 max/curr -------- 437,869,472 in 552,799: replacemalloc.cm.1 --19677-- client : 51 sbs, 587332 bs, 34531/34531 free bs, 613445632 mmap, 555834320 loan-------- Arena "client": 613445632 mmap'd, 555834320/555834320 max/curr -------- 555,834,320 in 552,801: replacemalloc.cm.1 --19677-- TT/TC: initialise sector 2--19677-- client : 63 sbs, 591866 bs, 69362/69362 free bs, 676491264 mmap, 611707216 loan-------- Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 |
|
From: Julian S. <js...@ac...> - 2012-01-23 10:25:28
|
You need to figure out whether the extreme growth is due to the behaviour of your program, or a bug in Valgrind. You can investigate the space usage of your program in various ways (some of which PhilippeW has already described) and you can also use the tools --tool=massif and (advanced, but sometimes interesting) --tool=exp-dhat. If you program is multithreaded you might also want to consider if there are any threading errors that could account for the possibly inconsistent behaviour. J |
|
From: Peter T. <pt...@li...> - 2012-01-23 10:25:50
|
Unfortunally you have the last valgrind message below, just before the "crash". As I wrote in the previous email I have part of my setup as a binary only library, and I fear that this one is the one to blaim. I am very familiar with gdb, but I am a bit unsure what you mean by "Valgrind gdbserver+gdb". If I have a binary "foo" compiled with "-g -O0" or alike, what then? This will be quite painful, since I see the problem every 0.3% of the cases I have - at random... Best Peter On Mon, 23 Jan 2012 10:34:07 +0100, WAROQUIERS Philippe wrote: > Valgrind is managing the memory in multiple arena. > We see in the below some information about the client arena and tool arena. Details about an arena > is shown each time the memory usage of this arena increases by 5%. > (the client is the application running under valgrind, while the tool arena is in this case the memory > needed by memcheck tool). There are other arenas (not shown in the extract below) for things > such as debug info, stack traces, ... > > In the below, we see mostly the client arena growing. > The client arena first stat shows about 100 Mb of mmap'd memory, for 570,271 blocks allocated > by the client (i.e,. nr of malloc-ed blocks). > We then see the client arena growing in memory to about 676 Mb, for 522,504 blocks allocated. > (the difference between the mmap-d memory and the details about replacemalloc is the overhead > for each malloc-ed block). > > Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 > .... > Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 > > So, it looks like the application is freeing small blocks and re-allocating bigger blocks. > > If the below trace is the moment at which Valgrind crashes out of memory, it is strange as the tool arena > is about 63 Mb, and the last client state is 676 Mb. So, this is far to be 3 Gb (the other arenas are usually > not very big). > What is the last error msg produced by Valgrind at the moment of the crash ? > > You might get more information about what the client application is doing by running with 3.7.0 and using > Valgrind gdbserver+gdb to insert breakpoints in your application at relevant places, and then > use the leak_check monitor command (using the "delta leak search") to see which small blocks are released > and which bigger blocks are allocated. > > Philippe > >> ------------------------- >> FROM: Peter Toft [mailto:pt...@li...] >> SENT: Sunday 22 January 2012 12:42 >> TO: WAROQUIERS Philippe >> CC: Valgrind users >> SUBJECT: RE: [Valgrind-users] Extreme memory growth with Valgrind 3.6.0 and 3.7.0 >> >> Thanx Philippe >> >> I re-ran valgrind 3.6.0 with "-v -v --stats=yes --profile-heap=yes" and just before the memory crash I do see new output, but I cannot understand what I should see from this. >> >> --19677-- client : 23 sbs, 573568 bs, 3297/3297 free bs, 100470784 mmap, 53280112 loan-------- Arena "client": 100470784 mmap'd, 53280112/53280112 max/curr -------- 53,280,112 in 570,271: replacemalloc.cm.1 >> --19677-- tool : 15 sbs, 745570 bs, 7/7 free bs, 62914560 mmap, 26228624 loan-------- Arena "tool": 62914560 mmap'd, 26228624/26228624 max/curr -------- 128 in 2: hashtable.Hc.1 128 in 2: commandline.sua.2 128 in 2: commandline.sua.3 256 in 1: initimg-linux.sce.5 528 in 2: mc.cSVT.1 (sec VBit table) 704 in 30: mc.resi.1 1,264 in 7: errormgr.mre.2 6,160 in 1: hashtable.Hc.2 8,368 in 166: mc.iaLL.1 2,359,456 in 2: hashtable.resize.1 23,851,504 in 745,348: mc.cMC.1 (a MC_Chunk) >> --19677-- client : 29 sbs, 733108 bs, 9946/9946 free bs, 131190784 mmap, 68070720 loan-------- Arena "client": 131190784 mmap'd, 68070720/68070720 max/curr -------- 68,070,720 in 723,162: replacemalloc.cm.1 >> --19677-- client : 30 sbs, 678741 bs, 7692/7692 free bs, 135385088 mmap, 76847008 loan-------- Arena "client": 135385088 mmap'd, 76847008/76847008 max/curr -------- 76,847,008 in 671,049: replacemalloc.cm.1 >> --19677-- client : 33 sbs, 679112 bs, 7710/7710 free bs, 176037888 mmap, 115553744 loan-------- Arena "client": 176037888 mmap'd, 115553744/115553744 max/curr -------- 115,553,744 in 671,402: replacemalloc.cm.1 >> --19677-- TT/TC: initialise sector 1--19677-- client : 35 sbs, 679573 bs, 7752/7752 free bs, 186687488 mmap, 128247232 loan-------- Arena "client": 186687488 mmap'd, 128247232/128247232 max/curr -------- 128,247,232 in 671,821: replacemalloc.cm.1 >> --19677-- REDIR: 0x31706796f0 (strcspn) redirected to 0x4908e00 (strcspn)--19677-- client : 38 sbs, 627674 bs, 18400/18400 free bs, 201531392 mmap, 141531824 loan-------- Arena "client": 201531392 mmap'd, 141531824/141531824 max/curr -------- 141,531,824 in 609,274: replacemalloc.cm.1 >> --19677-- client : 39 sbs, 626844 bs, 18316/18316 free bs, 216150016 mmap, 157200624 loan-------- Arena "client": 216150016 mmap'd, 157200624/157200624 max/curr -------- 157,200,624 in 608,528: replacemalloc.cm.1 >> --19677-- client : 43 sbs, 621501 bs, 17684/17684 free bs, 232927232 mmap, 173009392 loan-------- Arena "client": 232927232 mmap'd, 173009392/173009392 max/curr -------- 173,009,392 in 603,817: replacemalloc.cm.1 >> --19677-- client : 47 sbs, 610298 bs, 23535/23535 free bs, 250429440 mmap, 190310464 loan-------- Arena "client": 250429440 mmap'd, 190310464/190310464 max/curr -------- 190,310,464 in 586,763: replacemalloc.cm.1 >> --19677-- client : 50 sbs, 587330 bs, 34531/34531 free bs, 495476736 mmap, 437869472 loan-------- Arena "client": 495476736 mmap'd, 437869472/437869472 max/curr -------- 437,869,472 in 552,799: replacemalloc.cm.1 >> --19677-- client : 51 sbs, 587332 bs, 34531/34531 free bs, 613445632 mmap, 555834320 loan-------- Arena "client": 613445632 mmap'd, 555834320/555834320 max/curr -------- 555,834,320 in 552,801: replacemalloc.cm.1 >> --19677-- TT/TC: initialise sector 2--19677-- client : 63 sbs, 591866 bs, 69362/69362 free bs, 676491264 mmap, 611707216 loan-------- Arena "client": 676491264 mmap'd, 611707216/611707216 max/curr -------- 611,707,216 in 522,504: replacemalloc.cm.1 -- Peter Toft, PhD http://petertoft.dk |
|
From: WAROQUIERS P. <phi...@eu...> - 2012-01-23 11:12:25
|
> Unfortunally you have the last valgrind message below, just before the
"crash".
If Valgrind cannot get more memory while the client is requesting some,
it will (silently) return NULL to the client.
If Valgrind cannot get more memory while it requests itself some memory
(e.g.
for the tool arena), it will exits with an "internal error" msg.
> I am very familiar with gdb, but I am a bit unsure what you mean by
"Valgrind gdbserver+gdb".
> If I have a binary "foo" compiled with "-g -O0" or alike, what then?
You launch valgrind like this:
valgrind --vgdb-error=0 foo
Then in another window:
gdb foo
... and you follow the instructions to connect gdb to the valgrind
gdbserver...
... typically, you give to gdb something like
target remote | ..../vgdb --pid=xxxxxx
... once gdb is connected, you can put breakpoints, then continue.
... when foo is stopped on a breakpoint, you can e.g. do
(gdb) monitor leak_check
... for more info, use 'monitor help' in gdb,
... or read the user manual sections
3.2 Debugging your program using Valgrind gdbserver and GDB
4.6 Memcheck Monitor Commands
> This will be quite painful, since I see the problem every 0.3% of the
cases I have - at random...
If you apply the patch below to a valgrind 3.7.0
coregrind/m_mallocfree.c file,
the Valgrind gdbserver will activate itself when the client runs out of
memory.
You can then investigate the state of the client memory using e.g.
monitor leak_check and its various parameters
(hoping the leak search does not need too much memory :).
Philippe
*** m_mallocfree.c.orig Wed Oct 26 23:24:45 2011
--- m_mallocfree.c Mon Jan 23 11:55:05 2012
***************
*** 42,47 ****
--- 42,48 ----
#include "pub_core_threadstate.h" // For VG_INVALID_THREADID
#include "pub_core_transtab.h"
#include "pub_core_tooliface.h"
+ #include "pub_core_gdbserver.h"
#include "valgrind.h"
//zz#include "memcheck/memcheck.h"
*************** Superblock* newSuperblock ( Arena* a, Si
*** 752,759 ****
else
sres = VG_(am_sbrk_anon_float_client)
( cszB, VKI_PROT_READ|VKI_PROT_WRITE|VKI_PROT_EXEC
);
! if (sr_isError(sres))
return 0;
sb = (Superblock*)(AddrH)sr_Res(sres);
// Mark this segment as containing client heap. The leak
// checker needs to be able to identify such segments so as not
--- 753,764 ----
else
sres = VG_(am_sbrk_anon_float_client)
( cszB, VKI_PROT_READ|VKI_PROT_WRITE|VKI_PROT_EXEC
);
! if (sr_isError(sres)) {
! VG_(umsg)("(action on client out of memory) vgdb me ... \n");
! VG_(gdbserver)(1); // no tid available here => tid 1
arbitrarily chosen.
! VG_(umsg)("Continuing ...\n");
return 0;
+ }
sb = (Superblock*)(AddrH)sr_Res(sres);
// Mark this segment as containing client heap. The leak
// checker needs to be able to identify such segments so as not
|
|
From: Peter T. <pt...@li...> - 2012-01-23 10:31:03
|
On Mon, 23 Jan 2012 11:20:20 +0100, Julian Seward wrote: > You need to figure out whether the extreme growth is due to the > behaviour of your program, or a bug in Valgrind. You can investigate > the space usage of your program in various ways (some of which > PhilippeW has already described) and you can also use the tools > --tool=massif and (advanced, but sometimes interesting) --tool=exp-dhat. > > If you program is multithreaded you might also want to consider > if there are any threading errors that could account for the > possibly inconsistent behaviour. > > J Thanx Julian. I have no problems in my program when NOT running with valgrind, but I fully understand that this is NOT a proof for not having a bug. I will surely follow the good hints from PhilippeW. I did try massif, but the program is terminated (3G limit) before I get any output. It seems to happen extremely fast. I do have 3 threads, so your idea about exp-dhat is the next thing I will try. Thanx -- Peter Toft, PhD http://petertoft.dk |
|
From: Julian S. <js...@ac...> - 2012-01-23 10:40:10
|
> I did try massif, but the program is terminated (3G limit) > before I get any output. It seems to happen extremely fast. Maybe try to find a smaller workload that allows Massif to complete successfully, so you can get some results from it. Also as Philippe says, if Memcheck indicates any errors (uninitialised value, or out of range / free-memory access) in your code then that is probably a serious problem. The first thing I would do before profiling is to make your code run Memcheck-clean. > I do have > 3 threads, so your idea about exp-dhat is the next thing I will try. Ok, but be aware that exp-dhat is another kind of space profiler; it won't tell you about threading errors. For that you need --tool=helgrind or --tool=drd (but read the documentation before using). J |