Menu
▾
▴
valgrind-users
|
From: 王阳 <412...@qq...> - 2015-06-10 11:18:13
|
hello， I noticed that Valgrind maybe use memory more than twice orignal memory that program need in your guide, Now I got a error from valgrind which hint me that I have use more 32G memory on 64-bit platform, My program orginally use 5G memory ,when I use helgrind to analyze my program, I found it really used more than 32G memory(use top -p pid to see)， My answer is that helgrind really need more than 6 times or even more memory than before ,is that true? or helgrind have a bug? By the way ,my program have used mmap function. |
|
From: Philippe W. <phi...@sk...> - 2015-06-10 19:42:27
|
On Wed, 2015-06-10 at 19:17 +0800, 王阳 wrote: > My answer is that helgrind really need more than 6 times or even more > memory than before ,is that true? > or helgrind have a bug? The memory needed by helgrind might depend on what your program does. Typically, it needs twice the original, but it might be a lot more. Which version of Valgrind are you using ? In the SVN version, some memory improvements have been done. The best is to get the latest SVN version, compile it, then run with --stats=yes. That will give some indications where the memory is taken. More detailed information can also be obtained by using --profile-heap=yes Philippe |
|
From: 王阳 <412...@qq...> - 2015-06-11 01:20:51
|
Hi. Thank you for your reply. Verison is 3.6.0,very old one,I will try version 3.10.1 and svn version,and also try two options you said. ------------------ 原始邮件 ------------------ 发件人: "Philippe Waroquiers";<phi...@sk...>; 发送时间: 2015年6月11日(星期四) 凌晨3:42 收件人: "王阳"<412...@qq...>; 抄送: "valgrind-users"<val...@li...>; 主题: Re: [Valgrind-users] helgrind use more than 32G memory. On Wed, 2015-06-10 at 19:17 +0800, 王阳 wrote: > My answer is that helgrind really need more than 6 times or even more > memory than before ,is that true? > or helgrind have a bug? The memory needed by helgrind might depend on what your program does. Typically, it needs twice the original, but it might be a lot more. Which version of Valgrind are you using ? In the SVN version, some memory improvements have been done. The best is to get the latest SVN version, compile it, then run with --stats=yes. That will give some indications where the memory is taken. More detailed information can also be obtained by using --profile-heap=yes Philippe |
|
From: 王阳 <412...@qq...> - 2015-06-11 10:19:43
|
Hi. I download the latest version 3.10.1,but helgrind use memory over 64G and got a error from valgrind because of over the momory limit. I used the option --stats=yes, I noticed that valgrind print message below endless util to use 64G memory. --28682-- univ_laog_do_GC enter cardinality 9614 --28682-- univ_laog_do_GC exit seen 6591 next gc at cardinality 9615 why? DRD have not that problem ,but DRD 's message is not accurate comparing with helgrind. ------------------ 原始邮件 ------------------ 发件人: "我自己的邮箱";<412...@qq...>; 发送时间: 2015年6月11日(星期四) 上午9:20 收件人: "Philippe Waroquiers"<phi...@sk...>; 抄送: "valgrind-users"<val...@li...>; 主题: [Valgrind-users] 回复： helgrind use more than 32G memory. Hi. Thank you for your reply. Verison is 3.6.0,very old one,I will try version 3.10.1 and svn version,and also try two options you said. ------------------ 原始邮件 ------------------ 发件人: "Philippe Waroquiers";<phi...@sk...>; 发送时间: 2015年6月11日(星期四) 凌晨3:42 收件人: "王阳"<412...@qq...>; 抄送: "valgrind-users"<val...@li...>; 主题: Re: [Valgrind-users] helgrind use more than 32G memory. On Wed, 2015-06-10 at 19:17 +0800, 王阳 wrote: > My answer is that helgrind really need more than 6 times or even more > memory than before ,is that true? > or helgrind have a bug? The memory needed by helgrind might depend on what your program does. Typically, it needs twice the original, but it might be a lot more. Which version of Valgrind are you using ? In the SVN version, some memory improvements have been done. The best is to get the latest SVN version, compile it, then run with --stats=yes. That will give some indications where the memory is taken. More detailed information can also be obtained by using --profile-heap=yes Philippe |
|
From: Philippe W. <phi...@sk...> - 2015-06-11 18:45:48
|
On Thu, 2015-06-11 at 18:19 +0800, 王阳 wrote:
> --28682-- univ_laog_do_GC enter cardinality 9614
> --28682-- univ_laog_do_GC exit seen 6591 next gc at cardinality 9615
yes, one of the things that --stats=yes activates is some information
about laog GC.
>
> why?
> DRD have not that problem ,but DRD 's message is not accurate
> comparing with helgrind.
helgrind data structures are different of drd (a.o. to be able
to give precise information about race conditios).
Normally, --stats=yes should have produced statistics
at the end of your program (if your program exited due to
out of memory). If valgrind itself encountered the oom situation
then it should equally have produced statistics.
Can you post here these statistics ?
If they are not produced (unclear why), then you could instead
regularly run in another window
vgdb -c v.info stats -c v.info memory
to capture stats/memory during the run, before it reaches 64G.
Philippe
|
|
From: 王阳 <412...@qq...> - 2015-06-12 07:04:03
|
Hi Philippe.
Thank you for your reply.
>Can you post here these statistics ?
yes，I can，but I will do it tomorrow.Doing one time will cost over 24hours. It sounds unbelievable.
But it is true.
When I use helgrind to analyse myprog, it runs so slowly (more than 1000 times slowly,user guide say it will be 20~50times slowly). So I want to know why it runs so slowly. I use memcheck to analyse myprog, and it run 20~50times slowly,but why helgrind can not do it, and it cost so many memory.
I print memcheck log of myprog below,I think whether helgrind can not deal with these operation of alloc big memory.
------------------------------
==63283== Warning: set address range perms:large range[0x3a04c000,0x7a04f000)(defined)
==63283== Warning: set address range perms:large range[0x7a04f040,0xba052040)(defined)
==63283== Warning: set address range perms:large range[0xd4674000,0x114677000)(defined)
...
==63283== Warning: set address range perms:large range[0x288ba7000,0x298baa000)(defined)
------------------------------
I use picture as follows.
------------------ 原始邮件 ------------------
发件人: "Philippe Waroquiers";<phi...@sk...>;
发送时间: 2015年6月12日(星期五) 凌晨2:45
收件人: "王阳"<412...@qq...>;
抄送: "valgrind-users"<val...@li...>;
主题: Re: 回复：[Valgrind-users] 回复： helgrind use more than 32G memory.
On Thu, 2015-06-11 at 18:19 +0800, 王阳 wrote:
> --28682-- univ_laog_do_GC enter cardinality 9614
> --28682-- univ_laog_do_GC exit seen 6591 next gc at cardinality 9615
yes, one of the things that --stats=yes activates is some information
about laog GC.
>
> why?
> DRD have not that problem ,but DRD 's message is not accurate
> comparing with helgrind.
helgrind data structures are different of drd (a.o. to be able
to give precise information about race conditios).
Normally, --stats=yes should have produced statistics
at the end of your program (if your program exited due to
out of memory). If valgrind itself encountered the oom situation
then it should equally have produced statistics.
Can you post here these statistics ?
If they are not produced (unclear why), then you could instead
regularly run in another window
vgdb -c v.info stats -c v.info memory
to capture stats/memory during the run, before it reaches 64G.
Philippe |
|
From: 王阳 <412...@qq...> - 2015-06-15 11:23:32
|
HI Philippe,
I read valgrind user manual, there are some hints which are related to my problem I guess. As follows,
Myprog uses tons of mmap and memory pool ,and do not use free/delete to give back memory to pool.
My question is that If mypog use memoy pool without free/delete and don't use VALGRIND_HG_CLEAN_MEMORY will lead to myprog do mmap endlessly until use 64G memory?
--------------------------------
Avoid memory recycling. If you can’t avoid it, you must use tell Helgrind what is going on via the
VALGRIND_HG_CLEAN_MEMORY client request (in helgrind.h).
Helgrind is aware of standard heap memory allocation and deallocation that occurs via malloc/free/new/delete
and from entry and exit of stack frames. In particular, when memory is deallocated via free, delete, or
function exit, Helgrind considers that memory clean, so when it is eventually reallocated, its history is irrelevant.
However, it is common practice to implement memory recycling schemes. In these, memory to be freed is not
handed to free/delete, but instead put into a pool of free buffers to be handed out again as required. The
problem is that Helgrind has no way to know that such memory is logically no longer in use, and its history is
irrelevant. Hence you must make that explicit, using the VALGRIND_HG_CLEAN_MEMORY client request to
specify the relevant address ranges. It’s easiest to put these requests into the pool manager code, and use them either when memory is returned to the pool, or is allocated from it.
---------------------------
> --28682-- univ_laog_do_GC enter cardinality 9614
> --28682-- univ_laog_do_GC exit seen 6591 next gc at cardinality 9615
yes, one of the things that --stats=yes activates is some information
about laog GC.
>
> why?
> DRD have not that problem ,but DRD 's message is not accurate
> comparing with helgrind.
helgrind data structures are different of drd (a.o. to be able
to give precise information about race conditios).
Normally, --stats=yes should have produced statistics
at the end of your program (if your program exited due to
out of memory). If valgrind itself encountered the oom situation
then it should equally have produced statistics.
Can you post here these statistics ?
If they are not produced (unclear why), then you could instead
regularly run in another window
vgdb -c v.info stats -c v.info memory
to capture stats/memory during the run, before it reaches 64G.
Philippe |
|
From: Philippe W. <phi...@sk...> - 2015-06-15 20:14:31
|
On Mon, 2015-06-15 at 19:23 +0800, 王阳 wrote: > HI Philippe, > I read valgrind user manual, there are some hints which are related > to my problem I guess. As follows, > Myprog uses tons of mmap and memory pool ,and do not use free/delete > to give back memory to pool. > My question is that If mypog use memoy pool without free/delete and > don't use VALGRIND_HG_CLEAN_MEMORY will lead to myprog do mmap > endlessly until use 64G memory? No, HG_CLEAN_MEMORY is not useful to use less memory. It is only useful if you recycle memory, and you get false positive race errors due to this recycling. The best way to understand where valgrind/helgrind spends memory is to use --stats=yes and post the result here. Really, it is really the best way :). If it takes too long to reach the end (or it crashes before producing the stats), you can from the command line use vgdb v.info stats to get the needed info while helgrind is running. Philippe |
|
From: 王阳 <412...@qq...> - 2015-06-16 03:06:05
|
HI Philippe, 》memory is to use --stats=yes and post the result here. >Really, it is really the best way :). I can not export log from my PC, because PC is isolated from internet. Can I take a photo of log to you or Can pick up some key information from log to you. By the way, the size of email is limited to 40KB, am I right? so posting a photo maybe not work out. > HI Philippe, > I read valgrind user manual, there are some hints which are related > to my problem I guess. As follows, > Myprog uses tons of mmap and memory pool ,and do not use free/delete > to give back memory to pool. > My question is that If mypog use memoy pool without free/delete and > don't use VALGRIND_HG_CLEAN_MEMORY will lead to myprog do mmap > endlessly until use 64G memory? No, HG_CLEAN_MEMORY is not useful to use less memory. It is only useful if you recycle memory, and you get false positive race errors due to this recycling. The best way to understand where valgrind/helgrind spends memory is to use --stats=yes and post the result here. Really, it is really the best way :). If it takes too long to reach the end (or it crashes before producing the stats), you can from the command line use vgdb v.info stats to get the needed info while helgrind is running. Philippe |
Thanks for helping keep SourceForge clean.
X