[Valgrind-users] reply： helgrind use more than 32G memory.

[王阳 <412...@qq...>]

Hi Philippe,


Thank you for your reply.
>Ok, the origin of the memory is related to the locks/nr of locks
>your application creates and/or locks/keeps locked.
>About 50Gb of memory is consumed for hg.ids.4.
What is the  hg.ids.4? Is it helgrind's memory using for analysing lock?
I know that a lock can not cost big memory, how many memory  does helgrind cost for analysing one lock?
It is unbelievable  that  helgrind will cost 50GB for 1,029,288 locks.




>From the above, I guess your application has thousands
>of locks, and that each thread has sometimes hundreds
>of locks held.Myprog have creates about 4096*64 locks, sometimes myprog will lock almost all the locks at one time, and release them later.


>There is no garbage collection of the lock set universe.
You mean the big memory allocated using for analysing locks can not be reused/recyled by  valgrind?




>At this point, I think there are 2 possible approaches:
>  * implement in helgrind a garbage collection of the univ_lsets
>    (unclear if this is implementable, and for sure not trivial)
>  * you change the limits of Valgrind max memory, to e.g. go
>    to 128G or 256G.
Both ways are so difficult for me, I am want to try the second way, but I need your your help.
Can you give me a patch for 3.10.1 to  change the limits of Valgrind max memory to 256G.Thanks.




> I have done it two times with different options series. The result is
> as follows:
> 1.--tool=helgrind --gen-suppressions=all --stats=yes
> --profile-heap=yes --track-lockorders=no --read-var-info=yes
> --trace-children=yes --error-limit=no --log-file=xxx.log myprog 
Ok, the origin of the memory is related to the locks/nr of locks
your application creates and/or locks/keeps locked.

About 50Gb of memory is consumed for hg.ids.4.
> 51,760,182,272 in   341,498: hg.ids.4
This is the 'universe of locksets' : ie. all the locksets that
helgrind has ever seen.
It looks like your application uses a lot of locks
and keeps a lot of locks in a status 'locked' shown by:
>           locks: 1,029,288 acquires, 915,546 releases
From the above, I guess your application has thousands
of locks, and that each thread has sometimes hundreds
of locks held.

Well, if that is the case, I think helgrind data structure
is not done for such a behaviour :(  
There is no garbage collection of the lock set universe.

At this point, I think there are 2 possible approaches:
  * implement in helgrind a garbage collection of the univ_lsets
    (unclear if this is implementable, and for sure not trivial)
  * you change the limits of Valgrind max memory, to e.g. go
    to 128G or 256G.
    You might refer for that to the revision r13278, which increased
    from 32G to 64G. That gives the various constants to multiply by 2
    or 4 to go to 128G or 256G.
    So, checkout the SVN version, then do
    svn diff -r13277:13278
    and follow that pattern to increase to 128 or 256G
    
Philippe