Re: [Valgrind-developers] Custom modified Callgrind tool uses too much memory

[Philippe W. <phi...@sk...>]

On Fri, 2014-06-13 at 19:42 -0400, Siddharth Nilakantan wrote:
> Philippe,
> 
> 
> Thanks very much for that. I ran with what you said on my
> single-threaded benchmark and got a line like the following just
> before the tool died. 
> 
> 
> -------- Arena "tool": 31444697088/31444697088 max/curr mmap'd, 0/0
> unsplit/split sb unmmap'd, 19661485344/19661485344 max/curr on_loan
> --------
> 
> <Here there is a breakdown of the various mallocs in the code>
> 
> 
> This is a single-threaded benchmark, which creates just one "tool"
> arena. 
> This indicates that the arena's size is close to 29.3GB while the
> actual useable bytes (on_loan) are 18.3G. My calculation of malloc'ed
> bytes also yields 18.3G so that part is correct.
> Why is the arena size much larger? I currently do not free any memory,
> so is the data segment growing in discrete and huge chunks on certain
> malloc calls? 
Effectively, an arena is managed by mmap-ing big blocks of memory
(e.g. 4Mb) which are split in smaller blocks according to what is
VG_(malloc)-ed.

Such a big difference between the mmap-ed and on_loan can be caused
by 2 things:
* allocating a lot of very small blocks (each malloc-ed block has
  an overhead. So, very small blocks proportionally causes a lot of
  overhead.
* or by a 'unfriendly' pattern of malloc sizes.
  E.g. if the arena big block size is 4Mb, and you allocate a lot
  of 2Mb+1 "small blocks", that will cause a huge overhead.

The details given by --profile-heap=yes should give some details
about what is happening.

Philippe