|
From: Graydon H. <gr...@po...> - 2006-01-26 03:00:42
|
Hi, I'm working on characterizing mozilla's memory use, in order to eventually make it use less. I am hoping to use valgrind for this. To do this I am hoping to add a small mozilla component to our debugging builds, such that when it runs under valgrind the component can make memcheck client requests to measure the reachable and leaking portions of the heap. I see that memcheck already has some client requests for this purpose, and I see that the manual suggests that the VALGRIND_DO_LEAK_CHECK request "could be used to incrementally check for leaks between arbitrary places in the program's execution". This comment fills me with hope. I wonder if I could ask some advice on using this request, and/or extending the request to slightly new behavior. What I want to do is employ a *pair* of requests, a "punch-in" and a "punch-out" request, such that the *differences* in both the reachable and leaked sets, between punch-in and punch-out, can be reported. The use case I envision is that "debugging users" can press a button in their "debugging browser" to "start recording", then execute some actions which should ideally free all the memory they allocate (say, open a tab to a particular page, browse around, and close the tab), and then press the button again to reveal how much memory leaked *and* how much new reachable memory was allocated in the heap (cycles, accidentally-held references, etc). For all such memory allocated in the "recording window" -- leaks and otherwise -- I will of course want to retrieve allocation call stacks. I believe that *most* of the machinery for this task is present already in memcheck. I wonder if anyone has any advice about how to "finish the job", and make this functionality real. I'm happy to do all the work myself, but I figure some valgrind old hands might be able to reply with a helpful suggestion or two which might save me a few weeks of dead-ends while tying it all together. Thanks, -graydon |
|
From: Oswald B. <os...@kd...> - 2006-01-26 06:26:37
|
On Wed, Jan 25, 2006 at 06:36:19PM -0800, Graydon Hoare wrote: > I'm working on characterizing mozilla's memory use, in order to > eventually make it use less. I am hoping to use valgrind for this. > ok, maybe i'm totally wrong, but i would not use valgrind for this. as you want to do allocation and leak tracking only, you don't need any instrumentation except wrapping the allocator functions (which, i guess, the nspr(?) does anyway) and doing a snapshot each time you need it (memcheck does the same, afaik). that would mean that integrating a *way* more lightweight "classical" memory debugger library would be more appropriate. depends on what else you want to track at the same time ... -- Hi! I'm a .signature virus! Copy me into your ~/.signature, please! -- Chaos, panic, and disorder - my work here is done. |
|
From: Ashley P. <as...@qu...> - 2006-01-26 15:26:21
|
On Thu, 2006-01-26 at 07:26 +0100, Oswald Buddenhagen wrote:
> On Wed, Jan 25, 2006 at 06:36:19PM -0800, Graydon Hoare wrote:
> > I'm working on characterizing mozilla's memory use, in order to
> > eventually make it use less. I am hoping to use valgrind for this.
> >
> ok, maybe i'm totally wrong, but i would not use valgrind for this.
> as you want to do allocation and leak tracking only, you don't need any
> instrumentation except wrapping the allocator functions (which, i guess,
> the nspr(?) does anyway) and doing a snapshot each time you need it
> (memcheck does the same, afaik). that would mean that integrating a
> *way* more lightweight "classical" memory debugger library would be more
> appropriate. depends on what else you want to track at the same time ...
Sorry if this is too off-topic for this list...
I've done exactly this to track memory leaks in programs, it's actually
quite easy once you know how.
There are hooks inside libc (man 3 malloc_hook) you can use to intercept
every malloc/calloc/memalign/free call from everywhere in the
application/library's it's using (you don't even need the source to do
this). These callbacks are given the size and base of the buffer but
also the return address for the function that called malloc() in the
first place.
I simply print these out to a file using this format:
Inside malloc:
printf("libc + %zi %p %p\n",size,result,caller);
Inside free:
printf("libc - %p %p\n",start,caller);
I've then got a piece of perl which parses this log file to find buffers
which were allocated but not freed and uses addr2line to convert these
back into source file and line numbers. I'll send on the perl and hooks
if you want them... It doesn't get you full stack traces but it's often
good enough, particularly if you know the source well anyway.
Ashley,
|
|
From: Howard C. <hy...@sy...> - 2006-01-26 15:42:14
|
Ashley Pittman wrote: > On Thu, 2006-01-26 at 07:26 +0100, Oswald Buddenhagen wrote: > >> On Wed, Jan 25, 2006 at 06:36:19PM -0800, Graydon Hoare wrote: >> >>> I'm working on characterizing mozilla's memory use, in order to >>> eventually make it use less. I am hoping to use valgrind for this. >>> >>> >> ok, maybe i'm totally wrong, but i would not use valgrind for this. >> as you want to do allocation and leak tracking only, you don't need any >> instrumentation except wrapping the allocator functions (which, i guess, >> the nspr(?) does anyway) and doing a snapshot each time you need it >> (memcheck does the same, afaik). that would mean that integrating a >> *way* more lightweight "classical" memory debugger library would be more >> appropriate. depends on what else you want to track at the same time ... >> > > Sorry if this is too off-topic for this list... > > I've done exactly this to track memory leaks in programs, it's actually > quite easy once you know how. > > There are hooks inside libc (man 3 malloc_hook) you can use to intercept > every malloc/calloc/memalign/free call from everywhere in the > application/library's it's using (you don't even need the source to do > this). These callbacks are given the size and base of the buffer but > also the return address for the function that called malloc() in the > first place. > Those hooks only exist for glibc. My FunctionCheck code shows how to intercept malloc/etc. for any libdl-based dynamic library platform (e.g. Solaris, Irix, Linux, etc...) http://www.highlandsun.com/hyc/#fncchk -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/ |
|
From: Graydon H. <gr...@po...> - 2006-01-26 19:30:20
|
Ashley Pittman wrote:
> On Thu, 2006-01-26 at 07:26 +0100, Oswald Buddenhagen wrote:
>> On Wed, Jan 25, 2006 at 06:36:19PM -0800, Graydon Hoare wrote:
>>> I'm working on characterizing mozilla's memory use, in order to
>>> eventually make it use less. I am hoping to use valgrind for this.
>>>
>> ok, maybe i'm totally wrong, but i would not use valgrind for this.
> ...
> I've done exactly this to track memory leaks in programs, it's actually
> quite easy once you know how.
I wish it were this easy, but alas, valgrind is a much more appropriate
context for the instrumentation we want. We already do what you're
suggesting (and much more), and it is insufficient to characterize the
problems. In particular:
- We need to differentiate between "failure to free because it's
reachable" and "failure to free because it's unreachable". That is,
we need instrumentation which scans the entire memory image looking
for pointers.
- The mozilla source is far too large for a single "point of
allocation" to be of any use. We need stack traces 20+ frames deep
at minimum.
- We also need (eventually) to characterize reachable *components*
within the pointer graph, in the sense that a conservative GC does.
Before you ask, yes, there is Boehm. We do actually have a build which
uses Boehm for straight leak detection, and it is somewhat useful. I am
pursuing the option of extending our use of it as well. But I am not
certain I'll be able to get it to intercept all allocation paths
(including in libraries and code not modified with GC_ALLOC), nor record
all call stacks (including those which pass through XPConnect), nor
easily delineate its instrumentation phases. So I am pursuing the task
here in parallel: at least with valgrind I know there will be few
limitations on what it can measure.
-graydon
|
|
From: Julian S. <js...@ac...> - 2006-02-14 16:06:15
|
Hi Graydon. Apologies for very slow reply. It sounds like a plausible idea. I guess you need to make friends with the leak checker, which pretty self-contained and is in memcheck/mac_leakcheck.c. I'm not clear how the leak checker works now (JeremyF added some cycle-detecting cleverness, but I don't know exactly how it works). J > I believe that *most* of the machinery for this task is present already > in memcheck. I wonder if anyone has any advice about how to "finish the > job", and make this functionality real. I'm happy to do all the work > myself, but I figure some valgrind old hands might be able to reply with > a helpful suggestion or two which might save me a few weeks of dead-ends > while tying it all together. |
|
From: Graydon H. <gr...@po...> - 2006-02-14 20:49:37
|
Julian Seward wrote: > Hi Graydon. Apologies for very slow reply. > > It sounds like a plausible idea. I guess you need to make friends with > the leak checker, which pretty self-contained and is in > memcheck/mac_leakcheck.c. I'm not clear how the leak checker works now > (JeremyF added some cycle-detecting cleverness, but I don't know > exactly how it works). Ok, I'll look there. I'm also playing with massif a bit (put a patch in the massif bz the other day) since it seems to produce interesting stuff too. I'm shying away from using client requests though. They're a little more invasive and inflexible than I want. I'm considering instead something like callgrind's --dump-before, --dump-after, --toggle-collect, or --zero-before flags. I wonder if it makes sense to try to generalize these flags across tools; it seems like the sort of thing which helps isolate badness-triggering input (for example, we'd like to get per-webpage-load "dump" granularity; leaks in memcheck's case, censuses in massif's case). What do you think? -graydon |