|
From: Michael B A. <io...@gm...> - 2008-07-09 01:27:52
|
Hi, I have a leak in my Apache module but I can't tell where it is because there are no symbols in the backtraces. I've done this before successfully but I don't know why it's not working anymore and I would appreciate some tips. Specifically I'm doing: # valgrind -v --tool=memcheck --num-callers=20 --leak-check=yes --leak-resolution=high /usr/sbin/httpd -X Interesting parts of the output are: <snip> --15717-- Discarding syms at 0x1158E000-0x1199E000 in /usr/lib64/libplexcel.so.2.7.10 due to munmap() <snip> ==15717== 9 bytes in 9 blocks are definitely lost in loss record 8 of 60 ==15717== at 0x4A05809: malloc (vg_replace_malloc.c:149) ==15717== by 0x11627041: ??? ==15717== by 0x11628609: ??? ==15717== by 0x11628E7B: ??? ==15717== by 0x116292AE: ??? ==15717== by 0x115F419D: ??? ==15717== by 0x115E33D7: ??? ==15717== by 0x11387AB4: ??? ==15717== by 0xC863CD1: ??? ==15717== by 0xC853E1B: ??? ==15717== by 0xC863731: ??? ==15717== by 0xC853E1B: ??? ==15717== by 0xC863731: ??? ==15717== by 0xC853E1B: ??? ==15717== by 0xC836F3D: ??? ==15717== by 0xC7FBFB6: ??? ==15717== by 0xC8B4405: ??? ==15717== by 0x297F9: ap_run_handler (in /usr/sbin/httpd) ==15717== by 0x2CC71: ap_invoke_handler (in /usr/sbin/httpd) ==15717== by 0x375E7: ap_process_request (in /usr/sbin/httpd) Complete output is here: http://www.ioplex.com/~miallen/vg.txt I know the leak is coming from krb5_get_init_creds_password in libplexcel.so. That library has the desired symbols and backtraces earlier in the leak report are resolved from this library successfully. # objdump -Tt /usr/lib64/libplexcel.so.2.7 | egrep 'krb5_get_init_creds_pass' 000000000009b029 g F .text 00000000000002bc krb5_get_init_creds_password 000000000009b029 g DF .text 00000000000002bc Base krb5_get_init_creds_password >From looking at the leak report it seems the symbols are discarded before the backtraces are emitted. Is that the problem? I have to hit Ctrl-C to stop Apache so maybe that's triggering valgrind to prematurely discard the symbols? How can I track down this problem? I'm using valgrind-3.2.1-6.el5 on CentOS 5. Mike |
|
From: Howard C. <hy...@sy...> - 2008-07-09 01:51:22
|
Michael B Allen wrote: > Hi, > > I have a leak in my Apache module but I can't tell where it is because > there are no symbols in the backtraces. I've done this before > successfully but I don't know why it's not working anymore and I would > appreciate some tips. >> From looking at the leak report it seems the symbols are discarded > before the backtraces are emitted. Is that the problem? > > I have to hit Ctrl-C to stop Apache so maybe that's triggering > valgrind to prematurely discard the symbols? > > How can I track down this problem? > > I'm using valgrind-3.2.1-6.el5 on CentOS 5. Typically this happens because the main program has unloaded the shared objects containing the relevant symbols in its shutdown/cleanup processing. We do this in OpenLDAP slapd too, and I #if out the unload calls when I need to do this kind of tracking. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ |
|
From: Michael B A. <io...@gm...> - 2008-07-09 02:13:50
|
On 7/8/08, Howard Chu <hy...@sy...> wrote: > Michael B Allen wrote: > > > Hi, > > > > I have a leak in my Apache module but I can't tell where it is because > > there are no symbols in the backtraces. I've done this before > > successfully but I don't know why it's not working anymore and I would > > appreciate some tips. > > > > > > > > > From looking at the leak report it seems the symbols are discarded > > > > > before the backtraces are emitted. Is that the problem? > > > > I have to hit Ctrl-C to stop Apache so maybe that's triggering > > valgrind to prematurely discard the symbols? > > > > How can I track down this problem? > > > > I'm using valgrind-3.2.1-6.el5 on CentOS 5. > > > > Typically this happens because the main program has unloaded the shared > objects containing the relevant symbols in its shutdown/cleanup processing. > We do this in OpenLDAP slapd too, and I #if out the unload calls when I need > to do this kind of tracking. Is there any other workaround? Is it possible to somehow signal valgrind to generate the leak report at runtime as opposed to on shutdown? Otherwise I guess I'll have to create a custom build of the main program and comment out the dlclose() calls. That's annoying. Mike |
|
From: Howard C. <hy...@sy...> - 2008-07-09 02:50:08
|
Michael B Allen wrote: > On 7/8/08, Howard Chu<hy...@sy...> wrote: >> Typically this happens because the main program has unloaded the shared >> objects containing the relevant symbols in its shutdown/cleanup processing. >> We do this in OpenLDAP slapd too, and I #if out the unload calls when I need >> to do this kind of tracking. > > Is there any other workaround? > > Is it possible to somehow signal valgrind to generate the leak report > at runtime as opposed to on shutdown? > > Otherwise I guess I'll have to create a custom build of the main > program and comment out the dlclose() calls. That's annoying. You could always write a stub library with a no-op dlclose()... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ |
|
From: Michael B A. <io...@gm...> - 2008-07-09 03:18:41
|
On 7/8/08, Howard Chu <hy...@sy...> wrote: > Michael B Allen wrote: > > > On 7/8/08, Howard Chu<hy...@sy...> wrote: > > > > > Typically this happens because the main program has unloaded the shared > > > objects containing the relevant symbols in its shutdown/cleanup > processing. > > > We do this in OpenLDAP slapd too, and I #if out the unload calls when I > need > > > to do this kind of tracking. > > > > > > > Is there any other workaround? > > > > Is it possible to somehow signal valgrind to generate the leak report > > at runtime as opposed to on shutdown? > > > > Otherwise I guess I'll have to create a custom build of the main > > program and comment out the dlclose() calls. That's annoying. > > > > You could always write a stub library with a no-op dlclose()... I started to do that but then I started to worry that there could be some kind of chicken-and-egg issue using LD_PRELOAD to overload dlclose(). No matter. I compiled an vanilla Apache and now valgrind is seeing the symbols. I didn't even have to comment out dlclose() in apr/dso.c. I reconnoiter it's a timing thing. It would be slick if valgrind had an option to *not* discard symbols for a named library. There are a lot of DSOs being used these days. Thanks, Mike |
|
From: Bart V. A. <bar...@gm...> - 2008-07-10 05:46:02
|
On Wed, Jul 9, 2008 at 4:13 AM, Michael B Allen <io...@gm...> wrote: > Otherwise I guess I'll have to create a custom build of the main > program and comment out the dlclose() calls. That's annoying. An alternative to commenting out dlclose() calls in your application is to let Valgrind ignore these calls. E.g. the source file memcheck/mc_replace_strmem.c lets Memcheck intercept functions like strchr(), index() and rindex(). See also the --trace-redir=yes command line option (http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.wrapping.debugging). Bart. |
|
From: Tom H. <to...@co...> - 2008-07-09 07:03:45
|
Michael B Allen wrote: > It would be slick if valgrind had an option to *not* discard symbols > for a named library. There are a lot of DSOs being used these days. If it was that easy we'd have done it long since - the problem is that a second DSO might get mapped at the same address and then you get a second set of symbols which overlap. So you need to know which set of symbols applied at the point that the stack trace was recorded. Tom -- Tom Hughes (to...@co...) http://www.compton.nu/ |
|
From: Michael B A. <io...@gm...> - 2008-07-09 15:07:32
|
On 7/9/08, Tom Hughes <to...@co...> wrote: > Michael B Allen wrote: > > > > It would be slick if valgrind had an option to *not* discard symbols > > for a named library. There are a lot of DSOs being used these days. > > > > If it was that easy we'd have done it long since - the problem is that a > second DSO might get mapped at the same address and then you get a second > set of symbols which overlap. > > So you need to know which set of symbols applied at the point that the > stack trace was recorded. It seems to me there are a number of ways to improve this situation. For example you could "lazily discard" symbols. Meaning, don't dicard symbols on munmap, discard them immediately before the loader tries to re-mmap a new lib into the same address range. I think that would work 98% of the time. Mike |
|
From: Tom H. <to...@co...> - 2008-07-09 15:18:23
|
In message <78c...@ma...>
Michael B. Allen <io...@gm...> wrote:
> It seems to me there are a number of ways to improve this situation.
I'm sure there are.
> For example you could "lazily discard" symbols. Meaning, don't dicard
> symbols on munmap, discard them immediately before the loader tries to
> re-mmap a new lib into the same address range. I think that would work
> 98% of the time.
I think it would work very often for some programs and very rarely
for others, depending on their pattern of dynamic library usage. That
is a complete guess though.
More to the point is that pontification on a solution is easy - if
you really think you have a solution then code it up. Lots of people
will be very grateful.
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|