valgrind-users

[Valgrind-users] Valgrind on MIPS Reports Zero Heap Usage

[Woodrow B. <wjb...@nc...>]

Greetings,

I'm having some trouble using the Valgrind application on a MIPS embedded
device, and, having exhausted my own troubleshooting capacity, I'm hoping
someone might be able to help me. I'm using valgrind (v3.10.0) to hunt down
a memory leak in a complex application (a heavily modified build of
net-snmp) that is being built as part of a bigger software suite. I am sure
there is a leak (the memory footprint of the application grows linearly
without bound), but valgrind always reports the following upon termination.

    ==1139== HEAP SUMMARY:
    ==1139==     in use at exit: 0 bytes in 0 blocks
    ==1139==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
    ==1139==
    ==1139== All heap blocks were freed -- no leaks are possible

The total heap usage cannot be zero -- there are many, many calls to
`malloc` and `free` throughout the application. Valgrind is still capable
of finding "Invalid Write" errors.

The application in question is being compiled, along with other software
packages, with a uclibc-gcc toolchain for the MIPS processor (uclibc
v0.9.29) to be flashed onto an embedded device running a busybox (v1.17.2)
linux shell. I am running valgrind directly on the device. I use the
following options when launching Valgrind:

    --tool=memcheck --leak-check=full --undef-value-errors=no
--trace-children=yes

Basically, Valgrind doesn't detect any heap usage even though I've used the
heap. Why might this be? Are any of my assumptions (below) wrong?

-----

# What I've Tried

## Simple Test Program

I compiled the simple test program (using the same target and toolchain as
the application above) from the Valgrind [quick-start tutorial](
http://valgrind.org/docs/manual/quick-start.html), to see if Valgrind would
detect the leak. The final output was the same as above: no heap usage.

## Linking Issues?

Valgrind documentation has the following to say on [the FAQ](
http://valgrind.org/docs/manual/faq.html#faq.hiddenbug):

> If your program is statically linked, most Valgrind tools will only work
well if they are able to replace certain functions, such as malloc, with
their own versions. By default, statically linked malloc functions are not
replaced. A key indicator of this is if Memcheck says "All heap blocks were
freed -- no leaks are possible".

The above sounds exactly like my problem, so I checked to see that it's
dynamically linked to the C libraries that contained `malloc` and `free`. I
used the uclibc toolchain's custom `ldd` executable ([I can't use the
native linux `ldd`](http://www.uclibc.org/FAQ.html#ldd)) and the output
included the following lines:

    libc.so.0 => not found (0x00000000)
    /lib/ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x00000000)

(The reason they're not found is because I'm running this on the x86 host
device; the mips target device doesn't have an ldd executable.) Based on my
understanding, `malloc` and `free` will be in one of these libraries, and
they seem to be dynamically linked. I also did `readelf` and `nm` on the
executable to confirm that the references to `malloc` and `free` are
undefined (which is characteristic of a dynamically linked executable).

Additionally, I tried launching Valgrind with the
`--soname-synonyms=somalloc=NONE` option as suggested by the FAQ.

## LD_PRELOAD support?

It has been suggested that my toolchain doesn't support LD_PRELOAD. In
order to confirm that it does, I followed [this example](
http://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-tricks-using-ld_preload-to-cheat-inject-features-and-investigate-programs/)
to create a simple test library and load it (I replaced `rand()` with a
function that just returns 42). The test worked, so it would seem that my
target supports LD_PRELOAD just fine.

 - Woodrow Barlow

Re: [Valgrind-users] Valgrind on MIPS Reports Zero Heap Usage

[Philippe W. <phi...@sk...>]

On Wed, 2014-10-08 at 14:46 -0400, Woodrow Barlow wrote:
> Basically, Valgrind doesn't detect any heap usage even though I've
> used the heap. Why might this be? Are any of my assumptions (below)
> wrong?
The 3 known causes are:
A statically linked
B LD_PRELOAD not supported/not configured
C linked with a library whose soname does not match the expected name

The tests you have done exclude causes A and B.
To exclude C, can you check the soname of the library that provides
malloc ?
I.e. something like:
  readelf -d /lib/libc.so.6 | grep SONAME
giving
  0x0000000e (SONAME)                     Library soname: [libc.so.6]

The resulting soname should match the default expected soname pattern,
i.e. on a linux, it is expected to match  libc.so*
If the soname of your lib does not match the above, then you can use
    --soname-synonyms=somalloc=xxxxxxxx
where xxxxxxxx is something which matches your soname

Assuming the above is not the problem, then something else/unknown is
happening.

Can you then run a small test program dynamically linked, that does 
a malloc call
and use valgrind options
   -v -v -v -d -d -d --trace-malloc=yes --trace-redir=yes
and send the resulting log file?
If the file is big, you can compress it, and send it only to me.

Hoping this helps

Philippe
 

Re: [Valgrind-users] Valgrind on MIPS Reports Zero Heap Usage

[Woodrow B. <wjb...@nc...>]

Hello Sir,

Thank you for the help. I'm not entirely sure whether malloc is provided by
libc.so.0 or by ld-uClibc.so.0, so I've executed the requested command
against both. I'm afraid I don't know how to interpret the output, but I've
included it below:

    $ readelf -d lib/libc.so.0 | grep SONAME
     0x0000000e (SONAME)                     Library soname: [libc.so.0]
    $ readelf -d lib/ld-uClibc.so.0 | grep SONAME
     0x0000000e (SONAME)                     Library soname:
[ld-uClibc.so.0]

I've also done a readelf against the executable which I am attempting to
test. I am including below a few lines of that output which may be useful.

    Dynamic section
      Tag        Type                         Name/Value
     0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
     0x00000001 (NEEDED)                     Shared library: [libc.so.0]
     0x0000000f (RPATH)                      Library rpath: [//lib]

    Symbol table '.dynsym'
       Num:    Value  Size Type    Bind   Vis      Ndx Name
        27: 00404a40     0 FUNC    GLOBAL DEFAULT  UND free
        97: 00404690     0 FUNC    GLOBAL DEFAULT  UND malloc

Finally, as requested, I've run Valgrind with the debugging options against
a very simple application with malloc (I ran it against the memory leak
test program from the Valgrind FAQ). The output is attached.


 - Woodrow Barlow

On Thu, Oct 9, 2014 at 5:54 PM, Philippe Waroquiers <
phi...@sk...> wrote:

> On Wed, 2014-10-08 at 14:46 -0400, Woodrow Barlow wrote:
> > Basically, Valgrind doesn't detect any heap usage even though I've
> > used the heap. Why might this be? Are any of my assumptions (below)
> > wrong?
> The 3 known causes are:
> A statically linked
> B LD_PRELOAD not supported/not configured
> C linked with a library whose soname does not match the expected name
>
> The tests you have done exclude causes A and B.
> To exclude C, can you check the soname of the library that provides
> malloc ?
> I.e. something like:
>   readelf -d /lib/libc.so.6 | grep SONAME
> giving
>   0x0000000e (SONAME)                     Library soname: [libc.so.6]
>
> The resulting soname should match the default expected soname pattern,
> i.e. on a linux, it is expected to match  libc.so*
> If the soname of your lib does not match the above, then you can use
>     --soname-synonyms=somalloc=xxxxxxxx
> where xxxxxxxx is something which matches your soname
>
> Assuming the above is not the problem, then something else/unknown is
> happening.
>
> Can you then run a small test program dynamically linked, that does
> a malloc call
> and use valgrind options
>    -v -v -v -d -d -d --trace-malloc=yes --trace-redir=yes
> and send the resulting log file?
> If the file is big, you can compress it, and send it only to me.
>
> Hoping this helps
>
> Philippe
>
>
>

Re: [Valgrind-users] Valgrind on MIPS Reports Zero Heap Usage

[Philippe W. <phi...@sk...>]

On Fri, 2014-10-10 at 08:11 -0400, Woodrow Barlow wrote:
> Hello Sir,
> 
> 
> Thank you for the help. I'm not entirely sure whether malloc is provided by libc.so.0 or by ld-uClibc.so.0,
> so I've executed the requested command against both. I'm afraid I don't know how to interpret the output,
>  but I've included it below:
> 
> 
>     $ readelf -d lib/libc.so.0 | grep SONAME
> 
>      0x0000000e (SONAME)                     Library soname: [libc.so.0]
>     $ readelf -d lib/ld-uClibc.so.0 | grep SONAME
>      0x0000000e (SONAME)                     Library soname: [ld-uClibc.so.0]
It is unlikely that ld-uClibc.so.0 does provide malloc.
So, probably it will be libc.so.0 that will provide it.



> Finally, as requested, I've run Valgrind with the debugging options against a very simple application with malloc
It looks like the trace below has some lines truncated at column 90 and
joined with the next line(s).
E.g. some aspacemgr lines, or (more annoying :), the preload_string value.
A full not truncated trace would be better.

In any case, the trace shows that LD_PRELOAD is working, as we
see e.g. the line:
--1209-- Reading syms from /mnt/usb1_1/valgrind/lib/valgrind/vgpreload_memcheck-mips32-linux.so

The difference I see between a working run on a x86/linux and the trace below
is that no 'ACTIVE' redirection is being added.
On the x86/linux, we obtain some lines such as:
...
--2582--    ------ ACTIVE ------
...
--2582--     0x00286b40 (malloc              ) R-> (1001.0) 0x040078cc malloc

which means that the malloc function at 0x00286b40 (which the intercepted malloc in libc)
will be redirected to 0x040078cc, which is the 'intercepting malloc' in vgpreload_memcheck.

So, it looks like in your case, the valgrind files to preloaded are effectively preloaded,
that the redirection 'instructions' from these preloaded files are properly built,
giving the 'TOPSPEC' lines
but that they do not become active when the functions to intercepts are seen/found.

Looking at the code that makes such redirections active, this might be caused by
having a stripped library.
At least, in the x86 working case, I see:
--2582-- Reading syms from /lib/libc-2.11.2.so
while no such Reading Syms lines are visible in your trace for the libc.

Valgrind is supposed to detect that the dynamic linker was stripped,
and report a fatal error, and indicate to install the libc debug info.
but from what I can see, in the mips case, the warning is only given
for ld.so.3, while in your case, the loader soname is ld-uClibc.so.0,
as you do not use the "standard" linux loader.

So, at this state, the best guess is that you have a somewhat stripped
libraries and/or libraries without debug info, and/or syms for these
libs are not read properly
and that makes redirection not working.

There are ways to have more tracing, e.g. by doing

valgrind -v -v -v -d -d -d --debug-dump=syms --trace-symtab-patt=*libc* --trace-symtab=yes --trace-redir=yes --trace-malloc=yes

but I think this should be tried only after having verified that libc debug info
is available and/or that symbols are available in your lib.

Philippe

Re: [Valgrind-users] Valgrind on MIPS Reports Zero Heap Usage

[Woodrow B. <wjb...@nc...>]

Hello,

I've re-built my toolchain to include debugging info (I'm not sure if it
was there before). Hopefully that would guarantee that the libraries aren't
stripped? The end result is, unfortunately, unchanged.

Due to the complexity in implementing Valgrind on our systems, our
priorities have shifted. I'm no longer actively attempting to make the
application function, but will still happily reply with more information if
it is helpful for developers (although it has become apparent that this is
a problem with our own setup, and not with the valgrind product). If you
would still like the results of the detailed detailed tracing, I can send
it to you off-list (it's too long to post here).


 - Woodrow Barlow

On Fri, Oct 10, 2014 at 4:59 PM, Philippe Waroquiers <
phi...@sk...> wrote:

> On Fri, 2014-10-10 at 08:11 -0400, Woodrow Barlow wrote:
> > Hello Sir,
> >
> >
> > Thank you for the help. I'm not entirely sure whether malloc is provided
> by libc.so.0 or by ld-uClibc.so.0,
> > so I've executed the requested command against both. I'm afraid I don't
> know how to interpret the output,
> >  but I've included it below:
> >
> >
> >     $ readelf -d lib/libc.so.0 | grep SONAME
> >
> >      0x0000000e (SONAME)                     Library soname: [libc.so.0]
> >     $ readelf -d lib/ld-uClibc.so.0 | grep SONAME
> >      0x0000000e (SONAME)                     Library soname:
> [ld-uClibc.so.0]
> It is unlikely that ld-uClibc.so.0 does provide malloc.
> So, probably it will be libc.so.0 that will provide it.
>
>
>
> > Finally, as requested, I've run Valgrind with the debugging options
> against a very simple application with malloc
> It looks like the trace below has some lines truncated at column 90 and
> joined with the next line(s).
> E.g. some aspacemgr lines, or (more annoying :), the preload_string value.
> A full not truncated trace would be better.
>
> In any case, the trace shows that LD_PRELOAD is working, as we
> see e.g. the line:
> --1209-- Reading syms from
> /mnt/usb1_1/valgrind/lib/valgrind/vgpreload_memcheck-mips32-linux.so
>
> The difference I see between a working run on a x86/linux and the trace
> below
> is that no 'ACTIVE' redirection is being added.
> On the x86/linux, we obtain some lines such as:
> ...
> --2582--    ------ ACTIVE ------
> ...
> --2582--     0x00286b40 (malloc              ) R-> (1001.0) 0x040078cc
> malloc
>
> which means that the malloc function at 0x00286b40 (which the intercepted
> malloc in libc)
> will be redirected to 0x040078cc, which is the 'intercepting malloc' in
> vgpreload_memcheck.
>
> So, it looks like in your case, the valgrind files to preloaded are
> effectively preloaded,
> that the redirection 'instructions' from these preloaded files are
> properly built,
> giving the 'TOPSPEC' lines
> but that they do not become active when the functions to intercepts are
> seen/found.
>
> Looking at the code that makes such redirections active, this might be
> caused by
> having a stripped library.
> At least, in the x86 working case, I see:
> --2582-- Reading syms from /lib/libc-2.11.2.so
> while no such Reading Syms lines are visible in your trace for the libc.
>
> Valgrind is supposed to detect that the dynamic linker was stripped,
> and report a fatal error, and indicate to install the libc debug info.
> but from what I can see, in the mips case, the warning is only given
> for ld.so.3, while in your case, the loader soname is ld-uClibc.so.0,
> as you do not use the "standard" linux loader.
>
> So, at this state, the best guess is that you have a somewhat stripped
> libraries and/or libraries without debug info, and/or syms for these
> libs are not read properly
> and that makes redirection not working.
>
> There are ways to have more tracing, e.g. by doing
>
> valgrind -v -v -v -d -d -d --debug-dump=syms --trace-symtab-patt=*libc*
> --trace-symtab=yes --trace-redir=yes --trace-malloc=yes
>
> but I think this should be tried only after having verified that libc
> debug info
> is available and/or that symbols are available in your lib.
>
> Philippe
>
>
>
>