valgrind-users

[Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Andrejs B. <sin...@gm...>]

Hi there! I'm quite new to Linux and was trying to use Valgrind for
finding leaks on 32-bit Linux ARMv7 (3.6.0, armv7l).
I crosscompiled Valgrind using non-stripped version of uClibc library.
I tried Valgrind version 3.9.0 and then newest from SVN (13985).

ldd's output for valgrind:
/# ldd /usr/bin/valgrind
ldd: can't open cache '/etc/ld.so.cache'
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb6f27000)
        libc.so.0 => /lib/libc.so.0 (0xb6ed2000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0xb6f3a000)


I tried detecting obvious leak in an example:
/* leak.c */
#include <stdlib.h>

int main()
{
	int *x = malloc(10 * sizeof(int));
	return 0;
}

On Linux/x86-64 leak was detected.
On Linux/ARM using --trace-redir=yes option I've found that
vgpreload_core-arm-linux.so and vgpreload_memcheck-arm-linux.so files
are not being read.
Using strace on Valgrind I found that vgpreload_memcheck-arm-linux.so
is accessed, but not used
(access("/usr/lib/valgrind/vgpreload_memcheck-arm-linux.so", R_OK) = 0).


I've checked ld-uClibc-0.9.33.2.so, libuClibc-0.9.33.2.so and
Valgrind's *.so files using 'file' and 'readelf'. They are not
stripped.
Also, have checked libuClibc-0.9.33.2.so for 'malloc' with 'readelf -a
libuClibc-0.9.33.2.so | grep malloc'. It's there.
Have checked sonames. They are same as shown in Valgrind's log.


Full Valgrind log on Linux/ARM:
/# valgrind --trace-redir=yes ./leak
==3526== Memcheck, a memory error detector
==3526== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3526== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info
==3526== Command: ./leak
==3526==
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_initialise) ------
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
--3526-- Reading syms from /transit/misc/leak
--3526--    svma 0x0000008338, avma 0x0000008338
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--3526--    TOPSPECS of soname NONE filename /transit/misc/leak
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
--3526-- Reading syms from /lib/ld-uClibc-0.9.33.2.so
--3526--    svma 0x0000000dc0, avma 0x0004000dc0
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--3526--    TOPSPECS of soname ld-uClibc.so.0 filename
/lib/ld-uClibc-0.9.33.2.so
--3526--    TOPSPECS of soname NONE filename /transit/misc/leak
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
--3526-- Reading syms from /usr/lib/valgrind/memcheck-arm-linux
--3526--    svma 0x00380000c0, avma 0x00380000c0
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--3526--    TOPSPECS of soname NONE filename
/usr/lib/valgrind/memcheck-arm-linux
--3526--    TOPSPECS of soname ld-uClibc.so.0 filename
/lib/ld-uClibc-0.9.33.2.so
--3526--    TOPSPECS of soname NONE filename /transit/misc/leak
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
--3526-- Reading syms from /lib/libgcc_s.so.1
--3526--    svma 0x0000003c58, avma 0x0004812c58
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--3526--    TOPSPECS of soname libgcc_s.so.1 filename /lib/libgcc_s.so.1
--3526--    TOPSPECS of soname NONE filename
/usr/lib/valgrind/memcheck-arm-linux
--3526--    TOPSPECS of soname ld-uClibc.so.0 filename
/lib/ld-uClibc-0.9.33.2.so
--3526--    TOPSPECS of soname NONE filename /transit/misc/leak
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
--3526-- Reading syms from /lib/libuClibc-0.9.33.2.so
--3526--    svma 0x000000a7b0, avma 0x000482c7b0
--3526-- <<
--3526--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--3526--    TOPSPECS of soname libc.so.0 filename /lib/libuClibc-0.9.33.2.so
--3526--    TOPSPECS of soname libgcc_s.so.1 filename /lib/libgcc_s.so.1
--3526--    TOPSPECS of soname NONE filename
/usr/lib/valgrind/memcheck-arm-linux
--3526--    TOPSPECS of soname ld-uClibc.so.0 filename
/lib/ld-uClibc-0.9.33.2.so
--3526--    TOPSPECS of soname NONE filename /transit/misc/leak
--3526--    TOPSPECS of soname (hardwired)
--3526--      ld-linux-armhf.so.3       memcpy             R->
(0000.0) 0x38075c50
--3526--      ld-linux.so.3             memcpy                 R->
(0000.0) 0x38075c50
--3526--      ld-linux-armhf.so.3       strlen                  R->
(0000.0) 0x38075c24
--3526--      ld-linux.so.3             strlen
R-> (0000.0) 0x38075c24
--3526--    ------ ACTIVE ------
--3526-- >>
==3526== Invalid read of size 4
==3526==    at 0x4005550: _dl_get_ready_to_run (in /lib/ld-uClibc-0.9.33.2.so)
==3526==  Address 0xbde78a9c is just below the stack ptr.  To
suppress, use: --workaround-gcc296-bugs=yes
==3526==
==3526== Invalid read of size 4
==3526==    at 0x4864C34: __uClibc_main (in /lib/libuClibc-0.9.33.2.so)
==3526==  Address 0xbde78c6c is just below the stack ptr.  To
suppress, use: --workaround-gcc296-bugs=yes
==3526==
==3526== Invalid read of size 4
==3526==    at 0x4864A30: __uClibc_fini (in /lib/libuClibc-0.9.33.2.so)
==3526==  Address 0xbde78c4c is just below the stack ptr.  To
suppress, use: --workaround-gcc296-bugs=yes
==3526==
==3526== Invalid read of size 4
==3526==    at 0x4000E1C: ??? (in /lib/ld-uClibc-0.9.33.2.so)
==3526==  Address 0xbde78c34 is just below the stack ptr.  To
suppress, use: --workaround-gcc296-bugs=yes
==3526==
==3526==
==3526== HEAP SUMMARY:
==3526==     in use at exit: 0 bytes in 0 blocks
==3526==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==3526==
==3526== All heap blocks were freed -- no leaks are possible
==3526==
==3526== For counts of detected and suppressed errors, rerun with: -v
==3526== ERROR SUMMARY: 32 errors from 4 contexts (suppressed: 0 from 0)


On Linux/x86-64 using --trace-redir=yes I've found, that functions
from libc.so.0 are being redirected after
vgpreload_memcheck-amd64-linux.so is being read.
Excerpt from log:
>>
--6671-- Reading syms from /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so
--6671--    svma 0x0000002f10, avma 0x0004c28f10
--6671-- <<
--6671--    ------ REDIR STATE after VG_(redir_notify_new_DebugInfo) ------
--6671--    TOPSPECS of soname NONE filename
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so
--6671--      libc.so*                  setenv                W->
(0000.0) 0x04c2f8a0
--6671--      libc.so*                  unsetenv            W->
(0000.0) 0x04c2f800
--6671--      libc.so*                  putenv                W->
(0000.0) 0x04c2f760
--6671--      libc.so*                  wcslen                R->
(2037.0) 0x04c2f740
--6671--      libc.so*                  strcasestr           R->
(2035.0) 0x04c2f680
--6671--      libc.so*                  strspn                 R->
(2034.0) 0x04c2f600


Also hardwired sonames are redirected on x86-64, but not on ARM.
But that's maybe cause soname 'ld-uClibc.so.0' differs from
'ld-linux.so.3' and 'ld-linux-armhf.so.3'.

Thank you very much in advance!
Best regards,
Andrejs

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Julian S. <js...@ac...>]

This is most likely to be a soname problem, in that ucLibc's soname
is different from the one used for the malloc/free intercepts.

J

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Andrejs B. <sin...@gm...>]

Hi Julian!

Soname for malloc intercepts should be one defined in VG_Z_LIBC_SONAME?
include/pub_tool_redir.h:244:#  define  VG_Z_LIBC_SONAME  libcZdsoZa
           // libc.so*
And soname for libuClibc-0.9.33.2.so is libc.so.0:
[andrejs@arch ~]$ readelf -a ./dev_libs/lib/libuClibc-0.9.33.2.so | grep soname
 0x0000000e (SONAME)                     Library soname: [libc.so.0]

Best regards,
Andrejs

On 5/21/14, Julian Seward wrote:
>
> This is most likely to be a soname problem, in that ucLibc's soname
> is different from the one used for the malloc/free intercepts.
>
> J
>
>

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Philippe W. <phi...@sk...>]

On Wed, 2014-05-21 at 14:14 +0300, Andrejs Bogdanovs wrote:

> On Linux/x86-64 leak was detected.
> On Linux/ARM using --trace-redir=yes option I've found that
> vgpreload_core-arm-linux.so and vgpreload_memcheck-arm-linux.so files
> are not being read.
> Using strace on Valgrind I found that vgpreload_memcheck-arm-linux.so
> is accessed, but not used
> (access("/usr/lib/valgrind/vgpreload_memcheck-arm-linux.so", R_OK) = 0).

If Valgrind does not preload the above, then the redirections will not
happen.

Running 
   valgrind -v -v -v -d -d -d env
might give some lights on what does not work
In particular, what is the value of the LD_PRELOAD env variable ?

Philippe

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Andrejs B. <sin...@gm...>]

On 5/21/14, Philippe Waroquiers wrote:
>
> If Valgrind does not preload the above, then the redirections will not
> happen.
>
> Running
>    valgrind -v -v -v -d -d -d env
> might give some lights on what does not work
> In particular, what is the value of the LD_PRELOAD env variable ?
>
> Philippe
>

Hi Philippe!
I tried command 'valgrind -v -v -v -d -d -d env'
And got:
LD_PRELOAD=/usr/lib/valgrind/vgpreload_core-arm-linux.so:/usr/lib/valgrind/vgpreload_memcheck-arm-linux.so

Full log: http://pastebin.com/JmzjaRKp

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Julian S. <js...@ac...>]

On 05/21/2014 05:24 PM, Andrejs Bogdanovs wrote:
> Hi Julian!
> 
> Soname for malloc intercepts should be one defined in VG_Z_LIBC_SONAME?
> include/pub_tool_redir.h:244:#  define  VG_Z_LIBC_SONAME  libcZdsoZa
>            // libc.so*
> And soname for libuClibc-0.9.33.2.so is libc.so.0:
> [andrejs@arch ~]$ readelf -a ./dev_libs/lib/libuClibc-0.9.33.2.so | grep soname
>  0x0000000e (SONAME)                     Library soname: [libc.so.0]

That soname looks ok, then.  But from the log you posted
later, it is clear that there is no malloc intercepting happening.

For malloc intercepting to work, the files vgpreload_core-arm-linux.so
and vgpreload_memcheck-amd64-linux.so need to be loaded into the process.

Comparing your log with a run with a glibc based system, I see this

  Reading syms from [..]/vgpreload_core-amd64-linux.so
  Reading syms from [..]/vgpreload_memcheck-amd64-linux.so

which is missing in your log.

Does uCLibc implement LD_PRELOAD?  I think that is how Valgrind causes
those two objects to be loaded into the process.

J

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Andrejs B. <sin...@gm...>]

On 5/22/14, Julian Seward wrote:
>
> That soname looks ok, then.  But from the log you posted
> later, it is clear that there is no malloc intercepting happening.
>
> For malloc intercepting to work, the files vgpreload_core-arm-linux.so
> and vgpreload_memcheck-amd64-linux.so need to be loaded into the process.
>
> Comparing your log with a run with a glibc based system, I see this
>
>   Reading syms from [..]/vgpreload_core-amd64-linux.so
>   Reading syms from [..]/vgpreload_memcheck-amd64-linux.so
>
> which is missing in your log.
>
> Does uCLibc implement LD_PRELOAD?  I think that is how Valgrind causes
> those two objects to be loaded into the process.
>
> J
>
That may be a clue!

Actually Valgrind sets LD_PRELOAD env variable to correct string,
pointing to existing /usr/lib/valgrind/vgpreload_core-arm-linux.so and
/usr/lib/valgrind/vgpreload_memcheck-arm-linux.so, but uClibc may have
been configured without LD_PRELOAD.
I've found this in blog entry from NeoSkye:
http://drivingtux.blogspot.com/2013/10/getting-valgrind-memcheck-to-work-with.html

He wrote:
"It turns out that valgrind actually sets the LD_PRELOAD environment
variable early in execution, before loading the executable that is to
be analyzed. This should instruct the runtime link loader to load the
libraries in LD_PRELOAD, which are the vgpreload ones. Unfortunately,
if your uclibc is configured without LD_PRELOAD (like ours was), the
loader will ignore LD_PRELOAD and valgrind won't know that the
vgpreload libraries were never loaded and malloc was not replaced.

tl;dr uclibc must be configured with LD_PRELOAD in order for
valgrind's memcheck to work. "

Will try to do that.

Re: [Valgrind-users] Valgrind doesn't detect leaks on Linux/ARM with uClibc

[Andrejs B. <sin...@gm...>]

On 5/22/14, Andrejs Bogdanovs wrote:
> On 5/22/14, Julian Seward wrote:
>>
>>
>> Does uCLibc implement LD_PRELOAD?  I think that is how Valgrind causes
>> those two objects to be loaded into the process.
>>
>>
>
> tl;dr uclibc must be configured with LD_PRELOAD in order for
> valgrind's memcheck to work. "
>
> Will try to do that.
>
Recompiled uClibc configured with LD_PRELOAD, recompiled example
program and Valgrind and it works now!
Thank you guys!