|
From: Bart V. A. <bar...@gm...> - 2006-12-13 08:10:28
|
I'm trying to get function wrapping working for pthread_spin_unlock()
in my drd tool, but have not been successful until now (function
wrapping for pthread_spin_lock() works however). I have enabled symtab
tracing in Valgrind, and it looks like Valgrind's ELF reader ignores
lipthread's pthread_spin_unlock() because it thinks this symbol has
value zero ? This is not how readelf sees libpthread.so however ...
$ VALGRIND_LIB=.in_place coregrind/valgrind --trace-redir=yes
--trace-symtab=yes --tool=drd drd/tests/pth_detached 0 2
[ ... ]
--6105-- Reading syms from
/home/bart/software/valgrind-svn/drd/vgpreload_drd-x86-linux.so
(0x401F000)
[ ... ]
raw symbol [ 38]: GLO FUN : val 0x000401FBD8, sz 83
_vgwZZ_libpthreadZdsoZd0_pthreadZuspinZuunlock
record [ 38]: val 0x000401FBD8, sz 83
_vgwZZ_libpthreadZdsoZd0_pthreadZuspinZuunlock
[ ... ]
--6105-- Reading syms from /lib/libpthread-2.4.so (0x403C000)
soname=libpthread.so.0
[ ... ]
raw symbol [ 107]: GLO FUN : val 0x0000000000, sz 13
pthread_spin_unlock@@GLIBC_2.2
ignore -- valu=0: pthread_spin_unlock@@GLIBC_2.2
[ ... ]
raw symbol [ 20]: GLO FUN : val 0x0000000000, sz 13 pthread_spin_unlock
ignore -- valu=0: pthread_spin_unlock
[ ... ]
$ readelf -a /lib/libpthread-2.4.so
[ ... ]
Symbol table '.dynsym' contains 339 entries:
Num: Value Size Type Bind Vis Ndx Name
[ ... ]
212: 000090e0 13 FUNC GLOBAL DEFAULT 13
pthread_spin_unlock@@GLIBC_2.2
[ ... ]
Symbol table '.symtab' contains 781 entries:
Num: Value Size Type Bind Vis Ndx Name
[ ... ]
654: 000090e0 13 FUNC GLOBAL DEFAULT 13 pthread_spin_unlock
[ ... ]
Bart.
|
|
From: Tom H. <to...@co...> - 2006-12-13 08:34:54
|
In message <e2e...@ma...>
Bart Van Assche <bar...@gm...> wrote:
> I'm trying to get function wrapping working for pthread_spin_unlock()
> in my drd tool, but have not been successful until now (function
> wrapping for pthread_spin_lock() works however). I have enabled symtab
> tracing in Valgrind, and it looks like Valgrind's ELF reader ignores
> lipthread's pthread_spin_unlock() because it thinks this symbol has
> value zero ? This is not how readelf sees libpthread.so however ...
I think the problem is that pthread_spin_unlock is an alias for
pthread_spin_init (ie they are the same routine) and valgrind sees
the init routine first so attaches that name to the address:
dellow [~] % objdump -T /lib64/libpthread.so.0 | grep pthread_spin_unlock
000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_unlock
dellow [~] % objdump -T /lib64/libpthread.so.0 | grep 000000382700ab70
000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_init
000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_unlock
I don't think valgrind's symbol table can cope with a many-one mapping
from names to addresses, so we just remember the first name we see.
Given that the main use of the symbol table is mapping the other way
for backtraces, this makes some sense, as you can only show one name
for an address in a backtrace.
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|
|
From: Bart V. A. <bar...@gm...> - 2006-12-13 09:26:08
|
On 12/13/06, Tom Hughes <to...@co...> wrote: > > I think the problem is that pthread_spin_unlock is an alias for > pthread_spin_init (ie they are the same routine) and valgrind sees > the init routine first so attaches that name to the address: > > dellow [~] % objdump -T /lib64/libpthread.so.0 | grep pthread_spin_unlock > 000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_unlock > > dellow [~] % objdump -T /lib64/libpthread.so.0 | grep 000000382700ab70 > 000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_init > 000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 pthread_spin_unlock > > I don't think valgrind's symbol table can cope with a many-one mapping > from names to addresses, so we just remember the first name we see. > > Given that the main use of the symbol table is mapping the other way > for backtraces, this makes some sense, as you can only show one name > for an address in a backtrace. > > Tom This explanation sounds perfectly reasonable, but doesn't give me a solution: does the above mean that it's impossible to implement a wrapper for pthread_spin_unlock() in a Valgrind tool ? |
|
From: Julian S. <js...@ac...> - 2006-12-14 00:02:42
|
On Wednesday 13 December 2006 09:26, Bart Van Assche wrote: > On 12/13/06, Tom Hughes <to...@co...> wrote: > > I think the problem is that pthread_spin_unlock is an alias for > > pthread_spin_init (ie they are the same routine) and valgrind sees > > the init routine first so attaches that name to the address: > > > > dellow [~] % objdump -T /lib64/libpthread.so.0 | grep > > pthread_spin_unlock 000000382700ab70 g DF .text 0000000000000009 > > GLIBC_2.2.5 pthread_spin_unlock > > > > dellow [~] % objdump -T /lib64/libpthread.so.0 | grep 000000382700ab70 > > 000000382700ab70 g DF .text 0000000000000009 GLIBC_2.2.5 > > pthread_spin_init 000000382700ab70 g DF .text 0000000000000009 > > GLIBC_2.2.5 pthread_spin_unlock > > > > I don't think valgrind's symbol table can cope with a many-one mapping > > from names to addresses, so we just remember the first name we see. > > > > Given that the main use of the symbol table is mapping the other way > > for backtraces, this makes some sense, as you can only show one name > > for an address in a backtrace. > > > > Tom > > This explanation sounds perfectly reasonable, but doesn't give me a > solution: does the above mean that it's impossible to implement a > wrapper for pthread_spin_unlock() in a Valgrind tool ? Well, you could write a wrapper for pthread_spin_init() instead? Or supply wrappers for both, factoring the real work into a routine shared by both wrappers. The latter sounds like a more robust approach. J |
|
From: Bart V. A. <bar...@gm...> - 2006-12-14 09:14:44
|
On 12/14/06, Julian Seward <js...@ac...> wrote: > Well, you could write a wrapper for pthread_spin_init() instead? > Or supply wrappers for both, factoring the real work into a routine > shared by both wrappers. The latter sounds like a more robust approach. The problem is that with the current implementation of function wrapping, calls to pthread_spin_unlock() are not wrapped at all -- neither to the wrapper defined for pthread_spin_unlock() nor to the wrapper defined for pthread_spin_init(). I see two possible solutions: - Change Valgrind's core such that it supports function wrapping for functions like pthread_spin_unlock(). - Convince Valgrind's users to use the pthread_mutex_...() calls instead of the pthread_spin_...() calls. At least with the glibc implementation of pthread_mutex_lock(), no system calls are performed if the mutex can be locked immediately. So there is no real performance advantage in using pthread_spin_...() calls over pthread_mutex_...() calls. Additionally, the pthread_mutex_lock() call is more OS-friendly then pthread_spin_lock() when it has to block. Bart. |
|
From: Tom H. <to...@co...> - 2006-12-14 09:30:44
|
In message <e2e...@ma...>
Bart Van Assche <bar...@gm...> wrote:
> On 12/14/06, Julian Seward <js...@ac...> wrote:
>
>> Well, you could write a wrapper for pthread_spin_init() instead?
>> Or supply wrappers for both, factoring the real work into a routine
>> shared by both wrappers. The latter sounds like a more robust approach.
>
> The problem is that with the current implementation of function
> wrapping, calls to pthread_spin_unlock() are not wrapped at all --
> neither to the wrapper defined for pthread_spin_unlock() nor to the
> wrapper defined for pthread_spin_init(). I see two possible solutions:
If you have setup a wrapper for init, and unlock is at the same
address in your copy of the library, then that wrapper should
absolutely be getting called on a call to unlock.
The name is only used to find the right code to wrap - the trap
into the wrapper is all based on the address so if code at the
wrapped address is executed the wrapper should be called.
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|
|
From: Bart V. A. <bar...@gm...> - 2006-12-14 10:43:18
Attachments:
pth_detached.c
vg-output.txt
|
On 12/14/06, Tom Hughes <to...@co...> wrote:
> Bart Van Assche <bar...@gm...> wrote:
>
> > On 12/14/06, Julian Seward <js...@ac...> wrote:
> >
> >> Well, you could write a wrapper for pthread_spin_init() instead?
> >> Or supply wrappers for both, factoring the real work into a routine
> >> shared by both wrappers. The latter sounds like a more robust approach.
> >
> > The problem is that with the current implementation of function
> > wrapping, calls to pthread_spin_unlock() are not wrapped at all --
> > neither to the wrapper defined for pthread_spin_unlock() nor to the
> > wrapper defined for pthread_spin_init(). I see two possible solutions:
>
> If you have setup a wrapper for init, and unlock is at the same
> address in your copy of the library, then that wrapper should
> absolutely be getting called on a call to unlock.
>
> The name is only used to find the right code to wrap - the trap
> into the wrapper is all based on the address so if code at the
> wrapped address is executed the wrapper should be called.
>
> Tom
I have performed the following test:
- Compiled Valgrind from source (revision 6400 + latest drd patch +
enabled pthread_spin_... wrappers in drd/drd_preloaded.c by changing
line 375 from #if 0 into #if 1).
- Used pth_detached.c as test program (see also attachment). I
modified pth_detached.c such that it uses spinlocks.
- Ran the following test:
$ VALGRIND_LIB=.in_place coregrind/valgrind -v --tool=drd
none/tests/pth_detached
I have attached the output (vg-output.txt). The
output of the following command is also very interesting, but turned
out to be too large to include as an attachment:
VALGRIND_LIB=.in_place coregrind/valgrind -v --trace-symtab=yes
--tool=drd none/tests/pth_detached
- As you can see, this test triggers an assertion failure in drd. This
test namely triggers a situation where drd thinks that two threads own
the spinlock at the same time. This is because Valgrind's core wrapped
pthread_spin_init() and pthread_spin_lock(), but not
pthread_spin_unlock().
- My conclusion is that Valgrind's core is unable to wrap
pthread_spin_unlock() because it does not add the symbol
pthread_spin_unlock to its symbol table while reading the ELF file
/lib/libpthread.so. I concluded this from the following output (with
--trace-symtab=yes):
raw symbol [ 103]: GLO FUN : val 0x0000000000, sz 13
pthread_spin_unlock@@GLIBC_2.2
ignore -- valu=0: pthread_spin_unlock@@GLIBC_2.2
Did I misinterprete Valgrind's debugging output ?
Bart.
|
|
From: Tom H. <to...@co...> - 2006-12-14 10:54:19
|
In message <e2e...@ma...>
Bart Van Assche <bar...@gm...> wrote:
> On 12/14/06, Tom Hughes <to...@co...> wrote:
>> Bart Van Assche <bar...@gm...> wrote:
>>
>> > On 12/14/06, Julian Seward <js...@ac...> wrote:
>> >
>> >> Well, you could write a wrapper for pthread_spin_init() instead?
>> >> Or supply wrappers for both, factoring the real work into a routine
>> >> shared by both wrappers. The latter sounds like a more robust approach.
>> >
>> > The problem is that with the current implementation of function
>> > wrapping, calls to pthread_spin_unlock() are not wrapped at all --
>> > neither to the wrapper defined for pthread_spin_unlock() nor to the
>> > wrapper defined for pthread_spin_init(). I see two possible solutions:
>>
>> If you have setup a wrapper for init, and unlock is at the same
>> address in your copy of the library, then that wrapper should
>> absolutely be getting called on a call to unlock.
>>
>> The name is only used to find the right code to wrap - the trap
>> into the wrapper is all based on the address so if code at the
>> wrapped address is executed the wrapper should be called.
>
> I have performed the following test:
We know that it isn't installing a wrapper for unlock (at least
under that name) but is it installing one for init? ie what does
trace-symtab about the init symbol?
More to the point, what does --trace-redir say? That should tell
you more about wrapper than watching the symbol table.
Even more to the point, what does objdump say about the symbols? ie
run this command:
objdump -T <path to pthread library> | fgrep pthread_spin
Tom
--
Tom Hughes (to...@co...)
http://www.compton.nu/
|
|
From: Bart V. A. <bar...@gm...> - 2006-12-14 12:40:05
|
On 12/14/06, Tom Hughes <to...@co...> wrote: > > We know that it isn't installing a wrapper for unlock (at least > under that name) but is it installing one for init? ie what does > trace-symtab about the init symbol? > > More to the point, what does --trace-redir say? That should tell > you more about wrapper than watching the symbol table. > > Even more to the point, what does objdump say about the symbols? ie > run this command: > > objdump -T <path to pthread library> | fgrep pthread_spin >From the --trace-redir=yes output (I have filtered the output on pthread_spin_ -- please note that pthread_spin_unlock is missing from the second REDIR STATE section): --5596-- << --5596-- ------ REDIR STATE after VG_(redir_notify_new_SegInfo) ------ --5596-- TOPSPECS of soname NONE --5596-- libpthread.so.0 pthread_create@* W-> 0x04020154 --5596-- libpthread.so.0 pthread_detach W-> 0x040200E4 --5596-- libpthread.so.0 pthread_spin_trylock W-> 0x0401FFBF --5596-- libpthread.so.0 pthread_spin_lock W-> 0x0401FF1F --5596-- libpthread.so.0 pthread_spin_destroy W-> 0x0401FE87 --5596-- libpthread.so.0 pthread_spin_init W-> 0x0401FDE2 --5596-- libpthread.so.0 pthread_cond_destroy@* W-> 0x0401FD4A --5596-- libpthread.so.0 pthread_cond_init@* W-> 0x0401FCA5 --5596-- libpthread.so.0 pthread_cond_timedwait@* W-> 0x0401FBB1 --5596-- libpthread.so.0 pthread_cond_wait@* W-> 0x0401FAC6 --5596-- libpthread.so.0 pthread_mutex_unlock W-> 0x0401FA2E --5596-- libpthread.so.0 pthread_mutex_trylock W-> 0x0401F98E --5596-- libpthread.so.0 pthread_mutex_lock W-> 0x0401F8EE --5596-- libpthread.so.0 pthread_mutex_destroy W-> 0x0401F856 --5596-- libpthread.so.0 pthread_mutex_init W-> 0x0401F7B1 --5596-- libpthread.so.0 pthread_join W-> 0x0401F708 --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname ld-linux.so.2 --5596-- ------ ACTIVE ------ --5596-- >> ... choosing between 'pthread_spin_destroy' and 'pthread_spin_destroy' choosing between 'pthread_spin_lock' and 'pthread_spin_lock' choosing between 'pthread_spin_trylock' and 'pthread_spin_trylock' choosing between 'pthread_spin_unlock' and 'pthread_spin_init' choosing between 'pthread_spin_unlock' and 'pthread_spin_init' ... choosing between 'pthread_spin_init' and 'pthread_spin_init' ... --5596-- << --5596-- ------ REDIR STATE after VG_(redir_notify_new_SegInfo) ------ --5596-- TOPSPECS of soname libpthread.so.0 --5596-- TOPSPECS of soname NONE --5596-- libpthread.so.0 pthread_create@* W-> 0x04020154 --5596-- libpthread.so.0 pthread_detach W-> 0x040200E4 --5596-- libpthread.so.0 pthread_spin_trylock W-> 0x0401FFBF --5596-- libpthread.so.0 pthread_spin_lock W-> 0x0401FF1F --5596-- libpthread.so.0 pthread_spin_destroy W-> 0x0401FE87 --5596-- libpthread.so.0 pthread_spin_init W-> 0x0401FDE2 --5596-- libpthread.so.0 pthread_cond_destroy@* W-> 0x0401FD4A --5596-- libpthread.so.0 pthread_cond_init@* W-> 0x0401FCA5 --5596-- libpthread.so.0 pthread_cond_timedwait@* W-> 0x0401FBB1 --5596-- libpthread.so.0 pthread_cond_wait@* W-> 0x0401FAC6 --5596-- libpthread.so.0 pthread_mutex_unlock W-> 0x0401FA2E --5596-- libpthread.so.0 pthread_mutex_trylock W-> 0x0401F98E --5596-- libpthread.so.0 pthread_mutex_lock W-> 0x0401F8EE --5596-- libpthread.so.0 pthread_mutex_destroy W-> 0x0401F856 --5596-- libpthread.so.0 pthread_mutex_init W-> 0x0401F7B1 --5596-- libpthread.so.0 pthread_join W-> 0x0401F708 --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname NONE --5596-- TOPSPECS of soname ld-linux.so.2 --5596-- ------ ACTIVE ------ --5596-- 0x04041670 (pthread_create@@GLIB) W-> 0x04020154 pthread_create@* --5596-- 0x04042070 (pthread_create@GLIBC) W-> 0x04020154 pthread_create@* --5596-- 0x04042530 (pthread_detach ) W-> 0x040200E4 pthread_detach --5596-- 0x04042590 (pthread_join ) W-> 0x0401F708 pthread_join --5596-- 0x040432D0 (pthread_mutex_init ) W-> 0x0401F7B1 pthread_mutex_init --5596-- 0x040433B0 (pthread_mutex_destro) W-> 0x0401F856 pthread_mutex_destroy --5596-- 0x040433E0 (pthread_mutex_lock ) W-> 0x0401F8EE pthread_mutex_lock --5596-- 0x04043720 (pthread_mutex_tryloc) W-> 0x0401F98E pthread_mutex_trylock --5596-- 0x04043E00 (pthread_mutex_unlock) W-> 0x0401FA2E pthread_mutex_unlock --5596-- 0x040445A0 (pthread_cond_init@@G) W-> 0x0401FCA5 pthread_cond_init@* --5596-- 0x04044650 (pthread_cond_destroy) W-> 0x0401FD4A pthread_cond_destroy@* --5596-- 0x04044760 (pthread_cond_wait@@G) W-> 0x0401FAC6 pthread_cond_wait@* --5596-- 0x04044970 (pthread_cond_timedwa) W-> 0x0401FBB1 pthread_cond_timedwait@* --5596-- 0x04044DB0 (pthread_cond_init@GL) W-> 0x0401FCA5 pthread_cond_init@* --5596-- 0x04044DE0 (pthread_cond_destroy) W-> 0x0401FD4A pthread_cond_destroy@* --5596-- 0x04044E10 (pthread_cond_wait@GL) W-> 0x0401FAC6 pthread_cond_wait@* --5596-- 0x04044E80 (pthread_cond_timedwa) W-> 0x0401FBB1 pthread_cond_timedwait@* --5596-- 0x04045090 (pthread_spin_destroy) W-> 0x0401FE87 pthread_spin_destroy --5596-- 0x040450A0 (pthread_spin_lock ) W-> 0x0401FF1F pthread_spin_lock --5596-- 0x040450C0 (pthread_spin_trylock) W-> 0x0401FFBF pthread_spin_trylock --5596-- 0x040450E0 (pthread_spin_init ) W-> 0x0401FDE2 pthread_spin_init --5596-- >> ... --5596-- REDIR: 0x40450E0 (pthread_spin_init) redirected to 0x401FDE2 (pthread_spin_init) ... .--5596-- REDIR: 0x40450A0 (pthread_spin_lock) redirected to 0x401FF1F (pthread_spin_lock) ... objdump output: $ objdump -T /lib/libpthread.so.0 |grep pthread_spin 000090c0 g DF .text 00000018 GLIBC_2.2 pthread_spin_trylock 000090a0 g DF .text 0000000f GLIBC_2.2 pthread_spin_lock 00009090 g DF .text 00000007 GLIBC_2.2 pthread_spin_destroy 000090e0 g DF .text 0000000d GLIBC_2.2 pthread_spin_unlock 000090e0 g DF .text 0000000d GLIBC_2.2 pthread_spin_init Bart. |
|
From: Julian S. <js...@ac...> - 2006-12-15 14:22:30
|
Bart As Tom says, --trace-redir=yes is your friend here. It tells you the "TOPSPECS" -- that is, the redirection specifications it knows about, and the "ACTIVE" state -- that is, the redirections that are actually active right now. If you haven't already done so, you should read section 2.10 of the manual (http://www.valgrind.org/docs/manual/manual-core.html#manual-core.wrapping) and particularly section 2.10.4, since you are making extensive use of function wrapping. I think you have what you want. Firstly > $ objdump -T /lib/libpthread.so.0 |grep pthread_spin > 000090e0 g DF .text 0000000d GLIBC_2.2 pthread_spin_unlock > 000090e0 g DF .text 0000000d GLIBC_2.2 pthread_spin_init shows that pthread_spin_init and pthread_spin_unlock have the same address, and the --trace-symtab output > choosing between 'pthread_spin_unlock' and 'pthread_spin_init' > choosing between 'pthread_spin_unlock' and 'pthread_spin_init' confirms it (because it says it is having to make a decision about which name to associate with that address. Unfortunately the message is a bit stupid and does not say which name it did choose.) Secondly, > --5596-- 0x040450E0 (pthread_spin_init ) W-> 0x0401FDE2 shows that (1) pthread_spin_init was chosen, and (2) you have an active wrapper for it. Observe that the page offset (0x0E0) is the same as objdump said, which helps us be sure it's the right symbol. Hence eventually > --5596-- REDIR: 0x40450E0 (pthread_spin_init) redirected to 0x401FDE2 > (pthread_spin_init) Isn't that what you want? J |
|
From: Bart V. A. <bar...@gm...> - 2006-12-15 16:21:37
|
On 12/15/06, Julian Seward <js...@ac...> wrote: > > Secondly, > > > --5596-- 0x040450E0 (pthread_spin_init ) W-> 0x0401FDE2 > > shows that (1) pthread_spin_init was chosen, and (2) you have an > active wrapper for it. Observe that the page offset (0x0E0) is the > same as objdump said, which helps us be sure it's the right symbol. > Hence eventually > > > --5596-- REDIR: 0x40450E0 (pthread_spin_init) redirected to 0x401FDE2 > > (pthread_spin_init) > > Isn't that what you want? By this time I found out that the pthread_spin_init() wrapper is called when the client calls pthread_spin_unlock(). Not really what I was hoping for, but it allowed me to add support for spinlocks to drd. Sorry for my confusion and that I didn't see this earlier. Bart. |
|
From: Nicholas N. <nj...@cs...> - 2006-12-15 22:25:21
|
On Fri, 15 Dec 2006, Julian Seward wrote: > shows that pthread_spin_init and pthread_spin_unlock have the same > address, and the --trace-symtab output > >> choosing between 'pthread_spin_unlock' and 'pthread_spin_init' >> choosing between 'pthread_spin_unlock' and 'pthread_spin_init' > > confirms it (because it says it is having to make a decision about > which name to associate with that address. Unfortunately the message > is a bit stupid and does not say which name it did choose.) I think it chooses the shorter one. Reason being is that often the two names will be "foo" vs "foo@@GLIBC2.1" or something like that. Nick |