|
From: Carl E. L. <ce...@us...> - 2016-11-09 22:25:49
|
Julian, Valgrind developers:
We have some people on the compiler team trying to help out on a
customer issue. The customer tried to use valgrind memcheck and got
several errors that we would like help understanding. In the first
experiment, they get messages about mismatched free() / delete /
delete[]. They had valgrind ignore those issues and then see erorrs on
"get_otrack_shadow_offset_wrk".
Here is the valgrind output from the experiments from the bugzilla
report:
valgrind --soname-synonyms=somalloc=NONE --track-origins=yes --leak-check=no ./mongos
==17387== Memcheck, a memory error detector
==17387== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==17387== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==17387== Command: ./mongos
==17387==
==17387== Mismatched free() / delete / delete []
==17387== at 0x4895888: free (in /usr/lib/valgrind/vgpreload_memcheck-ppc64le-linux.so)
==17387== by 0x59514F: deallocate (new_allocator.h:110)
==17387== by 0x59514F: deallocate (alloc_traits.h:517)
==17387== by 0x59514F: _M_deallocate_buckets (hashtable_policy.h:2010)
==17387== by 0x59514F: _M_deallocate_buckets (hashtable.h:356)
==17387== by 0x59514F: _M_deallocate_buckets (hashtable.h:361)
==17387== by 0x59514F: _M_rehash_aux (hashtable.h:1999)
==17387== by 0x59514F: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:1953)
==17387== by 0x595253: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, true>*) (hashtable.h:1600)
==17387== by 0x5954D3: std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (hashtable_policy.h:600)
==17387== by 0x593693: operator[] (unordered_map.h:668)
==17387== by 0x593693: mongo::InitializerDependencyGraph::addInitializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<mongo::Status (mongo::InitializerContext*)> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (initializer_dependency_graph.cpp:58)
==17387== by 0x591057: mongo::GlobalInitializerRegisterer::GlobalInitializerRegisterer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<mongo::Status (mongo::InitializerContext*)> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (global_initializer_registerer.cpp:44)
==17387== by 0x52D46F: __static_initialization_and_destruction_0(int, int) [clone .constprop.34] (mongos_options_init.cpp:39)
==17387== by 0x137FED3: __libc_csu_init (in /home/acm/opt/src/mongo/mongos)
==17387== by 0x4F830A7: generic_start_main.isra.0 (libc-start.c:247)
==17387== by 0x4F83337: (below main) (libc-start.c:116)
==17387== Address 0x5151fb0 is 0 bytes inside a block of size 16 alloc'd
==17387== at 0x48951D4: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-ppc64le-linux.so)
==17387== by 0x59328F: allocate (new_allocator.h:104)
==17387== by 0x59328F: allocate (alloc_traits.h:491)
==17387== by 0x59328F: std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, true> > >::_M_allocate_buckets(unsigned long) [clone .isra.108] (hashtable_policy.h:1996)
==17387== by 0x595093: _M_allocate_buckets (hashtable.h:347)
==17387== by 0x595093: _M_rehash_aux (hashtable.h:1974)
==17387== by 0x595093: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_rehash(unsigned long, unsigned long const&) (hashtable.h:1953)
==17387== by 0x595253: std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, true>*) (hashtable.h:1600)
==17387== by 0x5954D3: std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mongo::InitializerDependencyGraph::NodeData> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (hashtable_policy.h:600)
==17387== by 0x59356B: operator[] (unordered_map.h:668)
==17387== by 0x59356B: mongo::InitializerDependencyGraph::addInitializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<mongo::Status (mongo::InitializerContext*)> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (initializer_dependency_graph.cpp:46)
==17387== by 0x591057: mongo::GlobalInitializerRegisterer::GlobalInitializerRegisterer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<mongo::Status (mongo::InitializerContext*)> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (global_initializer_registerer.cpp:44)
==17387== by 0x52D46F: __static_initialization_and_destruction_0(int, int) [clone .constprop.34] (mongos_options_init.cpp:39)
==17387== by 0x137FED3: __libc_csu_init (in /home/acm/opt/src/mongo/mongos)
==17387== by 0x4F830A7: generic_start_main.isra.0 (libc-start.c:247)
==17387== by 0x4F83337: (below main) (libc-start.c:116)
So, that is a puzzle. However, I can instruct valgrind to ignore that. But it still fails to start, now with something more odd:
$ valgrind --show-mismatched-frees=no --soname-synonyms=somalloc=NONE --track-origins=yes --leak-check=no ./mongos
==19834== Memcheck, a memory error detector
==19834== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==19834== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==19834== Command: ./mongos
==19834==
MC_(get_otrack_shadow_offset)(ppc64)(off=1688,sz=8)
Memcheck: mc_machine.c:329 (get_otrack_shadow_offset_wrk): the 'impossible' happened.
host stacktrace:
==19834== at 0x3808D9B8: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x3808DB5F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x3808DCDB: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x38078CE3: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x38076FAB: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x380BAA2B: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x381B9BB7: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x380BE19F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x3810D04F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x3810FFEF: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
==19834== by 0x3812BB97: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
sched status:
running_tid=1
Thread 1: status = VgTs_Runnable (lwpid 19834)
==19834== at 0x4F3AC14: __lll_lock_elision (elision-lock.c:60)
==19834== by 0x4F2BBC7: pthread_mutex_lock (pthread_mutex_lock.c:92)
==19834== by 0x602753: mongo::DBConnectionPool::DBConnectionPool() (connpool.cpp:196)
==19834== by 0x5319EB: __static_initialization_and_destruction_0 (global_conn_pool.cpp:35)
==19834== by 0x5319EB: _GLOBAL__sub_I__ZN5mongo14globalConnPoolE (global_conn_pool.cpp:39)
==19834== by 0x137FED3: __libc_csu_init (in /home/acm/opt/src/mongo/mongos)
==19834== by 0x4F830A7: generic_start_main.isra.0 (libc-start.c:247)
==19834== by 0x4F83337: (below main) (libc-start.c:116)
You should be able to see the complete discussion of the issue at
https://bugs.launchpad.net/ubuntu/+source/gcc-5/+bug/1640518
If you have any insight on the errors or suggestions on how to proceed
with Valgrind, that would be appreciated.
Carl Love
|
|
From: Philippe W. <phi...@sk...> - 2016-11-09 22:46:39
|
On Wed, 2016-11-09 at 14:25 -0800, Carl E. Love wrote:
> ==17387== Mismatched free() / delete / delete []
> ==17387== at 0x4895888: free (in /usr/lib/valgrind/vgpreload_memcheck-ppc64le-linux.so)
> ==17387== by 0x59514F: deallocate (new_allocator.h:110)
> ==17387== by 0x59514F: deallocate (alloc_traits.h:517)
....
> ==17387== Address 0x5151fb0 is 0 bytes inside a block of size 16 alloc'd
> ==17387== at 0x48951D4: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-ppc64le-linux.so)
> ==17387== by 0x59328F: allocate (new_allocator.h:104)
> ==17387== by 0x59328F: allocate (alloc_traits.h:491)
....
I assume that the code is correct, and does not call 'free' on some
memory allocated with 'new[]'.
Some false positive have been seen already with such 'Mismatched ....'
due to compiler optimisation: e.g. if the deallocator at
new_allocator.h:110
is calling delete[] (I assume that is what is needed), the compiler
optimiser/inliner/.... might in fact remove the call to delete[] and
directly call the underlying free (if delete[] is implemented on top of
free).
For sure, some inlining was activated as we see several stack entries
that have the same IP (e.g. new_allocator.h:110 and alloc_traits.h:517).
>
> So, that is a puzzle. However, I can instruct valgrind to ignore that. But it still fails to start, now with something more odd:
>
> $ valgrind --show-mismatched-frees=no --soname-synonyms=somalloc=NONE --track-origins=yes --leak-check=no ./mongos
> ==19834== Memcheck, a memory error detector
> ==19834== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
> ==19834== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
> ==19834== Command: ./mongos
> ==19834==
> MC_(get_otrack_shadow_offset)(ppc64)(off=1688,sz=8)
>
> Memcheck: mc_machine.c:329 (get_otrack_shadow_offset_wrk): the 'impossible' happened.
Humph, for this, I do not have much knowledge, so here are some wild
guesses:
Maybe there are some new registers that have been added recently,
and the shadow offsets for this (or these) new registers is not correct.
>From the stacktrace below containing __lll_lock_elision, I assume this
is using transactions to implement some locking technique, and origin
tracking for this transaction register is not ok ?
It looks like there is a request for shadow offset of 8 bytes at offset
1688, but this is a 4 bytes register ?
So, I guess you need to avoid having origin tracking for the transaction
register, or hyou need to add something in mc_machine.c to cover this
register ?
>
> host stacktrace:
> ==19834== at 0x3808D9B8: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x3808DB5F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x3808DCDB: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x38078CE3: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x38076FAB: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x380BAA2B: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x381B9BB7: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x380BE19F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x3810D04F: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x3810FFEF: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
> ==19834== by 0x3812BB97: ??? (in /usr/lib/valgrind/memcheck-ppc64le-linux)
>
> sched status:
> running_tid=1
>
> Thread 1: status = VgTs_Runnable (lwpid 19834)
> ==19834== at 0x4F3AC14: __lll_lock_elision (elision-lock.c:60)
> ==19834== by 0x4F2BBC7: pthread_mutex_lock (pthread_mutex_lock.c:92)
> ==19834== by 0x602753: mongo::DBConnectionPool::DBConnectionPool() (connpool.cpp:196)
> ==19834== by 0x5319EB: __static_initialization_and_destruction_0 (global_conn_pool.cpp:35)
> ==19834== by 0x5319EB: _GLOBAL__sub_I__ZN5mongo14globalConnPoolE (global_conn_pool.cpp:39)
> ==19834== by 0x137FED3: __libc_csu_init (in /home/acm/opt/src/mongo/mongos)
> ==19834== by 0x4F830A7: generic_start_main.isra.0 (libc-start.c:247)
> ==19834== by 0x4F83337: (below main) (libc-start.c:116)
>
> You should be able to see the complete discussion of the issue at
> https://bugs.launchpad.net/ubuntu/+source/gcc-5/+bug/1640518
>
> If you have any insight on the errors or suggestions on how to proceed
> with Valgrind, that would be appreciated.
>
> Carl Love
>
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
|
|
From: Philippe W. <phi...@sk...> - 2016-11-09 23:02:42
|
On Wed, 2016-11-09 at 14:25 -0800, Carl E. Love wrote: > $ valgrind --show-mismatched-frees=no > --soname-synonyms=somalloc=NONE --track-origins=yes > --leak-check=no ./mongos > ==19834== Memcheck, a memory error detector > ==19834== Copyright (C) 2002-2015, and GNU GPL'd, by Julian > Seward et al. > ==19834== Using Valgrind-3.11.0 and LibVEX; rerun with -h for > copyright info > ==19834== Command: ./mongos > ==19834== > MC_(get_otrack_shadow_offset)(ppc64)(off=1688,sz=8) > > Memcheck: mc_machine.c:329 (get_otrack_shadow_offset_wrk): the > 'impossible' happened. BTW, waiting for a solution in Valgrind; you should be able to bypass this problem by not using --track-origins Philippe |
|
From: Paul F. <pa...@fr...> - 2016-11-10 07:03:38
|
On 9 Nov 2016, at 23:25, Carl E. Love wrote:
> Julian, Valgrind developers:
>
> We have some people on the compiler team trying to help out on a
> customer issue. The customer tried to use valgrind memcheck and got
> several errors that we would like help understanding. In the first
> experiment, they get messages about mismatched free() / delete /
> delete[]. They had valgrind ignore those issues and then see erorrs on
> "get_otrack_shadow_offset_wrk".
>
> Here is the valgrind output from the experiments from the bugzilla
> report:
>
> valgrind --soname-synonyms=somalloc=NONE --track-origins=yes --leak-check=no ./mongos
> ==17387== Memcheck, a memory error detector
> ==17387== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
> ==17387== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
> ==17387== Command: ./mongos
> ==17387==
> ==17387== Mismatched free() / delete / delete []
Hi Carl
I'd like to add a little to what Philippe wrote for this case. There are two kind-of 'false positive' situations that I've encountered. Firstly, as already mentioned, if operator new/operator delete is inlined. In particular I see this in class overloads of the operators, which tend to be inlined more often e.g.,
class MyClass
{
public:
void* operator new(std::size_t count); // out of line
void operator delete(void* ptr) { myFree(ptr); } // inline
};
The second situation that I've encountered is if you use an overload that uses a non-standard signature, Valgrind won't see it as a new or delete but will probably still see the underlying call to malloc/free and so will flag a mismatched free/delete.
Looking at your callstack you seem to be using a
std::unordered_map<std::string, std::pair<std::string, mongo::InitializerDependencyGraph::NodeData>
and the error occurs during rehashing. So I doubt that it is a non-standard overload.
Which compiler and standard library are you using?
A+
Paul
|
|
From: Andrew C. M. <and...@gm...> - 2016-11-10 14:35:04
|
On Thu, Nov 10, 2016 at 2:03 AM, Paul Floyd <pa...@fr...> wrote:
>
> On 9 Nov 2016, at 23:25, Carl E. Love wrote:
>
> > Julian, Valgrind developers:
> >
> > We have some people on the compiler team trying to help out on a
> > customer issue. The customer tried to use valgrind memcheck and got
> > several errors that we would like help understanding. In the first
> > experiment, they get messages about mismatched free() / delete /
> > delete[]. They had valgrind ignore those issues and then see erorrs on
> > "get_otrack_shadow_offset_wrk".
> >
> > Here is the valgrind output from the experiments from the bugzilla
> > report:
> >
> > valgrind --soname-synonyms=somalloc=NONE --track-origins=yes
> --leak-check=no ./mongos
> > ==17387== Memcheck, a memory error detector
> > ==17387== Copyright (C) 2002-2015, and GNU GPL'd, by Julian
> Seward et al.
> > ==17387== Using Valgrind-3.11.0 and LibVEX; rerun with -h for
> copyright info
> > ==17387== Command: ./mongos
> > ==17387==
> > ==17387== Mismatched free() / delete / delete []
>
> Hi Carl
>
> I'd like to add a little to what Philippe wrote for this case. There are
> two kind-of 'false positive' situations that I've encountered. Firstly, as
> already mentioned, if operator new/operator delete is inlined. In
> particular I see this in class overloads of the operators, which tend to be
> inlined more often e.g.,
>
> class MyClass
> {
> public:
> void* operator new(std::size_t count); // out of line
> void operator delete(void* ptr) { myFree(ptr); } // inline
> };
>
> The second situation that I've encountered is if you use an overload that
> uses a non-standard signature, Valgrind won't see it as a new or delete but
> will probably still see the underlying call to malloc/free and so will flag
> a mismatched free/delete.
>
> Looking at your callstack you seem to be using a
>
> std::unordered_map<std::string, std::pair<std::string, mongo::
> InitializerDependencyGraph::NodeData>
>
> and the error occurs during rehashing. So I doubt that it is a
> non-standard overload.
>
> Which compiler and standard library are you using?
>
> A+
> Paul
>
>
>
Hi -
As it happens, I'm both a lurker on this list from some long ago dev work
on valgrind, and I'm the customer referenced in the original email, so I
can answer questions directly.
We run our code under ASAN, which does detect mismatched free/delete, and
we are fairly certain that we don't have any instances of that problem. So
this is almost certainly due to the inlining issue as described above, and
all of the stacks do look like they reach into the std library headers.
That is sort of unfortunate, because it appears to mean that if you are
using a modern libstdc++ and an aggressive optimization level, you are
going to get a ton of false positives that would need to be suppressed
since they aren't under the control of the developer. Writing suppressions
for std library headers is going to be tricky and fragile. The alternative
is --show-mismatched-frees=no, which will work fine for us as a workaround
now, but would be not generally a good idea since it could mask real user
errors. In any event, we have a workaround for this.
For the other issue about the __lll_lock_elision error, that was
encountered running valgrind on a ppc64le POWER8 machine running Ubuntu
16.04. The compiler was the system GCC, version 5.4.0. I've just tried
again today without the --track-origins=yes flag, and I can confirm that
I'm able to get the process running under valgrind if I omit the
--track-origins=yes flag. This is unfortunate too, because if valgrind does
report an error, I've found that almost always you need --track-origins=yes
to really sort it out.
However, we have now encountered the following error:
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019|
--17827-- WARNING: unhandled ppc64le-linux syscall: 326
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019|
--17827-- You may be able to write your own handler.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019|
--17827-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.497+0000 s40019|
--17827-- Nevertheless we consider this a bug. Please report
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.497+0000 s40019|
--17827-- it at http://valgrind.org/support/bug_reports.html.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.509+0000 s40019|
2016-11-10T14:24:04.509+0000 I - [mongosMain] Assertion:
15863:listen(): invalid socket? Function not implemented src/mon
go/util/net/listen.cpp 184
Looking at listen.cpp:
182 SOCKET sock = ::socket(me.getType(), SOCK_STREAM, 0);
183 ScopeGuard socketGuard = MakeGuard(&closesocket, sock);
184 massert(15863,
185 str::stream() << "listen(): invalid socket? " <<
errnoWithDescription(),
186 sock >= 0);
The process ends up terminating after this because it can't open its
listening socket.
Are the socket syscalls somehow not implemented in valgrind for ppc64le?
Thanks,
Andrew
|
|
From: Julian S. <js...@ac...> - 2016-11-10 15:01:56
|
On 10/11/16 15:34, Andrew C. Morrow wrote:
Your message was timely; I was just about to reply to and it showed up
seconds before.
> [mismatched alloc/free messages]
> is --show-mismatched-frees=no, which will work fine for us as a workaround
> now, but would be not generally a good idea since it could mask real user
> errors. In any event, we have a workaround for this.
I've seen this a lot running Firefox on Valgrind. My impression is that it
is due to what you could call "differential inlining". Imagine that we have
this:
operator new (size_t size) { ... return malloc(size) ... }
operator delete ( void* p) { ... free(p) ... }
If both |new| and |delete| are inlined, then Memcheck only ever sees calls
to |malloc| and |free|, and there's no problem (and no mismatch checking).
If neither are inlined, then there's no problem and everything works as
originally designed. The problem occurs when one but not the other
is inlined. Then Memcheck observes, for example, allocations from
|new| (which is not inlined) being returned to |free| (because |delete|
got inlined).
I don't have a good fix for this, so I always just run with
--show-mismatched-frees=no.
> For the other issue about the __lll_lock_elision error, that was
> encountered running valgrind on a ppc64le POWER8 machine running Ubuntu
> 16.04. The compiler was the system GCC, version 5.4.0. I've just tried
> again today without the --track-origins=yes flag, and I can confirm that
> I'm able to get the process running under valgrind if I omit the
> --track-origins=yes flag. This is unfortunate too, because if valgrind does
> report an error, I've found that almost always you need --track-origins=yes
> to really sort it out.
Fixing this is a 1-liner if we can figure out what is at the given offset
of the ppc64le guest state. Which we could do if we can see the instruction
it's failing at. Ah, so it's failing at the basic block that starts here:
==19834== at 0x4F3AC14: __lll_lock_elision (elision-lock.c:60)
Can you objdump -d libpthread (I think it's that) and show the block
that follows? We're probably looking for an instruction that pokes
some very obscure system status register or something like that; we're
not looking for vanilla integer arithmetic or anything ordinary.
But first -- before you do *any* of that -- can you try with V 3.12.0
(or make sure you're using it)? I say that because Carl fixed a exactly
that stuff earlier this year.
r15895 | carll | 2016-06-27 17:50:29 +0200 (Mon, 27 Jun 2016) | 3 lines
Add the HW register support for missing registers in
get_otrack_shadow_offset_wrk(). The registers are: NRADDR, NRADDR_GPR2,
(REDIR_STACK, TFHAR, TEXASR, TEXASRU, TFIAR, PPR, PSPB.
> However, we have now encountered the following error:
> [..]
> Are the socket syscalls somehow not implemented in valgrind for ppc64le?
That's such basic functionality that it merely confirms my suspicion
that you're not running the latest (3.12.0).
J
|
|
From: Andrew C. M. <and...@gm...> - 2016-11-10 16:03:38
|
> > > > > However, we have now encountered the following error: > > [..] > > Are the socket syscalls somehow not implemented in valgrind for ppc64le? > > That's such basic functionality that it merely confirms my suspicion > that you're not running the latest (3.12.0). > > J > > Indeed we are not. Looks like Ubuntu 16.04 bundles 3.11.0: $ valgrind --version valgrind-3.11.0 I'll get 3.12 built from source and give it a try. Thanks for your help, Andrew |
|
From: Andrew C. M. <and...@gm...> - 2016-11-10 18:27:33
|
OK, up and running with 3.12. Thanks again. Andrew On Thu, Nov 10, 2016 at 11:03 AM, Andrew C. Morrow < and...@gm...> wrote: > >> >> > However, we have now encountered the following error: >> > [..] >> > Are the socket syscalls somehow not implemented in valgrind for ppc64le? >> >> That's such basic functionality that it merely confirms my suspicion >> that you're not running the latest (3.12.0). >> >> J >> >> > Indeed we are not. Looks like Ubuntu 16.04 bundles 3.11.0: > > $ valgrind --version > valgrind-3.11.0 > > I'll get 3.12 built from source and give it a try. > > Thanks for your help, > Andrew > > > |