|
From: Burlen L. <bl...@lb...> - 2016-03-26 19:15:31
|
Hi All,
I like to understand better the following reports. I have a buffer that
I allocated and that gets filled by MPI recv. However, for every access
to the buffer valgrind produce a report similar to the following first
report shown here:
Invalid read of size 4
Address 0x1788a6d0 is 0 bytes inside a block of size 30,121 alloc'd
in subsequent reports "0 bytes inside" the 0 is replaced with the offset
of the access in the buffer. Full output of the first few reports are
shown below.
what I don't understand about these reports are, if I am accessing
memory inside the buffer I allocated how that is a problem?? I spent
some time looking for problems in my code, and verifying that the recv'd
data is accurate. Finally I tried MPI from a different vendor and the
reports are gone!
Thanks for an insight into this!
Burlen
==23010== Invalid read of size 4
==23010== at 0x60683FA: void teca_binary_stream::unpack<unsigned
int>(unsigned int&) (teca_binary_stream.h:108)
==23010== by 0x6433AD9:
teca_metadata::from_stream(teca_binary_stream&) (teca_metadata.cxx:207)
==23010== by 0x5D560BF: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:736)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010== Address 0x1788a6d0 is 0 bytes inside a block of size 30,121
alloc'd
==23010== at 0x4C28C50: malloc (vg_replace_malloc.c:298)
==23010== by 0x4C2AC1C: realloc (vg_replace_malloc.c:785)
==23010== by 0x643276B: teca_binary_stream::resize(unsigned long)
(teca_binary_stream.cxx:87)
==23010== by 0x5D55F3C: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:725)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010==
unpacking 9
==23010== Invalid read of size 8
==23010== at 0x44AB7C: void teca_binary_stream::unpack<unsigned
long>(unsigned long&) (teca_binary_stream.h:108)
==23010== by 0x44A62A:
teca_binary_stream::unpack(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&) (teca_binary_stream.h:147)
==23010== by 0x6433B43:
teca_metadata::from_stream(teca_binary_stream&) (teca_metadata.cxx:214)
==23010== by 0x5D560BF: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:736)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010== Address 0x1788a6d4 is 4 bytes inside a block of size 30,121
alloc'd
==23010== at 0x4C28C50: malloc (vg_replace_malloc.c:298)
==23010== by 0x4C2AC1C: realloc (vg_replace_malloc.c:785)
==23010== by 0x643276B: teca_binary_stream::resize(unsigned long)
(teca_binary_stream.cxx:87)
==23010== by 0x5D55F3C: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:725)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010==
==23010== Invalid read of size 2
==23010== at 0x4C2D3E0: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1018)
==23010== by 0x9E5F2DC: std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned
long, unsigned long, char const*, unsigned long) (in
/usr/lib64/libstdc++.so.6.0.21)
==23010== by 0x44A658:
teca_binary_stream::unpack(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&) (teca_binary_stream.h:150)
==23010== by 0x6433B43:
teca_metadata::from_stream(teca_binary_stream&) (teca_metadata.cxx:214)
==23010== by 0x5D560BF: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:736)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== Address 0x1788a6dc is 12 bytes inside a block of size 30,121
alloc'd
==23010== at 0x4C28C50: malloc (vg_replace_malloc.c:298)
==23010== by 0x4C2AC1C: realloc (vg_replace_malloc.c:785)
==23010== by 0x643276B: teca_binary_stream::resize(unsigned long)
(teca_binary_stream.cxx:87)
==23010== by 0x5D55F3C: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:725)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010==
==23010== Invalid read of size 2
==23010== at 0x4C2D3EE: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1018)
==23010== by 0x9E5F2DC: std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned
long, unsigned long, char const*, unsigned long) (in
/usr/lib64/libstdc++.so.6.0.21)
==23010== by 0x44A658:
teca_binary_stream::unpack(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >&) (teca_binary_stream.h:150)
==23010== by 0x6433B43:
teca_metadata::from_stream(teca_binary_stream&) (teca_metadata.cxx:214)
==23010== by 0x5D560BF: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:736)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== Address 0x1788a6e0 is 16 bytes inside a block of size 30,121
alloc'd
==23010== at 0x4C28C50: malloc (vg_replace_malloc.c:298)
==23010== by 0x4C2AC1C: realloc (vg_replace_malloc.c:785)
==23010== by 0x643276B: teca_binary_stream::resize(unsigned long)
(teca_binary_stream.cxx:87)
==23010== by 0x5D55F3C: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:725)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010==
==23010== Invalid read of size 4
==23010== at 0x60683FA: void teca_binary_stream::unpack<unsigned
int>(unsigned int&) (teca_binary_stream.h:108)
==23010== by 0x6433B56:
teca_metadata::from_stream(teca_binary_stream&) (teca_metadata.cxx:217)
==23010== by 0x5D560BF: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:736)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
==23010== Address 0x1788a6e6 is 22 bytes inside a block of size 30,121
alloc'd
==23010== at 0x4C28C50: malloc (vg_replace_malloc.c:298)
==23010== by 0x4C2AC1C: realloc (vg_replace_malloc.c:785)
==23010== by 0x643276B: teca_binary_stream::resize(unsigned long)
(teca_binary_stream.cxx:87)
==23010== by 0x5D55F3C: teca_cf_reader::get_output_metadata(unsigned
int, std::vector<teca_metadata, std::allocator<teca_metadata> > const&)
(teca_cf_reader.cxx:725)
==23010== by 0x64266C4:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:579)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x642665B:
teca_algorithm::get_output_metadata(std::pair<std::shared_ptr<teca_algorithm>,
unsigned int>&) (teca_algorithm.cxx:573)
==23010== by 0x6426F5A: teca_algorithm::update(unsigned int)
(teca_algorithm.cxx:684)
==23010== by 0x642725D: teca_algorithm::update() (teca_algorithm.cxx:708)
==23010== by 0x4415FA: main (test_descriptive_statistics.cpp:100)
|
|
From: Tom H. <to...@co...> - 2016-03-27 10:52:29
|
On 26/03/16 19:15, Burlen Loring wrote: > I like to understand better the following reports. I have a buffer that > I allocated and that gets filled by MPI recv. However, for every access > to the buffer valgrind produce a report similar to the following first > report shown here: > > Invalid read of size 4 > Address 0x1788a6d0 is 0 bytes inside a block of size 30,121 alloc'd > > in subsequent reports "0 bytes inside" the 0 is replaced with the offset > of the access in the buffer. Full output of the first few reports are > shown below. That does look odd. The only obvious way I can think of to get a report like that is if you had explicitly told valgrind to consider that memory as inaccessible with the VALGRIND_MAKE_MEM_NOACCESS call. Tom -- Tom Hughes (to...@co...) http://compton.nu/ |
|
From: Burlen L. <bl...@lb...> - 2016-03-28 02:29:45
|
On 03/27/2016 03:08 AM, Tom Hughes wrote: > On 26/03/16 19:15, Burlen Loring wrote: > >> I like to understand better the following reports. I have a buffer that >> I allocated and that gets filled by MPI recv. However, for every access >> to the buffer valgrind produce a report similar to the following first >> report shown here: >> >> Invalid read of size 4 >> Address 0x1788a6d0 is 0 bytes inside a block of size 30,121 alloc'd >> >> in subsequent reports "0 bytes inside" the 0 is replaced with the offset >> of the access in the buffer. Full output of the first few reports are >> shown below. > > That does look odd. The only obvious way I can think of to get a > report like that is if you had explicitly told valgrind to consider > that memory as inaccessible with the VALGRIND_MAKE_MEM_NOACCESS call. I haven't done that, but maybe the Open MPI library does. It does indeed seem that they are trying to using these features of valgrind. At least one of the first search hit with key you mention is a question a few days ago to valgrind list about how to mark a buffer as read only. :-) I have notice some other runtime checking that they enabled, and also a number valgrind leak reports in there code. although ompi_info claims the features are disabled. Maybe I follow up with them. Thanks! Re: [Valgrind-users] can memchecker mark memory as read-only ? <https://sourceforge.net/p/valgrind/mailman/message/34933327/> |