|
From: Ashley P. <as...@qu...> - 2005-11-30 18:44:33
Attachments:
v.alloc
alloc_test.c
|
Hi all.
(This wasn't meant to turn into such an epic, sorry....)
I'm in the process of instrumenting my allocator so it works with
valgrind properly. Whilst the actual amount code involved is tiny (four
lines so far) it's proving to be a bit of a headache.
I've written a test program which does a sequence of alloc/free calls,
making sure to memset all memory it allocates and (optionally) doing one
invalid read (Address xx is 0 bytes after a block of size xx alloc'd)
for every free. Running this against libc this code is either valgrind
clean or reports exactly 1 error per free depending on usage which is
exactly what I'd expect.
When I use this same program to test my own allocator I get *only* get
331 errors from despite doing 505 alloc/free calls, I also get the
following assertion when the program finishes:
Memcheck: mac_leakcheck.c:705 (vgMAC_do_detect_memory_leaks): Assertion
'lc_shadows[i]->data + lc_shadows[i]->size < lc_shadows[i+1]->data'
failed.
The allocator I'm instrumenting is fairly simple, it's effectively one
big superblock (mmap()ed) with chunks carved out of it as needed. There
are no redzones although the allocator heeds alignment requests so
allocated blocks aren't tightly packed, the following comment in
<valgrind/valgrind.h> caught my eye:
If you're allocating memory via superblocks, and then handing out small
chunks of each superblock, if you don't have redzones on your small
blocks, it's worth marking the superblock with VALGRIND_MAKE_NOACCESS
when it's created, so that block overruns are detected. But if you can
put redzones on, it's probably better to not do this, so that messages
for small overruns are described in terms of the small block rather than
the superblock (but if you have a big overrun that skips over a redzone,
you could miss an error this way). See memcheck/tests/custom_alloc.c
for an example.
Calling VALGRIND_MAKE_NOACCESS doesn't seem to make any difference, I
get the same error both with and without it, more on this later...
The test code I have keeps allocating memory until it fails (alloc
returns NULL), frees all the memory again then repeats. On the first
loop V picks up 1 out of 131 invalid reads, on the second 82/254...
loop count: reads that V reports/actual invalid reads
1: 1/131
2: 82/254
3: 201/373
4: 331/505
5: 448/629
What looks to me to be happening is that VALGRIND_MAKE_NOACCESS isn't
doing anything, VALGRIND_MALLOCLIKE_BLOCK() is doing the right thing as
is VALGRIND_FREELIKE_BLOCK() however errors aren't being reported until
the second time the memory is used, that is on the second iteration of
my test the invalid read overflow is in a block that was freed in the
previous loop.... if you see what I mean.
On the basis of the above I changed the code to call NOACCESS call to
immediately tell V the memory has been allocated and then freed again
but it didn't make any difference, my bootstrap function now looks like
this:
VALGRIND_MAKE_NOACCESS(mbase+base,size);
if ( 1 ) {
VALGRIND_MALLOCLIKE_BLOCK(mbase+base,size,0,0);
VALGRIND_FREELIKE_BLOCK(mbase+base,0);
}
I've attached the source to my test code and a script of it running,
both in the ERROR_ON_FREE mode where V misses errors and asserts and in
normal mode where the are no errors but V still asserts.
Can you shed any light on the cause of either of these problems and is
it possible for me to get access to what V thinks the status of the
whole region is?
I'm running V from SVN, it's about two weeks old however, svnversion is
reporting "1446:5035M"
Another thing, VALGRIND_MAKE_NOACCESS is in memcheck.h still whereas the
other macros are in valgrind.h. valgrind also has MEMPOOL macros, what
are these and should I be using them instead?
Ashley,
Ps, It's probably not *too* hard to extract the allocator out into a
standalone entity, perhaps this would make a good test case?
|
|
From: Ashley P. <as...@qu...> - 2005-12-06 18:39:10
|
On Wed, 2005-11-30 at 18:34 +0000, Ashley Pittman wrote:
> Hi all.
>
> (This wasn't meant to turn into such an epic, sorry....)
>
> I'm in the process of instrumenting my allocator so it works with
> valgrind properly. Whilst the actual amount code involved is tiny (four
> lines so far) it's proving to be a bit of a headache.
>
> I've written a test program which does a sequence of alloc/free calls,
> making sure to memset all memory it allocates and (optionally) doing one
> invalid read (Address xx is 0 bytes after a block of size xx alloc'd)
> for every free. Running this against libc this code is either valgrind
> clean or reports exactly 1 error per free depending on usage which is
> exactly what I'd expect.
>
> When I use this same program to test my own allocator I get *only* get
> 331 errors from despite doing 505 alloc/free calls, I also get the
> following assertion when the program finishes:
>
> Memcheck: mac_leakcheck.c:705 (vgMAC_do_detect_memory_leaks): Assertion
> 'lc_shadows[i]->data + lc_shadows[i]->size < lc_shadows[i+1]->data'
> failed.
I've got a simple patch which fixes this first crash, it appears that my
allocator is packing chunks very tightly, often with no padding/redzone
at all. This patch handles the case where blocks are next to each other
in memory.
I'm still seeing both crashes within valgrind and no error reports where
there should be some but I'm no further forward finding the cause of
this yet. A typical stack trace looks like this with what appears to be
two assertion failures in mac_leakcheck.c, the first one at line 588
"tl_assert(p_min != NULL)" and a second one at 539.
Memcheck: mac_leakcheck.c:539 (full_report): Assertion
'lc_markstack[i].state == IndirectLeak' failed.
==9590== at 0xB000E279: report_and_quit (m_libcassert.c:136)
==9590== by 0xB000E412: vgPlain_assert_fail (m_libcassert.c:199)
==9590== by 0xB000090B: vgMAC_do_detect_memory_leaks
(mac_leakcheck.c:588)
==9590== by 0xB00049A3: mc_detect_memory_leaks (mc_main.c:2076)
==9590== by 0xB0004E19: mc_handle_client_request (mc_main.c:2437)
==9590== by 0xB002D828: do_client_request (scheduler.c:1098)
==9590== by 0xB002CDD7: vgPlain_scheduler (scheduler.c:721)
==9590== by 0xB003D036: thread_wrapper (syswrap-linux.c:86)
==9590== by 0xB003D0FC: run_a_thread_NORETURN (syswrap-linux.c:119)
Ashley,
ashley:valgrind> svn diff memcheck/mac_leakcheck.c
Index: memcheck/mac_leakcheck.c
===================================================================
--- memcheck/mac_leakcheck.c (revision 5296)
+++ memcheck/mac_leakcheck.c (working copy)
@@ -702,7 +702,7 @@
/* Sanity check -- make sure they don't overlap */
for (i = 0; i < lc_n_shadows-1; i++) {
tl_assert( lc_shadows[i]->data + lc_shadows[i]->size
- < lc_shadows[i+1]->data );
+ <= lc_shadows[i+1]->data );
}
if (lc_n_shadows == 0) {
|
|
From: Ashley P. <as...@qu...> - 2005-12-07 16:02:52
|
On Tue, 2005-12-06 at 18:37 +0000, Ashley Pittman wrote:
> I'm still seeing both crashes within valgrind and no error reports where
> there should be some but I'm no further forward finding the cause of
> this yet. A typical stack trace looks like this with what appears to be
> two assertion failures in mac_leakcheck.c, the first one at line 588
> "tl_assert(p_min != NULL)" and a second one at 539.
I gave up trying to work with my library and went back to basics coding
up a reproducer from scratch, can somebody take a look at this for me
please.
With the patch I posted yesterday it crashes with this error:
Memcheck: mac_leakcheck.c:539 (full_report): Assertion
'lc_markstack[i].state == IndirectLeak' failed.
==11809== at 0xB000E279: report_and_quit (m_libcassert.c:136)
==11809== by 0xB000E412: vgPlain_assert_fail (m_libcassert.c:199)
==11809== by 0xB000090B: vgMAC_do_detect_memory_leaks
(mac_leakcheck.c:588)
==11809== by 0xB00049A3: mc_detect_memory_leaks (mc_main.c:2078)
==11809== by 0xB0004E19: mc_handle_client_request (mc_main.c:2439)
==11809== by 0xB002D828: do_client_request (scheduler.c:1098)
==11809== by 0xB002CDD7: vgPlain_scheduler (scheduler.c:721)
==11809== by 0xB003D036: thread_wrapper (syswrap-linux.c:86)
==11809== by 0xB003D0FC: run_a_thread_NORETURN (syswrap-linux.c:119)
Ashley,
#include <stdlib.h>
#include <assert.h>
#include <sys/mman.h>
#include <valgrind/valgrind.h>
#include <valgrind/memcheck.h>
#define SIZE (64*1024)
int main () {
char *mbase = mmap(NULL,SIZE,PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_NORESERVE|MAP_ANONYMOUS, 0, 0);
assert(mbase != (void *)MAP_FAILED);
VALGRIND_MAKE_NOACCESS(mbase,SIZE);
VALGRIND_MALLOCLIKE_BLOCK(mbase+0,64,0,1);
VALGRIND_MALLOCLIKE_BLOCK(mbase+64,64,0,1);
/* Calling either this or using --leak-check=full results in a assert
failure */
VALGRIND_DO_LEAK_CHECK;
exit(0);
}
|
|
From: Jeroen N. W. <jn...@xs...> - 2005-12-17 09:11:38
|
> On Tue, 2005-12-06 at 18:37 +0000, Ashley Pittman wrote:
>> I'm still seeing both crashes within valgrind and no error reports where
>> there should be some but I'm no further forward finding the cause of
>> this yet. A typical stack trace looks like this with what appears to be
>> two assertion failures in mac_leakcheck.c, the first one at line 588
>> "tl_assert(p_min != NULL)" and a second one at 539.
>
> I gave up trying to work with my library and went back to basics coding
> up a reproducer from scratch, can somebody take a look at this for me
> please.
>
> With the patch I posted yesterday it crashes with this error:
>
> Memcheck: mac_leakcheck.c:539 (full_report): Assertion
> 'lc_markstack[i].state == IndirectLeak' failed.
[snip stacktrace]
I've hacked together a patch (for trunk, needing work for COMPVBITS) to
produce leak checks for custom allocated blocks and mempools, as well as
for malloc'd blocks. I basically did this by separating the bookkeeping
for custom blocks from that for malloc'd blocks, and invoking the leak
detector for each type (malloc, custom or mempool). The problem with this
patch is that I cannot get it to work without disabling this
'tl_assert(lc_markstack[i].state == IndirectLeak' in 'static void
full_report(ThreadId tid)'. For custom blocks, the state I get is
Unreached, 'definitely lost'.
Can anybody tell me if the tl_assert() is too critical, or if this is some
other bug? Thanks.
Jeroen.
>
> Ashley,
>
> #include <stdlib.h>
> #include <assert.h>
> #include <sys/mman.h>
>
> #include <valgrind/valgrind.h>
> #include <valgrind/memcheck.h>
>
> #define SIZE (64*1024)
> int main () {
> char *mbase = mmap(NULL,SIZE,PROT_READ|PROT_WRITE, MAP_PRIVATE|
> MAP_NORESERVE|MAP_ANONYMOUS, 0, 0);
> assert(mbase != (void *)MAP_FAILED);
>
> VALGRIND_MAKE_NOACCESS(mbase,SIZE);
>
> VALGRIND_MALLOCLIKE_BLOCK(mbase+0,64,0,1);
> VALGRIND_MALLOCLIKE_BLOCK(mbase+64,64,0,1);
>
> /* Calling either this or using --leak-check=full results in a assert
> failure */
> VALGRIND_DO_LEAK_CHECK;
> exit(0);
> }
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files
> for problems? Stop! Download the new AJAX search engine that makes
> searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
>
|
|
From: Ashley P. <as...@qu...> - 2005-12-22 15:21:37
|
On Sat, 2005-12-17 at 10:11 +0100, Jeroen N. Witmond wrote: > > On Tue, 2005-12-06 at 18:37 +0000, Ashley Pittman wrote: > >> I'm still seeing both crashes within valgrind and no error reports where > >> there should be some but I'm no further forward finding the cause of > >> this yet. A typical stack trace looks like this with what appears to be > >> two assertion failures in mac_leakcheck.c, the first one at line 588 > >> "tl_assert(p_min != NULL)" and a second one at 539. > > > > I gave up trying to work with my library and went back to basics coding > > up a reproducer from scratch, can somebody take a look at this for me > > please. > > > > With the patch I posted yesterday it crashes with this error: > > > > Memcheck: mac_leakcheck.c:539 (full_report): Assertion > > 'lc_markstack[i].state == IndirectLeak' failed. > [snip stacktrace] > > I've hacked together a patch (for trunk, needing work for COMPVBITS) to > produce leak checks for custom allocated blocks and mempools, as well as > for malloc'd blocks. I basically did this by separating the bookkeeping > for custom blocks from that for malloc'd blocks, and invoking the leak > detector for each type (malloc, custom or mempool). The problem with this > patch is that I cannot get it to work without disabling this > 'tl_assert(lc_markstack[i].state == IndirectLeak' in 'static void > full_report(ThreadId tid)'. For custom blocks, the state I get is > Unreached, 'definitely lost'. I'd be interested in seeing that patch. Is your patch so that leaks are reported differently for custom and heap blocks? > Can anybody tell me if the tl_assert() is too critical, or if this is some > other bug? Thanks. Well it's been removed now so it's not *too* critical. I found that when I removed it I got different error totals depending on if I set the --leak-check=full option or not :( I don't really understand why this is but hope to get time to at least look into in the new year. Ashley, |
|
From: Jeroen N. W. <jn...@xs...> - 2005-12-31 09:42:33
|
> On Sat, 2005-12-17 at 10:11 +0100, Jeroen N. Witmond wrote: >> > On Tue, 2005-12-06 at 18:37 +0000, Ashley Pittman wrote: >> >> I'm still seeing both crashes within valgrind and no error reports >> where >> >> there should be some but I'm no further forward finding the cause of >> >> this yet. A typical stack trace looks like this with what appears to >> be >> >> two assertion failures in mac_leakcheck.c, the first one at line 588 >> >> "tl_assert(p_min != NULL)" and a second one at 539. >> > >> > I gave up trying to work with my library and went back to basics >> coding >> > up a reproducer from scratch, can somebody take a look at this for me >> > please. >> > >> > With the patch I posted yesterday it crashes with this error: >> > >> > Memcheck: mac_leakcheck.c:539 (full_report): Assertion >> > 'lc_markstack[i].state == IndirectLeak' failed. >> [snip stacktrace] >> >> I've hacked together a patch (for trunk, needing work for COMPVBITS) to >> produce leak checks for custom allocated blocks and mempools, as well as >> for malloc'd blocks. I basically did this by separating the bookkeeping >> for custom blocks from that for malloc'd blocks, and invoking the leak >> detector for each type (malloc, custom or mempool). The problem with >> this >> patch is that I cannot get it to work without disabling this >> 'tl_assert(lc_markstack[i].state == IndirectLeak' in 'static void >> full_report(ThreadId tid)'. For custom blocks, the state I get is >> Unreached, 'definitely lost'. > > I'd be interested in seeing that patch. Is your patch so that leaks are > reported differently for custom and heap blocks? Yes. Read here: http://www.xs4all.nl/~jnw/valgrind/custom-alloc/custom-alloc.html for more details and the link to download the patch. Happy New Year! Jeroen. > >> Can anybody tell me if the tl_assert() is too critical, or if this is >> some >> other bug? Thanks. > > Well it's been removed now so it's not *too* critical. I found that > when I removed it I got different error totals depending on if I set the > --leak-check=full option or not :( I don't really understand why this > is but hope to get time to at least look into in the new year. > > Ashley, > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Valgrind-developers mailing list > Val...@li... > https://lists.sourceforge.net/lists/listinfo/valgrind-developers > |