|
From: omo <om...@do...> - 2008-06-16 14:18:04
|
Hello, I've prototyped a simple fault injector for malloc()-like routines, called "Failgrind". The tool can make fail malloc() and its variants which are called from specific functions. the basic idea is similar to the bug: http://bugs.kde.org/show_bug.cgi?id=163266 and failmalloc http://www.nongnu.org/failmalloc/ . This is just a small, proof-of-concept prototype, and there will be a lot of TODOs. Which is better to the patch to massif or current "separate tool" style? What are missing features? How should they be implemented? How to intercept specific functions? Does this type of tool make sense at all? Any comments, advices and ideas are welcome. Thank you in advance. ---- * Availabitity I host the code on my mercurial repository: - http://dodgson.org/omo/failgrind/hg/ snapshot is available at - http://dodgson.org/omo/failgrind/hg/?archive/tip.tar.gz The implemention is done as a separate valgrind tool and small patch to base Makefiles. $ tar xvjf ~/Download/valgrind-3.3.0.tar.bz2 $ tar xvzf ~/Download/failgrind-xxx.tar.gz $ mv failgrind-3336efc98b98 valgrind-3.3.0/failgrind $ cd valgrind-3.3.0 $ patch -p1 < failgrind/3_3_0_root.patch $ ./failgrind/autogen.sh $ ./configure --prefix=`pwd`/inst $ make && make install * Usage example Assume we have following code : ---- /* hello.c */ #include <stdlib.h> #include <stdio.h> void* fine() { return malloc(16); } void* how_are_you() { return fine(); } void* hello() { return malloc(16); } void* bye() { return malloc(16); } int main(int argc, char* argv[]) { void* h = 0; void* b = 0; void* hau = 0; h = hello(); if (!h) { printf("failed to alloc hello\n"); return 1; } hau = how_are_you(); if (!hau) { printf("failed to alloc how_are_you\n"); free(h); return 1; } b = bye(); if (!b) { printf("failed to alloc bye\n"); free(hau); free(h); return 1; } printf("all allocted successfully\n"); free(b); free(hau); free(h); return 0; } ---- Which run successfully on normal setting. $ gcc failgrind/tests/hello.c $ ./a.out all allocated successfully Using Failgrind, we can make it fail. $ ./inst/bin/valgrind -q --tool=failgrind --fail-conds=bye ./a.out failed to alloc bye Note that malloc() on bye() failed. Here are more examples: # didn't match bye() $ ./inst/bin/valgrind -q --tool=failgrind --fail-conds=bye/how_are_you ./a.out all allocated successfully # did match! $ ./inst/bin/valgrind -q --tool=failgrind --fail-conds=fine/how_are_you ./a.out failed to alloc how_are_you # invocation order is not what we expect. $ ./inst/bin/valgrind -q --tool=failgrind --fail-conds=how_are_you/fine ./a.out all allocated successfully * Possible TODOs - Port failmalloc features. - Support other functions to fail. - Add client request to controll the behaviour from unit-testing. - Add simpel memory leak detection. because Valgrind seems not to allow using two tools together, we cannot use Massif when we use Failgrind. -- morita |
|
From: John R.
|
omo wrote: > I've prototyped a simple fault injector for malloc()-like routines, > called "Failgrind". > The tool can make fail malloc() and its variants ... This is good! I would use it more if it were integrated into memcheck, with features activated depending on existence of environment variables. If an environment variable exists, then activate the corresponding feature. I suggest a related feature: FAILGRIND_MALLOC_ACTIVATIONS=min,max which would cause malloc never to fail [on purpose] on the first min-1 calls, nor on any calls after the first max calls. So the possibility of injected failure is restricted to calls numbered in the interval [min, max]. The rationale is that my programs have three phases: startup, choose strategy based on input and environment (including how much space is available), execute strategy. I want to concentrate on one phase at a time. It would be nice to have an optional error message: Injecting fault at malloc #12345; 678 bytes requested at foo (bar.c, line 234) by <<usual traceback...>> for each intentional fault. --- |
|
From: omo <om...@do...> - 2008-06-17 15:31:55
|
Hello John, Thank you for your feedback, It seems OK for me to integrate this one into Memcheck. and concept of "activtion range" also looks fine. I'll try memcheck integration and other change in this weekend, and post the patch to the list if it gets done. On Mon, Jun 16, 2008 at 11:59 PM, John Reiser <jr...@bi...> wrote: > omo wrote: > >> I've prototyped a simple fault injector for malloc()-like routines, >> called "Failgrind". >> The tool can make fail malloc() and its variants ... > > This is good! I would use it more if it were integrated into memcheck, > with features activated depending on existence of environment variables. > If an environment variable exists, then activate the corresponding feature. > > I suggest a related feature: FAILGRIND_MALLOC_ACTIVATIONS=min,max > which would cause malloc never to fail [on purpose] on the first min-1 calls, > nor on any calls after the first max calls. So the possibility of injected > failure is restricted to calls numbered in the interval [min, max]. > The rationale is that my programs have three phases: startup, choose > strategy based on input and environment (including how much space is > available), execute strategy. I want to concentrate on one phase at a time. > > It would be nice to have an optional error message: > Injecting fault at malloc #12345; 678 bytes requested > at foo (bar.c, line 234) > by <<usual traceback...>> > for each intentional fault. > > --- > -- -- morita |
|
From: Florian K. <br...@ac...> - 2008-06-17 16:24:32
|
On Monday 16 June 2008 10:17:54 am omo wrote: > Hello, > > I've prototyped a simple fault injector for malloc()-like routines, > called "Failgrind". > The tool can make fail malloc() and its variants which are called from > specific functions. > the basic idea is similar to the bug: > http://bugs.kde.org/show_bug.cgi?id=163266 > and failmalloc http://www.nongnu.org/failmalloc/ . > > This is just a small, proof-of-concept prototype, and there will be a > lot of TODOs. > Which is better to the patch to massif or current "separate tool" style? > What are missing features? How should they be implemented? > How to intercept specific functions? > Does this type of tool make sense at all? > This would be a very useful tool to have. And you already got a great name for it.. As the bugzilla entry suggests it would be good to not restrict it to just malloc etc. failures. File system failures, temporary network unavailability and other such things that are difficult to test otherwise could be emulated nicely this way. The hard part will be how to control where you want to insert the failure. This could be something as John has suggested. More difficult control mechanisms could include - make function foo in "bar.c", line 100 fail (whenever it is reached) - make some function fail only in a particular calling context - restrict failures to functions in a given DSO and so on. Valgrind provides a way to wrap/replace functions. Described here http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.wrapping This might be useful in case you want to extend the tool beyond handling malloc and friends. Florian |
|
From: omo <om...@do...> - 2008-06-18 14:38:12
|
Hello Florian, Thank you for your advice. > The hard part will be how to control where you want to insert the failure. Certainly. We may keep it simple and declarative to give it from clo. Another, more ambitious idea is to build a DSL to control these conditions. Such an approach may be similar to Dtrace-like aspect orientation. But it will be too long way to go... BTW, funtion wrapping functionaly looks very interesting, if it is available in the framework rather than in the client-side. I'll try it later. -- morita On Wed, Jun 18, 2008 at 1:24 AM, Florian Krohm <br...@ac...> wrote: > On Monday 16 June 2008 10:17:54 am omo wrote: >> Hello, >> >> I've prototyped a simple fault injector for malloc()-like routines, >> called "Failgrind". >> The tool can make fail malloc() and its variants which are called from >> specific functions. >> the basic idea is similar to the bug: >> http://bugs.kde.org/show_bug.cgi?id=163266 >> and failmalloc http://www.nongnu.org/failmalloc/ . >> >> This is just a small, proof-of-concept prototype, and there will be a >> lot of TODOs. >> Which is better to the patch to massif or current "separate tool" style? >> What are missing features? How should they be implemented? >> How to intercept specific functions? >> Does this type of tool make sense at all? >> > > This would be a very useful tool to have. And you already got a great name > for it.. As the bugzilla entry suggests it would be good to not restrict it > to just malloc etc. failures. File system failures, temporary network > unavailability and other such things that are difficult to test otherwise > could be emulated nicely this way. > The hard part will be how to control where you want to insert the failure. > This could be something as John has suggested. More difficult control > mechanisms could include > - make function foo in "bar.c", line 100 fail (whenever it is reached) > - make some function fail only in a particular calling context > - restrict failures to functions in a given DSO > > and so on. > Valgrind provides a way to wrap/replace functions. Described here > http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.wrapping > > This might be useful in case you want to extend the tool beyond handling > malloc and friends. > > Florian > -- -- morita |