Re: [Check-devel] Proposed way to override looping of test functions from environment...

[Branden A. <b.m...@gm...>]

Thank you for your interest in Check.

Although I did not develop the tcase_add_loop_test
<http://check.sourceforge.net/doc/doxygen/html/check_8h.html#aef9249759cf9efe4fcb432630a7c356c>
call
in Check, my understanding is that the expected use case would be testing
for functional correctness. Namely, the response to some code to various
index values is as expected. I would caution against using Check for
verifying that code is free from threading issues. From my experience a
piece of code involving multiple threads can work correctly on one machine
an innumerable number of times, but may quickly fail on another machine. It
is likely that a unit test would not be able to give high confidence that a
multhreaded program is correct.

For tracking down threading issues, I may suggest taking a look at the
hellgrind <http://valgrind.org/docs/manual/hg-manual.html> tool.

As for the suggested patch to Check, I thank you for the suggestion but am
uncomfortable with the idea. One may expect that a unit testing program
which utilizes tcase_add_loop_test
<http://check.sourceforge.net/doc/doxygen/html/check_8h.html#aef9249759cf9efe4fcb432630a7c356c>
may
set various start and end indices for tests where forcing a value using an
environment variable may lead to test failures. If you are interesting in
using Check to track down threading issues, the environment variable
technique does not seem so bad, or alternatively one could pass an argument
to main().

- Branden

On Fri, Apr 24, 2015 at 6:46 AM, Sebastian Rasmussen <
seb...@ax...> wrote:

> Hi!
>
> I have in several instances been trying to stress test cases that have
> been having threading problems. The approach I usually take is to do
> something along the lines of:
>
> int times = getenv("TIMES") ? atoi(getenv("TIMES")) : 1;
> [...]
> test_case_add_loop (tc_core, test_function, 0, times);
>
> The benefit to simply hardcode a value instead of the times variable is
> that under normal conditions I don't have to wait for the test to run
> 1000 times, but only when trying to debug the test/the code under test.
> This is why I prefer using an environment variable.
>
> Now, since I need to add this the snippet above to all tests when
> debugging them that seems a bit cumbersome. Therefore I'd like to have
> something similar introduced in libcheck itself.
>
> I have attempted to alter _tacse_add_test() a bit so that you can
> override values of loop_start and loop_end by setting the environment
> variable CK_LOOP_COUNT. My proposed approach is below. I refrained from
> implementing any test until you guys approve of the idea. Also I had a
> hard time understanding exactly how to add a test to the libcheck test
> suite to test a change like this.
>
> Please let me know what you think.
>
>   / Sebastian
>
>  From e621cc85b2bc38499b9d449a0b275835b20e2bff Mon Sep 17 00:00:00 2001
> From: Sebastian Rasmussen <se...@ax...>
> Date: Fri, 24 Apr 2015 12:30:46 +0200
> Subject: [PATCH] Allow environment to override loop iterations
>
> By setting CK_LOOP_COUNT (and maybe CK_RUN_*) it is now possible
> to alter the number of iterations each test function is run. This
> may be useful for test cases that need to be stressed.
> ---
>   src/check.c | 15 +++++++++++++++
>   1 file changed, 15 insertions(+)
>
> diff --git a/src/check.c b/src/check.c
> index d288e6d..3227596 100644
> --- a/src/check.c
> +++ b/src/check.c
> @@ -186,8 +186,23 @@ void _tcase_add_test(TCase * tc, TFun fn, const
> char *name, int _signal,
>           return;
>       tf = (TF *)emalloc(sizeof(TF));   /* freed in tcase_free */
>       tf->fn = fn;
> +
>       tf->loop_start = start;
>       tf->loop_end = end;
> +
> +    env = getenv("CK_LOOP_COUNT");
> +    if(env != NULL)
> +    {
> +        char *endptr = NULL;
> +        long tmp = strtol(env, &endptr, 10);
> +
> +        if (tmp >= 0 && endptr != env && (*endptr) == '\0')
> +        {
> +            tf->loop_start = 0;
> +            tf->loop_end = tmp;
> +        }
> +    }
> +
>       tf->signal = _signal;       /* 0 means no signal expected */
>       tf->allowed_exit_value = (WEXITSTATUS_MASK & allowed_exit_value);
>    /* 0 is default successful exit */
>       tf->name = name;
> --
> 2.1.4
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Check-devel mailing list
> Che...@li...
> https://lists.sourceforge.net/lists/listinfo/check-devel
>