valgrind-developers

[Valgrind-developers] [PATCH] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
when set skips printing parts of the test messages that may contain data
that differ on subsequent runs (e.g. pids).

With this you can run a test twice under a different conditions and
check if the test codeflow was identical by simply doing diff of the
outputs from the two runs.

Signed-off-by: Cyril Hrubis <ch...@su...>
Suggested-by: Martin Doucha <md...@su...>
CC: val...@li...
---
 lib/tst_test.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/tst_test.c b/lib/tst_test.c
index 2bb4519dd..3f94a1607 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -64,6 +64,7 @@ static int mntpoint_mounted;
 static int ovl_mounted;
 static struct timespec tst_start_time; /* valid only for test pid */
 static int tdebug;
+static int reproducible_output;
 
 struct results {
 	int passed;
@@ -304,6 +305,9 @@ static void print_result(const char *file, const int lineno, int ttype,
 	str += ret;
 	size -= ret;
 
+	if (reproducible_output)
+		goto print;
+
 	if (tst_color_enabled(STDERR_FILENO))
 		ret = snprintf(str, size, "%s%s: %s", tst_ttype2color(ttype),
 			       res, ANSI_COLOR_RESET);
@@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
 				"Next message is too long and truncated:");
 	}
 
+print:
 	snprintf(str, size, "\n");
 
 	/* we might be called from signal handler, so use write() */
@@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
 	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
 		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
 
+	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
+
 	assert_test_fn();
 
 	TCID = tid = get_tid(argv);
-- 
2.49.0

Re: [Valgrind-developers] [PATCH] lib: tst_test: Add reproducible output.

[Martin D. <md...@su...>]

Hi,
one suggestion below.

On 09. 05. 25 10:34, Cyril Hrubis wrote:
> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).
> 
> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.
> 
> Signed-off-by: Cyril Hrubis <ch...@su...>
> Suggested-by: Martin Doucha <md...@su...>
> CC: val...@li...
> ---
>   lib/tst_test.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..3f94a1607 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>   static int ovl_mounted;
>   static struct timespec tst_start_time; /* valid only for test pid */
>   static int tdebug;
> +static int reproducible_output;
>   
>   struct results {
>   	int passed;
> @@ -304,6 +305,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>   	str += ret;
>   	size -= ret;
>   
> +	if (reproducible_output)
> +		goto print;
> +

I'd move this goto one code chunk down (after the color if-else) so that 
we also get the res string in the output. Or at least add a strncpy(str, 
res, size) before the goto if you want to bypass the color logic.

>   	if (tst_color_enabled(STDERR_FILENO))
>   		ret = snprintf(str, size, "%s%s: %s", tst_ttype2color(ttype),
>   			       res, ANSI_COLOR_RESET);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>   				"Next message is too long and truncated:");
>   	}
>   
> +print:
>   	snprintf(str, size, "\n");
>   
>   	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>   	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>   		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
>   
> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
> +
>   	assert_test_fn();
>   
>   	TCID = tid = get_tid(argv);


-- 
Martin Doucha   md...@su...
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Re: [Valgrind-developers] [PATCH] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

Hi!
> >   struct results {
> >   	int passed;
> > @@ -304,6 +305,9 @@ static void print_result(const char *file, const int lineno, int ttype,
> >   	str += ret;
> >   	size -= ret;
> >   
> > +	if (reproducible_output)
> > +		goto print;
> > +
> 
> I'd move this goto one code chunk down (after the color if-else) so that 
> we also get the res string in the output. Or at least add a strncpy(str, 
> res, size) before the goto if you want to bypass the color logic.

The color output is disabled when output is redirected into a file so I
suppose that we can simply move the goto after we print the result. I
will send a v2.

-- 
Cyril Hrubis
ch...@su...

[Valgrind-developers] [PATCH v2] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
when set skips printing parts of the test messages that may contain data
that differ on subsequent runs (e.g. pids).

With this you can run a test twice under a different conditions and
check if the test codeflow was identical by simply doing diff of the
outputs from the two runs.

Signed-off-by: Cyril Hrubis <ch...@su...>
Suggested-by: Martin Doucha <md...@su...>
CC: val...@li...
---
 lib/tst_test.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/tst_test.c b/lib/tst_test.c
index 2bb4519dd..f14627544 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -64,6 +64,7 @@ static int mntpoint_mounted;
 static int ovl_mounted;
 static struct timespec tst_start_time; /* valid only for test pid */
 static int tdebug;
+static int reproducible_output;
 
 struct results {
 	int passed;
@@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
 	str += ret;
 	size -= ret;
 
+	if (reproducible_output)
+		goto print;
+
 	ssize = size - 2;
 	ret = vsnprintf(str, size, fmt, va);
 	str += MIN(ret, ssize);
@@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
 				"Next message is too long and truncated:");
 	}
 
+print:
 	snprintf(str, size, "\n");
 
 	/* we might be called from signal handler, so use write() */
@@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
 	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
 		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
 
+	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
+
 	assert_test_fn();
 
 	TCID = tid = get_tid(argv);
-- 
2.49.0

Re: [Valgrind-developers] [PATCH v2] lib: tst_test: Add reproducible output.

[Martin D. <md...@su...>]

Hi,
Reviewed-by: Martin Doucha <md...@su...>

On 09. 05. 25 11:28, Cyril Hrubis wrote:
> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).
> 
> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.
> 
> Signed-off-by: Cyril Hrubis <ch...@su...>
> Suggested-by: Martin Doucha <md...@su...>
> CC: val...@li...
> ---
>   lib/tst_test.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>   static int ovl_mounted;
>   static struct timespec tst_start_time; /* valid only for test pid */
>   static int tdebug;
> +static int reproducible_output;
>   
>   struct results {
>   	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>   	str += ret;
>   	size -= ret;
>   
> +	if (reproducible_output)
> +		goto print;
> +
>   	ssize = size - 2;
>   	ret = vsnprintf(str, size, fmt, va);
>   	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>   				"Next message is too long and truncated:");
>   	}
>   
> +print:
>   	snprintf(str, size, "\n");
>   
>   	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>   	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>   		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
>   
> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
> +
>   	assert_test_fn();
>   
>   	TCID = tid = get_tid(argv);


-- 
Martin Doucha   md...@su...
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

Re: [Valgrind-developers] [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Avinesh K. <ak...@su...>]

Hi,

Reviewed-by: Avinesh Kumar <ak...@su...>


On Friday, May 9, 2025 11:28:13 AM CEST Cyril Hrubis wrote:
> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).
> 
> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.
> 
> Signed-off-by: Cyril Hrubis <ch...@su...>
> Suggested-by: Martin Doucha <md...@su...>
> CC: val...@li...
> ---
>  lib/tst_test.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>  static int ovl_mounted;
>  static struct timespec tst_start_time; /* valid only for test pid */
>  static int tdebug;
> +static int reproducible_output;
>  
>  struct results {
>  	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>  	str += ret;
>  	size -= ret;
>  
> +	if (reproducible_output)
> +		goto print;
> +
>  	ssize = size - 2;
>  	ret = vsnprintf(str, size, fmt, va);
>  	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>  				"Next message is too long and truncated:");
>  	}
>  
> +print:
>  	snprintf(str, size, "\n");
>  
>  	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>  	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>  		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
>  
> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
> +
>  	assert_test_fn();
>  
>  	TCID = tid = get_tid(argv);
> 

Regards,
Avinesh

Re: [Valgrind-developers] [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Petr V. <pv...@su...>]

Hi Cyril,

> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).

> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.

LGTM, thanks!
Reviewed-by: Petr Vorel <pv...@su...>

Could you please add before the commit the variable to
print_help() and to the docs doc/users/setup_tests.rst?

I suggest the diff below, which also sort the entries (ideally it'd be
separated, but it's before the release, feel free to do it).

> Signed-off-by: Cyril Hrubis <ch...@su...>
> Suggested-by: Martin Doucha <md...@su...>
> CC: val...@li...
> ---
>  lib/tst_test.c | 7 +++++++
>  1 file changed, 7 insertions(+)

> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>  static int ovl_mounted;
>  static struct timespec tst_start_time; /* valid only for test pid */
>  static int tdebug;
> +static int reproducible_output;

>  struct results {
>  	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>  	str += ret;
>  	size -= ret;

> +	if (reproducible_output)
> +		goto print;
> +
>  	ssize = size - 2;
>  	ret = vsnprintf(str, size, fmt, va);
>  	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>  				"Next message is too long and truncated:");
>  	}

> +print:
>  	snprintf(str, size, "\n");

>  	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>  	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>  		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);

> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");

LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
This would be nice to unify, but I'm ok to ignore it as it's before release.

> +
>  	assert_test_fn();

>  	TCID = tid = get_tid(argv);

Adding docs (+ sort variables while at it).

Kind regards,
Petr

+++ doc/users/setup_tests.rst
@@ -49,11 +49,12 @@ users.
    * - LTP_DEV_FS_TYPE
      - Filesystem used for testing (default: ``ext2``).
 
-   * - LTP_TIMEOUT_MUL
-     - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow
-       machines to avoid unexpected timeout). It's mainly for shell API, which
-       does not have LTP_RUNTIME_MUL. In C API it scales the default 30 sec
-       safety margin, probably LTP_RUNTIME_MUL should be used instead.
+   * - LTP_IMA_LOAD_POLICY
+     - Load IMA example policy, see :master:`testcases/kernel/security/integrity/ima/README.md`.
+
+   * - LTP_REPRODUCIBLE_OUTPUT
+     - Dischard the actual content of the messages printed by the test
+       (suitable for a reproducible output).
 
    * - LTP_RUNTIME_MUL
      - Multiplies maximal test iteration runtime. Tests that run for more than a
@@ -61,8 +62,11 @@ users.
        both up and down with this multiplier. This is not yet implemented in the
        shell API.
 
-   * - LTP_IMA_LOAD_POLICY
-     - Load IMA example policy, see :master:`testcases/kernel/security/integrity/ima/README.md`.
+   * - LTP_TIMEOUT_MUL
+     - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow
+       machines to avoid unexpected timeout). It's mainly for shell API, which
+       does not have LTP_RUNTIME_MUL. In C API it scales the default 30 sec
+       safety margin, probably LTP_RUNTIME_MUL should be used instead.
 
    * - LTP_VIRT_OVERRIDE
      - Overrides virtual machine detection in the test library. Setting it to
diff --git lib/tst_test.c lib/tst_test.c
index f14627544f..a014ddc278 100644
--- lib/tst_test.c
+++ lib/tst_test.c
@@ -613,9 +613,10 @@ static void print_help(void)
 	fprintf(stderr, "LTP_COLORIZE_OUTPUT  Force colorized output behaviour (y/1 always, n/0: never)\n");
 	fprintf(stderr, "LTP_DEV              Path to the block device to be used (for .needs_device)\n");
 	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
+	fprintf(stderr, "LTP_REPRODUCIBLE_OUTPUT      Dischard the actual content of the messages printed by the test\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");

Re: [Valgrind-developers] [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

Hi!
> LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
> but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
> This would be nice to unify, but I'm ok to ignore it as it's before release.

We also have LTP_ENABLE_DEBUG that has the same values so let's keep the
API the same. I've send v3 with this and the docs changes.

> > +
> >  	assert_test_fn();
> 
> >  	TCID = tid = get_tid(argv);
> 
> Adding docs (+ sort variables while at it).

I've only added the docs for LTP_REPRODUCIBLE_OUTPUT in v3, the sorting
should go in a separate patch. We also miss at least TST_ENABLE_DEBUG in
the output of -h switch as well.

-- 
Cyril Hrubis
ch...@su...

Re: [Valgrind-developers] [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Martin C. <mc...@re...>]

On  Thu  2025-05-15  12:34 , Cyril Hrubis wrote:
> Hi!
> > LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
> > but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
> > This would be nice to unify, but I'm ok to ignore it as it's before release.
> 
> We also have LTP_ENABLE_DEBUG that has the same values so let's keep the
> API the same. I've send v3 with this and the docs changes.
> 
> > > +
> > >  	assert_test_fn();
> > 
> > >  	TCID = tid = get_tid(argv);
> > 
> > Adding docs (+ sort variables while at it).
> 
> I've only added the docs for LTP_REPRODUCIBLE_OUTPUT in v3, the sorting
> should go in a separate patch. We also miss at least TST_ENABLE_DEBUG in
> the output of -h switch as well.

Thank-you, guys.  I've added a preparation to the valgrind-ltp tester:

https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=036f196ce21ae6cc692de205ab49774638cfb925;hb=5894abc5fa9de30c6b4dde453bff3ac1034aa330#l19

Once the new version of LTP ships, we can try dropping our workarounds.

Cheers,
Martin

[Valgrind-developers] [PATCH v3] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
when set to 1 or y skips printing parts of the test messages that may
contain data that differ on subsequent runs (e.g. pids).

With this you can run a test twice under a different conditions and
check if the test codeflow was identical by simply doing diff of the
outputs from the two runs.

Signed-off-by: Cyril Hrubis <ch...@su...>
Suggested-by: Martin Doucha <md...@su...>
Reviewed-by: Martin Doucha <md...@su...>
Reviewed-by: Avinesh Kumar <ak...@su...>
Reviewed-by: Petr Vorel <pv...@su...>
CC: val...@li...
---
 doc/users/setup_tests.rst |  4 ++++
 lib/tst_test.c            | 33 ++++++++++++++++++++++-----------
 2 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/doc/users/setup_tests.rst b/doc/users/setup_tests.rst
index 2766ed719..2cce85fdf 100644
--- a/doc/users/setup_tests.rst
+++ b/doc/users/setup_tests.rst
@@ -42,6 +42,10 @@ users.
      - Path to the block device to be used. C Language: ``.needs_device = 1``.
        Shell language: ``TST_NEEDS_DEVICE=1``.
 
+   * - LTP_REPRODUCIBLE_OUTPUT
+     - When set to ``1`` or ``y`` discards the actual content of the messages
+       printed by the test (suitable for a reproducible output).
+
    * - LTP_SINGLE_FS_TYPE
      - Testing only - specifies filesystem instead all supported
        (for tests with ``.all_filesystems``).
diff --git a/lib/tst_test.c b/lib/tst_test.c
index 923ecf7be..297c376da 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -63,6 +63,7 @@ static int mntpoint_mounted;
 static int ovl_mounted;
 static struct timespec tst_start_time; /* valid only for test pid */
 static int tdebug;
+static int reproducible_output;
 
 struct results {
 	int passed;
@@ -316,6 +317,9 @@ static void print_result(const char *file, const int lineno, int ttype,
 	str += ret;
 	size -= ret;
 
+	if (reproducible_output)
+		goto print;
+
 	ssize = size - 2;
 	ret = vsnprintf(str, size, fmt, va);
 	str += MIN(ret, ssize);
@@ -333,6 +337,7 @@ static void print_result(const char *file, const int lineno, int ttype,
 				"Next message is too long and truncated:");
 	}
 
+print:
 	snprintf(str, size, "\n");
 
 	/* we might be called from signal handler, so use write() */
@@ -606,17 +611,18 @@ static void print_help(void)
 	/* see doc/users/setup_tests.rst, which lists also shell API variables */
 	fprintf(stderr, "Environment Variables\n");
 	fprintf(stderr, "---------------------\n");
-	fprintf(stderr, "KCONFIG_PATH         Specify kernel config file\n");
-	fprintf(stderr, "KCONFIG_SKIP_CHECK   Skip kernel config check if variable set (not set by default)\n");
-	fprintf(stderr, "LTPROOT              Prefix for installed LTP (default: /opt/ltp)\n");
-	fprintf(stderr, "LTP_COLORIZE_OUTPUT  Force colorized output behaviour (y/1 always, n/0: never)\n");
-	fprintf(stderr, "LTP_DEV              Path to the block device to be used (for .needs_device)\n");
-	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
-	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
-	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
-	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
+	fprintf(stderr, "KCONFIG_PATH            Specify kernel config file\n");
+	fprintf(stderr, "KCONFIG_SKIP_CHECK      Skip kernel config check if variable set (not set by default)\n");
+	fprintf(stderr, "LTPROOT                 Prefix for installed LTP (default: /opt/ltp)\n");
+	fprintf(stderr, "LTP_COLORIZE_OUTPUT     Force colorized output behaviour (y/1 always, n/0: never)\n");
+	fprintf(stderr, "LTP_DEV                 Path to the block device to be used (for .needs_device)\n");
+	fprintf(stderr, "LTP_DEV_FS_TYPE         Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
+	fprintf(stderr, "LTP_REPRODUCIBLE_OUTPUT When set to 1 or y discards the actual content of the messages printed by the test\n");
+	fprintf(stderr, "LTP_SINGLE_FS_TYPE      Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
+	fprintf(stderr, "LTP_TIMEOUT_MUL         Timeout multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL         Runtime multiplier (must be a number >=1)\n");
+	fprintf(stderr, "LTP_VIRT_OVERRIDE       Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
+	fprintf(stderr, "TMPDIR                  Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");
 
 	fprintf(stderr, "Timeout and runtime\n");
@@ -1298,6 +1304,7 @@ static const char *default_fs_type(void)
 static void do_setup(int argc, char *argv[])
 {
 	char *tdebug_env = getenv("LTP_ENABLE_DEBUG");
+	char *reproducible_env = getenv("LTP_REPRODUCIBLE_OUTPUT");
 
 	if (!tst_test)
 		tst_brk(TBROK, "No tests to run");
@@ -1316,6 +1323,10 @@ static void do_setup(int argc, char *argv[])
 	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
 		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
 
+	if (reproducible_env &&
+	    (!strcmp(reproducible_env, "1") || !strcmp(reproducible_env, "y")))
+		reproducible_output = 1;
+
 	assert_test_fn();
 
 	TCID = tid = get_tid(argv);
-- 
2.49.0

Re: [Valgrind-developers] [PATCH v3] lib: tst_test: Add reproducible output.

[Petr V. <pv...@su...>]

Hi Cyril,

> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set to 1 or y skips printing parts of the test messages that may
> contain data that differ on subsequent runs (e.g. pids).

> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.

Thanks for fixing n and 0 and updating the docs.

Tested-by: Petr Vorel <pv...@su...>

Kind regards,
Petr

Re: [Valgrind-developers] [LTP] [PATCH v3] lib: tst_test: Add reproducible output.

[Petr V. <pv...@su...>]

Hi Cyril,

+	fprintf(stderr, "LTP_REPRODUCIBLE_OUTPUT When set to 1 or y discards the actual content of the messages printed by the test\n");

nit: "When set to 1 or y" is slightly verbose to me.  Maybe just "1/y discards
the actual content of the messages printed by the test" ?

But anyway that's very minor, therefore I kept the same approach when adding
LTP_ENABLE_DEBUG in the next patch I'm going to send now.

Kind regards,
Petr

Re: [Valgrind-developers] [LTP] [PATCH v3] lib: tst_test: Add reproducible output.

[Cyril H. <ch...@su...>]

Hi!
> +	fprintf(stderr, "LTP_REPRODUCIBLE_OUTPUT When set to 1 or y discards the actual content of the messages printed by the test\n");
> 
> nit: "When set to 1 or y" is slightly verbose to me.  Maybe just "1/y discards
> the actual content of the messages printed by the test" ?

I've shortened it to "Values 1 or y discard ..." and pushed the patch.

-- 
Cyril Hrubis
ch...@su...