Re: [libseccomp-discuss] [PATCH v2] tests: Add regression test automation support

[Paul M. <pm...@re...>]

On Tuesday, March 27, 2012 10:21:51 AM Corey Bryant wrote:
> On 03/26/2012 06:19 PM, Paul Moore wrote:
> > On Friday, March 23, 2012 05:32:42 PM Corey Bryant wrote:

...

> >> +#
> >> +# Run the specified bpf-sim batch test
> >> +#
> >> +# Arguments:
> >> +#     1    line number of test from batch file
> > 
> > I think you mean the test number, yes?
> 
> Yes, thanks, I meant the test number.  In other words first test in a
> batch file is 1, second test is 2, etc.

Okay, good.  Just wanted to make sure I was understanding things correctly.

> >> +#     2    string containing line of test data from batch file
> >> +#
> >> +function run_test_bpf_sim() {
> >> +	#Split up the test data from the line
> >> +	line=($2)
> >> +	testname=${line[0]}
> >> +	syscall=${line[1]}
> >> +	arg0=${line[2]}
> >> +	arg1=${line[3]}
> >> +	arg2=${line[4]}
> >> +	arg3=${line[5]}
> >> +	arg4=${line[6]}
> >> +	arg5=${line[7]}
> >> +	result=${line[8]}
> >> +
> >> +	# Convert the syscall to a number if it was specified as a name
> >> +	if [[ ! $syscall =~ ^[0-9]+$ ]]; then
> >> +		number=`../tools/sys_resolver $syscall`
> > 
> > I'm wondering if it makes sense to create a function which performs the
> > sys_resolver task so that we aren't dependent on having the sys_resolver
> > script in a specific location on the filesystem.  I'm not certain I care
> > too much either way right now, just mentioning it.
> 
> I can see the desire.  The problem I see is that we're still dependent
> on /tools/bpf_sim even if the /tools/sys_resolver code was embedded in a
> function in this script.

Good point, in this case I'd leave it as-is.

> >> +#
> >> +# Run a single test from the specified batch
> >> +#
> >> +# Arguments:
> >> +#     1    line number of test from batch file
> >> +#     2    string containing line of test data from batch file
> >> +#     3    string containing batch that this test belongs to
> >> +#
> >> +function run_test() {
> >> +	if $verbose; then
> >> +		printf "Test %03d data:    %s\n" $1 "$2">&$logfd
> >> +	fi
> >> +
> >> +	if [[ "$3" == "bpf-sim" ]]; then
> > 
> > I'm still a little fuzzy on how this is intended to be expanded over time.
> >  It appears that you can only use test files named "bpf-sim.tests" unless
> > you add a new entry here?  How about adding some sort of identifier
> > inside the test file indicating the type of test file, in this case
> > 'bpf-sim', this way we could run arbitrary batch files while still
> > keeping the ability to have multiple test types/formats?
> > 
> > Or am I missing something?
> 
> I think you're understanding how it works.  The only difference is that
> the identifier you'd like to see added is currently just built into the
> file name.  Another option, instead of adding the identifier in the
> batch file, is to keep it built into the file name as the prefix.  So
> you could have multiple batch files that use the bpf-sim format, they
> just have to be named bpf-sim-something.tests.

Okay, I could go along with the latter approach, but what if we used the 
identifier as the file extension, e.g. batch-foo.bpf-sim-tests?

-- 
paul moore
security and virtualization @ redhat