iverilog-devel

[Iverilog-devel] Regression suite coverage statistics

[Nick G. <ni...@ni...>]

I've been experimenting with gcov (GCC's code coverage tool) on the VHDL 
code generator to find out what proportion of the code is covered by the 
current test cases.

I've also managed to run the vvp_reg regression suite with 
instrumentation, to get a code coverage analysis for the core compiler 
and the VVP target, which you guys might be interested in.

The aggregated results (formatted with lcov) are here: 
http://www.nickg.me.uk/~nick/vvp_cov/

It appears that the regression suite hits about 75% of the total source 
lines (slightly higher in the vvp and tgt-vvp directories). The results 
also give per-file coverage and a count of how many times each line was 
executed.

If you're interested in playing around with this I can post the makefile 
changes + Perl script I used.


Nick

Re: [Iverilog-devel] Regression suite coverage statistics

[Cary R. <cy...@ya...>]

--- On Wed, 7/30/08, Nick Gasson <ni...@ni...> wrote:

> I've been experimenting with gcov (GCC's code
> coverage tool) on the VHDL 
> code generator to find out what proportion of the code is
> covered by the 
> current test cases.

That's great!

> I've also managed to run the vvp_reg regression suite
> with 
> instrumentation, to get a code coverage analysis for the
> core compiler 
> and the VVP target, which you guys might be interested in.

I have only poked at this a bit, but one interesting thing
is the results for LineInfo.cc. It's claiming the return is
never called, but the rest of the routine is called 217
times. Do you have optimization turned on and that is what
is confusing gcov?

> If you're interested in playing around with this I can
> post the makefile 
> changes + Perl script I used.

Once you get all the kinks worked out both Steve and I should
probably have a copy of the changes or it should be fully
integrated as a special target (make gcov).

Cary


      

Re: [Iverilog-devel] Regression suite coverage statistics

[Nick G. <ni...@ni...>]

Cary R. wrote:
>
> I have only poked at this a bit, but one interesting thing
> is the results for LineInfo.cc. It's claiming the return is
> never called, but the rest of the routine is called 217
> times. Do you have optimization turned on and that is what
> is confusing gcov?
>
>   

Yes, I'd built it with the default -O2. I think you're right about this 
confusing gcov, and the manual suggests the same. I've re-built it with 
-O0 and the results are here: http://www.nickg.me.uk/~nick/vvp_cov_O0/
Interestingly, the total coverage decreased slightly by 0.5%, but 
LineInfo.cc is now 100% covered.

Also, I encountered the "missing vvp_net_t delete operator" problem when 
I tried this with GCC 4.1. However, there were no problems with -O0 on 
either GCC 3.3 or 4.3.

> Once you get all the kinks worked out both Steve and I should
> probably have a copy of the changes or it should be fully
> integrated as a special target (make gcov).
>   

It's currently implemented as a --enable-coverage flag to configure 
which sets the necessary flags and now forces -O0. (Fairly simple: I 
added an AC_CODE_COVERAGE macro to aclocal.m4 which checks the enable 
flag, and call this in any configure.in where we might want to enable 
coverage statistics.)

I've added a `coverage' branch to my github account with two patches 
against the master branch. The regression wrapper script could go in the 
testsuite repository.

Re: [Iverilog-devel] Regression suite coverage statistics

[Cary R. <cy...@ya...>]

--- On Thu, 7/31/08, Nick Gasson <ni...@ni...> wrote:

> Yes, I'd built it with the default -O2. I think
> you're right about this 
> confusing gcov, and the manual suggests the same. I've
> re-built it with 
> -O0 and the results are here:
> http://www.nickg.me.uk/~nick/vvp_cov_O0/
> Interestingly, the total coverage decreased slightly by
> 0.5%, but 
> LineInfo.cc is now 100% covered.

I think this is to be expected. We now have a true count
of the lines being checked. Because of optimizations you
can get lines that are not really checked being marked as
checked since they become intertwined with other lines.
 
> Also, I encountered the "missing vvp_net_t delete
> operator" problem when 
> I tried this with GCC 4.1. However, there were no problems
> with -O0 on 
> either GCC 3.3 or 4.3.

I sent Steve the delete stub I wrote, so he can decide what
he thinks is best. I don't see how the stub would cause any
problems and if it helps under 4.1 we should probably include
something like it.

Cary


      

Re: [Iverilog-devel] Regression suite coverage statistics

[Cary R. <cy...@ya...>]

--- On Thu, 7/31/08, Nick Gasson <ni...@ni...> wrote:

> I've re-built it with 
> -O0 and the results are here:
> http://www.nickg.me.uk/~nick/vvp_cov_O0/
> Interestingly, the total coverage decreased slightly by
> 0.5%, but LineInfo.cc is now 100% covered.

I looked at this briefly and things look mostly as expected
with this information we should be able to write some tests
that check a few of the missing things. We can't check all
the error code since some of the errors should never happen.
Checking stop.cc in the vvp directory is also hard since it
is meant to be used interactively.

Are you also running the vpi tests? They will exercise parts
of vvp that nothing else will. Also do you know why the three
header files (arith.h, bufif.h and npmos.h) are marked as 0%
when the associated cc files are not.

These results may also help use track down some of the memory
leaks (destructors that are never called).

Cary


      

Re: [Iverilog-devel] Regression suite coverage statistics

[Nick G. <ni...@ni...>]

Cary R. wrote:
> Are you also running the vpi tests? They will exercise parts
> of vvp that nothing else will. Also do you know why the three
> header files (arith.h, bufif.h and npmos.h) are marked as 0%
> when the associated cc files are not.
>   

I've just run these now (in addition to the vvp_reg tests, results here: 
http://www.nickg.me.uk/~nick/vvp_vpi_cov/). However, the VPI tests did 
not complete successfully: while iverilog-vpi runs OK, the tests fail 
when executing vvp with this error:

Compiling vpi/hello_poke.c...
Making hello_poke.vpi from  hello_poke.o...
hello_poke:`./hello_poke.vpi' failed to open using dlopen() because:
    ./hello_poke.vpi: undefined symbol: __gcov_merge_add.
$hello_poke: This task not defined by any modules. I cannot compile it.
vsim: Program not runnable, 1 errors.

I'm guessing this is a problem with the flags which are/aren't used to 
compile the VPI module. -fprofile-arcs -ftest-coverage are used to build 
the compiler, and these pull in the __gcov_XXX symbols. I noticed 
there's a "compiler options" field in the vpi_regress.list file. Do any 
additional flags get passed to the compiler?

> Also do you know why the three
> header files (arith.h, bufif.h and npmos.h) are marked as 0%
> when the associated cc files are not.

I'm not really sure. The highlighted lines are the class definitions, 
and from a small sample (arith.h, delay.h) it seems to be classes that 
use the default destructor. In e.g. arith.h the classes that do have 
destructors don't have this unexecuted highlighting. So'd I'd guess it's 
indicating that the default destructor is not being called.


Nick

Re: [Iverilog-devel] Regression suite coverage statistics

[Cary R. <cy...@ya...>]

--- On Fri, 8/1/08, Nick Gasson <ni...@ni...> wrote:

> However, the VPI tests did
> not complete successfully: while iverilog-vpi runs OK, the
> tests fail when executing vvp with this error:
>
> Compiling vpi/hello_poke.c...
> Making hello_poke.vpi from  hello_poke.o...
> hello_poke:`./hello_poke.vpi' failed to open using
> dlopen() because:
>     ./hello_poke.vpi: undefined symbol: __gcov_merge_add.
> $hello_poke: This task not defined by any modules. I cannot
> compile it.
> vsim: Program not runnable, 1 errors.
>
> I'm guessing this is a problem with the flags which
> are/aren't used to
> compile the VPI module. -fprofile-arcs -ftest-coverage are
> used to build
> the compiler, and these pull in the __gcov_XXX symbols. I
> noticed
> there's a "compiler options" field in the
> vpi_regress.list file. Do any
> additional flags get passed to the compiler?

If the -f flags are part of CFLAGS they should get propagated
to the iverilog-vpi script. You could look there and if they
are missing you could add them and see if that fixes the
problem. The compiler options are passed to iverilog-vpi, but
it ignores options it doesn't understand instead of passing
them untouched.

Cary