We do have internal tests and a build system that allows to check the coverage; for current coverage details, check out the CI generated coverage (unzip, open index.html):
To create the coverage on your system configure with --enable-debug --enable-code-coverage and then after makeexecute nice make check-code-coverage TESTSUITEFLAGS="--jobs=$(($(nproc)+1))" (drop the TESTSUITEFLAGS or adjust by specifying the actual number of cores you want to spend on that).
For more information on the coverage and on TESTSUITEFLAGS see the page [Hacking GnuCOBOL].
The task that needs to be done is to inspect the coverage, then deduce what COBOL code needs to be written (that's sometimes quite obvious because of the comments in the code or because of messages raised for compiler checks, sometimes need a bit more search in the rest of the code - in this case please drop a note because then it is likely useful to also add "missing code comments") and write those.
Ideally (but not necessary) adding them to the testsuite sources directly and create a patch or a text "this should be added to the following file", otherwise just provide the COBOL sources with a note which coverage should be improved.
If you add this to the testsuite on your local system (for questions on this just post here) you can also (re-) run the coverage on your machine and check if the parts missing before are now covered (you may want to explicit note the tests via TESTSUITEFLAGS to only check for the new tests to save computing energy).
Examples:
One option would be to inspect for missing "run" tests - then navigate to libcob; quite easy should be strings.c and intrinsics.c (the comments note which COBOL code triggers those parts in most places), more complex but still possible would be for example fileio.c where we see a bunch of callable system routines that are obviously not under automated tests yet, further down there are (likely) SORT related functions). Sometimes you'd need to search in cobc where those function calls are generated.
Another option would be to inspect mostly the compiler part (cobc), ppparse.def would be very easy (just use the internal constants for conditional compilation in a COBOL program) or codeoptim.c (check where the enum values like COB_POINTER_MANIP are referenced, then write up some come code that should trigger it). Those above are also runtime checks; but we also have "compile only (syntax) checks", for this inspect parser.y and maybe typeck.c.
If you want to give that a try - help is definitely needed and very useful here.
Side note: To be able to distribute those additional tests (= include it in our testsuite), we'd need either a copyright disclaimer or assignment, see the [Copyright reassignment] page for detail on this.
Here is my first modest contribution ..
I've added the following code to the run_extension.at file
AT_SETUP([INSPECTTRAILING])#Shouldn't this be in run_fundamentals?AT_KEYWORDS([extensions])AT_DATA([prog.cob],[identificationdivision.program-id .prog.DATADIVISION .WORKING-STORAGE SECTION.01 W01-STRINGPIC X(20)VALUE '0123456789 '.01 W01-INDEXPIC 9(04)BINARY .PROCEDUREDIVISION . *> MOVE0 TOW01-INDEX.INSPECTW01-STRINGTALLYINGW01-INDEXFORTRAILINGSPACE .IFW01-INDEXNOT =10THENDISPLAY 'Bad Result for Inspect Trailing Case 1'END-IF . *>INSPECTW01-STRINGREPLACINGTRAILINGSPACEBY 'A'.IFW01-STRINGNOT ='0123456789AAAAAAAAAA'THENDISPLAY 'Bad Result for Inspect Trailing Case 2'END-IF . *>MOVE0 TOW01-INDEX.INSPECTW01-STRINGTALLYINGW01-INDEXFORTRAILINGSPACE REPLACINGTRAILING 'A'BY 'B'. *>IF W01-STRINGNOT ='0123456789BBBBBBBBBB'THENDISPLAY 'Bad Result for Inspect Trailing Case 3'END-IF . *>MOVE0 TOW01-INDEX.MOVESPACESTOW01-STRING.INSPECTW01-STRINGTALLYINGW01-INDEXFORTRAILING 'A'. *>IFW01-INDEXNOT =0THENDISPLAY 'Bad Result for Inspect Trailing Case 4'W01-INDEXEND-IF . *> MOVE0 TOW01-INDEX.MOVESPACESTOW01-STRING.INSPECTW01-STRINGTALLYINGW01-INDEXFORTRAILINGSPACES . *>IFW01-INDEXNOT =20THENDISPLAY 'Bad Result for Inspect Trailing Case 5'W01-INDEXEND-IF . *>GOBACK .])AT_CHECK([$COMPILEprog.cob],[0],[],[])AT_CHECK([$COBCRUN_DIRECT./prog],[0],[],[])AT_CLEANUP
It allows to complete the code coverage in the inspect_common_no_replace function ( case type == INSPECT_TRAILING).
Last edit: Simon Sobisch 2022-09-30
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you, that's actually a nice addition!
Checked in already with [r4721]. I haven't checked the code-coverage change but guess you did that :-) .
Please see the [Copyright reassignment] page on necessary papers to allow us to also include more / bigger changes to the testsuite (that page misses the option to use a copyright disclaimer, but as I hope you are able to contribute more [a bunch of or even regular] tests, an assignment would be definitely preferable).
Yes , I 've checked the code coverage :-)
I use this method as i don't know if it's the best one :
I have downloaded the snapshot and unzip it in two different directories :
- gnucobol-code-r4719-branches-gnucobol-3.x
- gnucobol-code-r4719-test
I have run the testsuite with the code coverage in he first directory, which is my reference, and i work in the second and then visually compare the resulting html to the reference files via my web browser
For the copyright i'll will see that this we.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'd use one snapshot, then run the coverage check once, rename the resulting directory, then adjust the testsuite source and re-run - this way there is no need to build twice while still having both coverages to compare.
Thanks again!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for the test. Checked in with modifications to make it more concise with slightly less readability with [r4728].
Aiming for general 100% function coverage in each file seems definitely useful :-)
Doing that for the runtime first will also include line and branch coverage in the compiler, so that's a very reasonable way to tackle that.
Tackling coverage under cobc should mostly be about improving line coverage for syntax checks.
Note: In every case you may have seen LCOV exclusion markers, if you ever stumble over those and thing "that can be tested" - please drop a note, similar for code that seems to be not possible to test; which commonly should only be code that leads to an abort during compile or runtime, or is only called in that context.
... but often often that is possible to test, too, for example by producing too much syntax errors /non-COBOL input files or a runtime error.
Just out of interest: is there already a kind of list of things you want to tackle for coverage next?
I saw you have integrate the nex test program ...
I have leaved the program name to UnstringTest ...but i guess it had to be rename to prog to be integrated to the testsuite ?
Denis
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
:-) Using "prog" for the filename is a way to not need to think about good names; as long as it is compiled to an executable the internal entry name does not matter (to not have that matter for modules is a FR somewhere with a clear solution that is possibly outlined there, it is just one of the not-that-important things currently more or less on hold because too much to do; some of those are also "good first issue" points - so issues people that want to start contributing to the code could have a look at after discussion in the specific issue).
I've found your program id matching the test and left it in as "part of the style used by the test writer".
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I indeed prepare some notes on C code to be cover .. that i not able yet to link to Cobol fonctionnalities .
A question : is is possible to include the NIST testsuite to the code coverage tools. Is it's increase the code coverage we would be able to pick the rights cases, recode them in the gnucobol testsuite ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A question : is is possible to include the NIST testsuite to the code coverage tools. Is it's increase the code coverage we would be able to pick the rights cases, recode them in the gnucobol testsuite ?
Hm, yes, that would be possible. When looking at the Makefile we see that check-code-coverage consist of two steps:
# Use recursive makes in order to ignore errors during checkcheck-code-coverage:
-$(AM_V_at)$(MAKE)$(AM_MAKEFLAGS) -k check
$(AM_V_at)$(MAKE)$(AM_MAKEFLAGS) code-coverage-capture
So we can also do those manually - and extending it in between. I think the following should be what you want to do:
$ make -k check
$ make -k test
$ make code-coverage-capture
In general the data is likely still available, in this case you'd only want to run the NIST part and specify a different output directory. If that works - I did not tested that yet - you can do:
$ make check-code-coverage # run the test and write the normal output
$ make -k test# add the NIST testsuite to the coverage data
$ make code-coverage-capture CODE_COVERAGE_OUTPUT_DIRECTORY=extended-coverage CODE_COVERAGE_OUTPUT_FILE=extended-coverage.info # calculate coverage, now from both, write the output to a separate place
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is it's increase the code coverage we would be able to pick the rights cases, recode them in the gnucobol testsuite ?
That may be harder than it sounds - because the NIST suite is huge and you'd have to first find out where something comes from.
But if the suggested way of "adding" to the coverage data works you can likely also do make test -C cobol85/RW to be able to inspect the coverage increase for a smaller subset (a single module only).
"In the end" the goal is that make test adds zero coverage, so we are quite sure that running it is not as necessary as it is currently; but I guess that will be a long way (if your combined coverage works then you likely know)...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Simon
Another program to improve strings.c code coverage (cobol string statement) (87,7% --> 91%)
Line coverage 87,7% ---> 91%
It remains some code in the cob_inspect_characters to complete.
Hope i will have some time next week ...as i begin a new job ... and i have received all the stuffs that
i ordered to built my DIY NAS :-)
Hi Simon , i have been quiet busy these last weeks.
But i have finished some complementayr cases tests for move statement.
I'll send you on a svn patch format this evening
Denis
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here is a patch taht contains nex tests programs and a correction of one that is already included
With this patch we increase move.c coverage from 67.7 to 72.2
A question ..
How could I insert in some libcob source file, a instruction that prints the current cobol program running ?
IS there a macro or a function that could do the job ?
Regards
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You can compile with -ftrace and run with COB_SET_TRACE=1.
For the internal testsuite that would be make check COBOL_FLAGS=-ftrace + adding export COB_SET_TRACE=1 to test/atlocal - obvioulsy most of the testsuite cases wouzld fail because of the unexpected output, but this can be reduced to only the tracing tests by also doing export COB_TRACE_FILE /some/dir/testtrace.log.
For NIST the procedure you'd need to directly add the flag to COBC.
I guess that possibly answers your question?
If not: from C COB_MODULE_PTR->module_name gives you the current module, with GC3.2's extended frame information you also have section + paragraph names along with the statement in COB_MODULE_PTR->frame_ptr.
Last edit: Simon Sobisch 2023-02-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
All compile without warning with -Wall excepted Move_Basic_P_Pic.cbl
Move_Basic_P_Pic.cbl :
This program contains basic tests with P Picture
I know that the P picture functionalities are not finished but as the NIST tests contains
some programs with P Picture I wrote a small tests
The GC compiler generates warning about truncation but it runs ok.
Run status :
All run ok except MoveTestCobMove.cbl that displays errors due to the bug i mentioned
regarding 'binary-c..' (Tickets #877).
I'm pretty confident about these test programs, but due to the different behavior of
cobol compilers if you could test on MF the following two programs it would be an additional guarantee :
- MoveAlphanumToDisplay.cbl
- MoveEditedToDisplay.cbl
Results of move.c coverage with make check-code-coverage
Hi Simon
I'll have som others tests programs that i will be able to send until end of July for c api move.c functions.
I 'have begun to work on test programs for class clause and swith.
Regarding class clause, the following programs does not compile , i'm not sur that this is a "normal" compilation error.
We do have internal tests and a build system that allows to check the coverage; for current coverage details, check out the CI generated coverage (unzip, open index.html):
To create the coverage on your system configure with
--enable-debug --enable-code-coverageand then aftermakeexecutenice make check-code-coverage TESTSUITEFLAGS="--jobs=$(($(nproc)+1))"(drop theTESTSUITEFLAGSor adjust by specifying the actual number of cores you want to spend on that).For more information on the coverage and on
TESTSUITEFLAGSsee the page [Hacking GnuCOBOL].The task that needs to be done is to inspect the coverage, then deduce what COBOL code needs to be written (that's sometimes quite obvious because of the comments in the code or because of messages raised for compiler checks, sometimes need a bit more search in the rest of the code - in this case please drop a note because then it is likely useful to also add "missing code comments") and write those.
Ideally (but not necessary) adding them to the testsuite sources directly and create a patch or a text "this should be added to the following file", otherwise just provide the COBOL sources with a note which coverage should be improved.
If you add this to the testsuite on your local system (for questions on this just post here) you can also (re-) run the coverage on your machine and check if the parts missing before are now covered (you may want to explicit note the tests via
TESTSUITEFLAGSto only check for the new tests to save computing energy).Examples:
One option would be to inspect for missing "run" tests - then navigate to libcob; quite easy should be strings.c and intrinsics.c (the comments note which COBOL code triggers those parts in most places), more complex but still possible would be for example fileio.c where we see a bunch of callable system routines that are obviously not under automated tests yet, further down there are (likely) SORT related functions). Sometimes you'd need to search in cobc where those function calls are generated.
Another option would be to inspect mostly the compiler part (cobc), ppparse.def would be very easy (just use the internal constants for conditional compilation in a COBOL program) or codeoptim.c (check where the enum values like
COB_POINTER_MANIPare referenced, then write up some come code that should trigger it). Those above are also runtime checks; but we also have "compile only (syntax) checks", for this inspect parser.y and maybe typeck.c.If you want to give that a try - help is definitely needed and very useful here.
Side note: To be able to distribute those additional tests (= include it in our testsuite), we'd need either a copyright disclaimer or assignment, see the [Copyright reassignment] page for detail on this.
Related
Wiki: Copyright reassignment
Wiki: Hacking GnuCOBOL
Last edit: Simon Sobisch 2024-03-03
HI Simon
Here is my first modest contribution ..
I've added the following code to the run_extension.at file
It allows to complete the code coverage in the
inspect_common_no_replacefunction ( casetype == INSPECT_TRAILING).Last edit: Simon Sobisch 2022-09-30
Thank you, that's actually a nice addition!
Checked in already with [r4721]. I haven't checked the code-coverage change but guess you did that :-) .
Please see the [Copyright reassignment] page on necessary papers to allow us to also include more / bigger changes to the testsuite (that page misses the option to use a copyright disclaimer, but as I hope you are able to contribute more [a bunch of or even regular] tests, an assignment would be definitely preferable).
Related
Commit: [r4721]
Wiki: Copyright reassignment
Yes , I 've checked the code coverage :-)
I use this method as i don't know if it's the best one :
I have downloaded the snapshot and unzip it in two different directories :
- gnucobol-code-r4719-branches-gnucobol-3.x
- gnucobol-code-r4719-test
I have run the testsuite with the code coverage in he first directory, which is my reference, and i work in the second and then visually compare the resulting html to the reference files via my web browser
For the copyright i'll will see that this we.
I'd use one snapshot, then run the coverage check once, rename the resulting directory, then adjust the testsuite source and re-run - this way there is no need to build twice while still having both coverages to compare.
Thanks again!
HI Simon
Please find in attached another program that allows to increase the strings.c test code coverage.
On the r4722-branches-gnucobol-3.x snapshot :
Strings.c :
Line coverage 85,6 % ---> 87,7%
Functions coverage 96,4%---> 100 %
Denis.
PS : i don't forget what you wrote me about copyright. I just had a glance ... and have to read it better in order to understand what i have to do :-)
Thank you for the test. Checked in with modifications to make it more concise with slightly less readability with [r4728].
Aiming for general 100% function coverage in each file seems definitely useful :-)
Doing that for the runtime first will also include line and branch coverage in the compiler, so that's a very reasonable way to tackle that.
Tackling coverage under cobc should mostly be about improving line coverage for syntax checks.
Note: In every case you may have seen LCOV exclusion markers, if you ever stumble over those and thing "that can be tested" - please drop a note, similar for code that seems to be not possible to test; which commonly should only be code that leads to an abort during compile or runtime, or is only called in that context.
... but often often that is possible to test, too, for example by producing too much syntax errors /non-COBOL input files or a runtime error.
Just out of interest: is there already a kind of list of things you want to tackle for coverage next?
Related
Commit: [r4728]
Hi Simon
I saw you have integrate the nex test program ...
I have leaved the program name to UnstringTest ...but i guess it had to be rename to prog to be integrated to the testsuite ?
Denis
:-) Using "prog" for the filename is a way to not need to think about good names; as long as it is compiled to an executable the internal entry name does not matter (to not have that matter for modules is a FR somewhere with a clear solution that is possibly outlined there, it is just one of the not-that-important things currently more or less on hold because too much to do; some of those are also "good first issue" points - so issues people that want to start contributing to the code could have a look at after discussion in the specific issue).
I've found your program id matching the test and left it in as "part of the style used by the test writer".
I indeed prepare some notes on C code to be cover .. that i not able yet to link to Cobol fonctionnalities .
A question : is is possible to include the NIST testsuite to the code coverage tools. Is it's increase the code coverage we would be able to pick the rights cases, recode them in the gnucobol testsuite ?
Hm, yes, that would be possible. When looking at the Makefile we see that
check-code-coverageconsist of two steps:So we can also do those manually - and extending it in between. I think the following should be what you want to do:
In general the data is likely still available, in this case you'd only want to run the NIST part and specify a different output directory. If that works - I did not tested that yet - you can do:
That may be harder than it sounds - because the NIST suite is huge and you'd have to first find out where something comes from.
But if the suggested way of "adding" to the coverage data works you can likely also do
make test -C cobol85/RWto be able to inspect the coverage increase for a smaller subset (a single module only)."In the end" the goal is that
make testadds zero coverage, so we are quite sure that running it is not as necessary as it is currently; but I guess that will be a long way (if your combined coverage works then you likely know)...Ok , i'll try :-)
Simon
Another program to improve strings.c code coverage (cobol string statement) (87,7% --> 91%)
Line coverage 87,7% ---> 91%
It remains some code in the cob_inspect_characters to complete.
Hope i will have some time next week ...as i begin a new job ... and i have received all the stuffs that
i ordered to built my DIY NAS :-)
Regards
@dhugonnard Do we still have some of those missing and/or there are new tests from your side to add?
Hi Simon , i have been quiet busy these last weeks.
But i have finished some complementayr cases tests for move statement.
I'll send you on a svn patch format this evening
Denis
Hi Simon
Here is a patch taht contains nex tests programs and a correction of one that is already included
With this patch we increase move.c coverage from 67.7 to 72.2
Denis
Hi Simon
Did you received my previous poist with new programs. ?
Denis.
Regards
Yes, thank you. I want to have a look at them in about 8 days (I'm not near a computer over the next week).
Hi Simon
A question ..
How could I insert in some libcob source file, a instruction that prints the current cobol program running ?
IS there a macro or a function that could do the job ?
Regards
You can compile with
-ftraceand run withCOB_SET_TRACE=1.For the internal testsuite that would be
make check COBOL_FLAGS=-ftrace+ addingexport COB_SET_TRACE=1to test/atlocal - obvioulsy most of the testsuite cases wouzld fail because of the unexpected output, but this can be reduced to only the tracing tests by also doingexport COB_TRACE_FILE /some/dir/testtrace.log.For NIST the procedure you'd need to directly add the flag to
COBC.I guess that possibly answers your question?
If not: from C
COB_MODULE_PTR->module_namegives you the current module, with GC3.2's extended frame information you also have section + paragraph names along with the statement inCOB_MODULE_PTR->frame_ptr.Last edit: Simon Sobisch 2023-02-27
Yes it's ok with COB_MODULE_PTR->module_name
Thx
Hi Simon
Please find here the following programs to enhanced the MOVE statement tests
(Some i have already sent to you)
MoveAlphanumToDisplay.cbl
Move_Basic_P_Pic.cbl
MoveDeEditing_2.cbl
MoveDeEditing_3.cbl
MoveDeEditing_4.cbl
MoveDeEditing_5.cbl
MoveDeEditing.cbl
MoveEditedToDisplay.cbl
MoveJustified.cbl
MoveOtherCases.cbl
MoveTestCobMove.cbl
MoveToEditedZero.cbl
Compilation status :
All compile without warning with -Wall excepted Move_Basic_P_Pic.cbl
Move_Basic_P_Pic.cbl :
This program contains basic tests with P Picture
I know that the P picture functionalities are not finished but as the NIST tests contains
some programs with P Picture I wrote a small tests
The GC compiler generates warning about truncation but it runs ok.
Run status :
All run ok except MoveTestCobMove.cbl that displays errors due to the bug i mentioned
regarding 'binary-c..' (Tickets #877).
I'm pretty confident about these test programs, but due to the different behavior of
cobol compilers if you could test on MF the following two programs it would be an additional guarantee :
- MoveAlphanumToDisplay.cbl
- MoveEditedToDisplay.cbl
Results of move.c coverage with make check-code-coverage
So, there are still some few cases to add that I will look into, but most of
the remaining code to test in move.c is the c api "convenient function"
Denis
As requested I gave it a try with MF:
MoveAlphanumToDisplay:
runtime error with MF on line 22:
This also happens without the "+" (I'd consider that bad and if it works with GC consider it superior here).
similar for MoveEditedToDisplay, same runtime error message, but this is a testcase problem, the testcase should use:
not sure what it should test the value against, for MF it contains numeric 12345.0, data (which you'd get to by using
(1:)) contains "00000123450000".Hi Simon
I'll have som others tests programs that i will be able to send until end of July for c api move.c functions.
I 'have begun to work on test programs for class clause and swith.
Regarding class clause, the following programs does not compile , i'm not sur that this is a "normal" compilation error.
PS
Regarding the previous batch of programs i sent , none of them have been integrated, does they have to be modified ?
Denis