Emma reporting on main() routine

[Nobody/Anonymous]

I just switched from another code coverage tool to Emma and have a problem with Emma.

I've got a junit test that calls my main() routine, and I've got some System.err.println() commands that are printing out, but Emma claims that I've got 0% code coverage on that method.

Any ideas why this would be?

[Vlad Roubtsov]

Without seeing the code, I don't know off the top of my head why this could happen. Is it possible that your main() is a single basic block and it throws an exception when the test executes?

[Nobody/Anonymous]

My main routine in my NetTranslate class looks like:

    /**
     * The main routine for NetTranslate.
     *
     * @param args the command line arguments
     * @throws FileNotFoundException thrown if the input file is not found.
     * @throws IOException thrown if the output file can not be created or
     * flushed or if either file can not be closed.
     **/
    public static void main (String [] args)
        throws FileNotFoundException,
               IOException
    {
        System.err.println("!!!!####!!!!####!!!!####");

        NetTranslate n = new NetTranslate();
        System.err.println("####!!!!####!!!!####!!!!####");

        if (! n.process (args))
        {
            System.exit(1);
        }
        System.err.println("!!!!####!!!!####!!!!####!!!!####");
    }

My testMain junit test looks like:

    /** Test of main method, of class NetTranslate. */
    public void testMain ()
    {
        boolean exception = false;
        String [] args = {"-i", "DummyFileName",
                          "-n", "Net",
                          "-o", "DummyFileName"};

        // This should throw an exception due to the input file not existing.
        System.err.println("!!!###!!!###!!!###!!!###!!!###");
        try
        {
            System.err.println("###!!!###!!!###!!!###!!!###!!!###");
            NetTranslate.main(args);
            System.err.println("!!!###!!!###!!!###!!!###!!!###!!!###");
        }
        catch (Exception ex)
        {
            exception = true;
        }
       
        if (! exception)
        {
            fail ("NetTranslate.main did not throw an Exception.");
        }
    }

The NetTranslate.main throws a FileNotFoundException like it should.  The System.err.println calls before that point all print out like they should so I know it is running this test.  JUnit shows it as a successful test. 

And yet emma reports my main method as not having been hit at all.

I do have other methods in NetTranslate and test methods for them and those are all showing as tested in emma.

My ant task for doing the junit tests with emma watching things looks like:

    <!-- Run Unit Tests and check Code Coverage on this build -->
    <target name="CodeCoverage"
            depends=""
            description="Runs the Unit tests for Code Coverage.">

        <echo>
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ~~~~~~~~~~ Running CodeCoverage
        </echo>

        <path id="emma.lib">
            <pathelement location="${emma.root}\emma.jar" />
            <pathelement location="${emma.root}\emma_ant.jar" />
        </path>

        <taskdef resource="emma_ant.properties"
                 classpathref="emma.lib" />

        <property name="emma.instr.dir"
                  value="${build.dir}\instr" />
        <delete dir="${emma.instr.dir}" />
        <mkdir dir="${emma.instr.dir}" />

        <!-- Instrument the classes to check coverage on. -->
        <emma>
            <instr mode="fullcopy"
                   outdir="${emma.instr.dir}"
                   metadatafile="${rpt.dir}/metadata.emma"
                   merge="false">
                <instrpath>
                    <fileset dir="${lib.root}">
                        <include name="${project.name}.jar" />
                    </fileset>
                </instrpath>
            </instr>
        </emma>

        <!-- Run the instrumented unit tests. -->
        <junit printsummary="yes">
            <jvmarg value="-Demma.coverage.out.file=${rpt.dir}\Coverage.emma" />
            <jvmarg value="-Demma.coverage.out.merge=true" />
            <classpath>
                <pathelement location="${build.dir}" />
                <pathelement location="${junit.root}\junit.jar" />
                <pathelement location="${emma.instr.dir}\lib\${project.name}.jar" />
                <pathelement location="${emma.root}\emma.jar" />
            </classpath>
            <formatter type="xml" />
            <batchtest fork="yes"
                       todir="${rpt.dir}">
                <fileset dir="${build.dir}">
                    <include name="com\whoever\nettranslate\NetTranslateSuite.class" />
                </fileset>
            </batchtest>
        </junit>

        <!-- Generate the Unit Test report. -->
        <junitreport todir="${rpt.dir}">
            <fileset dir="${rpt.dir}">
                <include name="TEST-*.xml" />
            </fileset>
            <report format="noframes"
                    todir="${rpt.dir}" />
        </junitreport>

        <!-- Generate the Coverage report. -->
        <emma>
            <report sourcepath="${source.root}\src"
                    sort="+block,+name,+method,+class"
                    metrics="method:100,block:100,line:100,class:100">
                <infileset dir="${rpt.dir}"
                           includes="*.emma" />
                <html outfile="${rpt.dir}/Coverage.html"
                      depth="method"
                      columns="name,class,method,block,line" />
            </report>
        </emma>

        <!-- ZIP all of the coverage report files into a new ZIP file. -->
        <zip basedir="${rpt.dir}"
             destfile="${coverage.zip.filename}"
             includes="**/*.html"
             excludes="**/junit-noframes.html" />

        <!-- Copy the reports to where BuildMonitor will find them. -->
        <copy todir="${rpt.root}">
            <fileset dir="${rpt.dir}">
                <include name="junit-noframes.html" />
                <include name="${coverage.zip.filename}" />
            </fileset>
        </copy>

        <tstamp prefix="CodeCoverage">
            <format property="time"
                    pattern="MM/d/yyyy hh:mm:ss.SSS" />
        </tstamp>
        <echo>
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ~~~~~~~~~~ Finished running CodeCoverage (${CodeCoverage.time})
        </echo>
    </target>

[Nobody/Anonymous]

The key seems to be that because I'm throwing an exception from main, emma isn't counting main as having been hit even though it was.

This seems like a bug to me.  I'm specifically testing an exception situation and emma isn't counting it in the code coverage.

How would I bring this to the attention of the emma developers?

[Vlad Roubtsov]

EMMA considers a method covered if its first basic block is covered. EMMA considers a basic block covered if it was executed in its entirety, from the very first to the very last opcode.

If an instruction inside a basic block throws an exception, the basic block will be considered as not having executed at all.

(http://emma.sourceforge.net/faq.html)

This design encourages writing code which does not throw during normal execution. I would argue that this makes sense, because most business logic does what it's supposed to do without throwing exceptions. Exceptions are for handling abnormal situations.

In your case, instead of worrying about counting main() as covered when it threw and basically didn't do anything, why not add another test to your test suite where main() actually completes successfully? That would seem like a better test suite to begin with and will eventually mark main() as covered.

It seems to me that EMMA is pointing you in the right direction here (add a better test). If you disagree, would you suggest a different design? If so, please elaborate. For example, when would you count a Java method as having been covered?

[Nobody/Anonymous]

My main class was pretty simple because I wanted to control how the main logic exited in a simple way.  Most of my junit tests act on the process method which is where all the logic is.

So when it came time to do a junit test on the main method, I figured the only thing left to test was what main would do when it threw an exception for a missing file.  So that was the only test I setup.  This is a valid junit test, it just doesn't let EMMA know that main has been tested.

I have since added a test for everything working.  Basically repeating a test I'm doing on the process method.  So now I'm testing the main method both ways which is probably better anyways.

Thanks for your explanation of how EMMA is working and especially the why part.  I really appreciate it.