Re: [Jxcl-devel] RE: So what's the status of JXCL

[Jim D. <jd...@di...>]

On Sat, 28 Feb 2004, King Dale wrote:

> > > I have some ideas about JXCL.
> > >
> > > One is that there are tools like Jcoverage that instrument the
> > > code separately from testing it and JXCL which instruments it
> > > on the fly. Both have advantages and disadvantages. But
> > > there is no reason why JXCL couldn't do both. In both cases
> > > you have to read a class, instrument it, and produce data in
> > > class file format. That could be called from either a
> > > classloader or from a tool that wrote it to disk.
> >
> > As you describe it, this is a fairly minor change.  All that you
> > have to do is write the byte array containing the instrumented code
> > to disk.
>
> Sure its easy, but you aren't exposing that functionality.

Well, of course I am -- it's a public interface.  However, I think that
you are talking about something else, like perhaps a command line option
that means "oh, and write the instrumented code into a separate directory
tree".

> > > I also want to be able to work in what I think is an absolute
> > > must for any coverage tool, which is the ability to declare
> > > certain code that isn't supposed to be executed so that it is
> > > not included in the total for calculating percentage. But it is
> > > absolutely vital that it also report it as an error if it is
> > > executed. I think it may be possible to implement that in JXCL
> > > even though it is only working from byte code.
> >
> > Can you define exactly what you mean here?  Are you talking about
> > marking certain ranges of source code?  Or would you be content with
> > excluding code at the method level?
>
> I mean marking ranges of source code, not at the method level! I actually
> tried to make the case to Joshua Bloch that the new metadata facility in 1.5
> should include annotating within a method for this very kind of thing, but
> was turned down.
>
> As I understand your code, you compute a control flow graph of the method so

It's not exactly a control flow graph...

> basically you could determine where the end of a block is. If we could
> insert a harmless statement into the code that could be easily identified by
> the instrumentation code then we should be able to mark code as unreachable.

Cheapest thing is a boolean, which is also intuitive.  Call it perhaps
org.jxcl.Counting.z$$zCount.  Then I think that it's easy for JXCL to pick
up
  z$$zCount = false;
to mean turn off counting and
  z$$zCount = true;
to turn it back on.  The compiler, of course, has to be persuaded not to
optimize the statements away or move them around.

> The first idea is simply a static method call on a certain class. So I could
> have:
>
> 	if( param == null ) // Should never happen
> 	{
> 		JXCL.unreachableCode();
> 		throw new NullPointerException();
> 	}
> 	doSomething();
>
> The method does not even have to do anything. It is just a signal to the
> instrumentation code, which simply looks for an invokestatic on that method.
> The tough part is that you have to figure out where the end of the block is
> because none of the rest of the code should be reachable from after that
> point until the end of the block. I think that should be possible from your
> control flow analysis.

What exactly do you mean by "the end of the block"?  If you mean the end
of the method, no problem.  But if you mean it literally (I mean, "as in
C"), as for example
	{ // beginning of the block
		// statements
	} // end of the block
it will probably be difficult, because you have no guarantee as to what
the compiler will generate from this sort of input.

> I also want to make this configurable because you might have different types
> of testing like unit testing, integration testing, system testing, etc.
> Whether this statement is reachable may vary based on that. For instance, in
> unit testing you might be able to pass a null param, but in integration and
> functional testing that should never happen. So it is reachable in unit
> testing but not integration testing.

JXCL allows you to write ANYTHING into the bytecode, or drop anything from
the bytecode.  So the thing to focus on is elegance.

> Under most coverage tools that will actually decrease your code coverage
> totals when in reality it is a good thing that you never pass a bad
> parameter.
>
> So I am thinking of having an overload of that method that takes a parameter
> to say when it is unreachable. Either an int which is a bit mask or a
> string. The assigning of bits or string tokens would be up to the user and
> not dictated by JXCL. So for this scenario we might have:
>
> 	JXCL.unreachableCode( TESTING.Integration | TESTING.System );
>
> or
>
> 	JXCL.unreachableCode( "integration, system" );
>
> There are probably several other ways to do it, but this is what I am
> contemplating. It just has to be something that the instrumentation code can
> easily identify and that is configurable to allow different testing
> scenarios.

I suspect that this would probably better be handled using assignments,
perhaps something like
	org.jxcl.TestRegime.unreachableDuring = TestRegime.SYSTEM;

However, I think that there should be a better way of expressing the basic
idea here.

> Another way I just thought of that also eliminates any dependencies on JXCL
> in the regular code would be allow the user to specify a method name that is
> used during instrumentation to flag unreachable code. The type of testing is
> then controlled by the class name upon which it is invoked. So the above
> could be done as:
>
> 	IntegrationTesting.unreachableCode();
> 	SystemTesting.unreachableCode();
>
> The definition of the IntegrationTesting and SystemTesting are not dictated
> by JXCL. It is just looking for a static method call of a given name with no
> parameters.

Yes.  This is more or less the approach taken by JUnit.

> I think this new idea might be better because it removes the need to figure
> out what the parameter is to the method call. You just have to look for
> certain invoke static calls. I like the idea of not having any dependence on

(Something missing in your text here.)

> All the instrumentation code has to do is include data in about what lines
> are marked unreachable in its output. The work of looking for executions of
> these lines is simply part of the reporting code.
>
> This kind of feature has not been implemented by anybody that I know of and
> to me seem like a basic necessity in code coverage. Without it you will
> never be able to reach 100% coverage. But remember that the reporting has to
> signal an error if one of these lines is executed. This prevents someone

This is a separate behaviour, but easily handled.

> from marking a whole bunch of code as unreachable to inflate the coverage
> numbers. The report should also indicate the percent of code marked
> unreachable so someone can't mark most of the code unreachable and not
> exercise it.

In the current version of the code, line numbering is erratic.  For
example, the reported line numbers sometimes go backwards.  But I believe
this can easily be fixed (famous last words).

> > > But then again I have never actually seen JXCL in action. I've
> > > had to work to even get it to build and run on Windows.
> >
> > The examples don't work?
>
> I haven't gotten them to work. I am trying from Windows, which is obviously
> something you had not tested (see my patch). The issue may be the Ant
> classpath bug you talked about.
>
> You report that you were geting several test failures with Maven. After my

In the current version of the code there is exactly one failure during
the Ant build, in TestFinally.

> patch I only got the ones to do with finally clauses. I started looking at
> those. TestFinally fails because the code generated by the compiler is not
> how the code thinks it is. You expect a try finally to have 3 null handlers
> when it only has two and you get an array bounds exception. Perhaps the code

Don't remember, to be honest, but I think that I get a different error.

> generated for finally has changed from what you expected.

I need to review the code.  As I recall, the vertex has a complex
connector with lessee ...
  edge to first vertex in try block
  one edge for each catch handler
  one edge for the finally handler

I should be looking at this over coffee in the morning.
--
Jim Dixon  jd...@di...   tel +44 117 982 0786  mobile +44 797 373 7881
"Be liberal in what you accept,                                Jon Postel
and conservative in what you send."                               RFC 793
http://jxcl.sourceforge.net                       Java unit test coverage
http://xlattice.sourceforge.net         p2p communications infrastructure