#31 Students can access grader test cases

Don Blaheta

Any student can see many of the reference test cases simply by submitting Java classes that omit the tested method (or misdeclare it)---this causes a compiler error and Web-CAT obligingly prints out pages of error messages, including every assertion line that doesn't compile.

To reproduce,


  • Don Blaheta

    Don Blaheta - 2010-01-21

    File that comments out one of the required methods

  • Don Blaheta

    Don Blaheta - 2010-01-21

    Ok, so much for just attaching a file; it submitted the whole bug instead. Anyway, the attachments represent a file with omitted method and a JUnit test suitable for dropping into JavaTDD to see the pages of error messages.

  • Don Blaheta

    Don Blaheta - 2010-01-21

    JUnit test cases that are revealed to students wiht compiler errors.

  • Don Blaheta

    Don Blaheta - 2010-01-21
    • priority: 5 --> 6
  • Stephen H. Edwards

    Good point. However, this is more of a limitation than a bug. First I'll describe the root problem, and then present two workarounds.

    First, the problem is that students may (intentionally or unintentionally), write code with signatures that do not conform to the assignment. In such a case, reference tests that directly refer to the method(s) (or even classes) in question just won't compile. That also means the reference tests won't run, of course, and the student will end up with a zero.

    But the real problem is: how do you communicate to the student what is keeping their work from being assessed? As you point out, the JavaTddPlugin presents the compiler messages that result, together with the single line of source code that resulted in each compiler error. Really, the plug-in has no other information about what might be wrong, and this is all it can really show to the student (other than an extremely unhelpful "your code is wrong and you get a zero, go figure it out yourself"). There are two serious drawbacks to this approach: (1) intro students often don't really know how to interpret raw compiler errors shown completely out of context, especially if they are learning using an IDE like BlueJ or Eclipse, where they really don't have to deal directly with command line errors much; and (2) this allows the potential for leakage of possibly confidential information from the reference tests, via the single line of source code quoted for each error. You point out issue (2). But the real problem is: what should be shown to the student in this situation? In my opinion, showing nothing, or showing just the compiler errors taken completely out of context, are both insufficient to allow students to recover from their errors (assuming they aren't doing this intentionally, which is usually the case).

    So now, on to the workarounds. There are two ways to deal with this situation that we've used heavily in the past. The first (and somewhat less sophisticated) approach is just to write reference tests so that lines that directly refer to methods from the class under test do not contain any sensitive information. For example, consider a hypothetical student assignment that determines whether a string is a palindrome (OK, I can't always think of good examples on the fly ;-)). You might write a test case
    this way:


    You care completely correct that, if isPalindrome() is misdeclared, showing the compiler errors will completely give away the exact reference tests being performed. However, you could also write the tests this way:

    String word = "racecar";
    boolean result = obj.isPalindrome(word);
    word = "tower";
    result = obj.isPalindrome(word);

    Notice that local variables have been used to (a) separate literal values out of the method calss, and also (b) separate method results from assertion statements. Now, you can show the full text of each compiler error to students and no "information" is leaked from the test case beyond the minimum information necessary to understand the context of the compiler error.

    That workaround works, but it is cumbersome. I instead typically use a second approach that is really aimed at another problem, but happens to work around this one as well. The other problem is that, when reference tests fail to compile, they don't run. So students get no credit, even if some of the tests would have passed. In other words, messing up the signature of just one method prevents all tests from running. We have a nice little reflection support class built into the JavaTddPlugin (and available in the Web-CAT support jar you can download here). It allows you to fairly easily write your reference tests so that they use pure reflection, rather than directly referring to class features. The net result is that (a) reference tests *always* compile; (b) students can get partial credit for what does work, even if some methods are not declared correctly; and (c) specific diagnostic hints that are more targeted than raw compiler errors are generated back to the student from test case failures (i.e., "method xyz() was not declared public" or "method abc() should be a void method", or "method def() did not produce a result of type Foo", or "method ghi() cannot be called with arguments of type (String, String): incorrect parameter type(s)", etc.).

    The assertions written above would look like this:

    // In setUp():
    Object obj = create("PalindromeChecker");

    // In test cases:
    assertTrue(invoke(obj, Boolean.class, "isPalindrome", "word"));
    assertFalse(invoke(obj, Boolean.class, "isPalindrome", "tower"));

    // To call a void method:

    invoke(obj, "someMethod", param1, param2, param3);

    The reflection library heavily uses both generics (for type safety, and so no type casts are needed) and variable argument lists to give you a fairly simple way of calling constructors using create(...) or call methods using invoke(...).

    So, for now, these are the best workarounds. In the long term, we hope to build a custom class loader that can take plain old reference tests (probably compiled against a reference solution) and dynamically rewrite them using the reflection support calls. This will allow instructors to write "plain old reference tests", but have them run as fully reflective test cases, so the issue of compilation errors never comes up at all, and instructors get the full benefits of the second workaround described above without having to do anything extra. I've lowered the priority of the feature request, however, since we've got a number of more pressing issues to work on before we can tackle this one. In the mean time, please consider one of the two workarounds described here.

  • Stephen H. Edwards

    • labels: --> Grading, Java
    • milestone: --> Java_TDD_grading_plug-in
    • priority: 6 --> 3
    • assigned_to: nobody --> stedwar2
    • status: open --> open-accepted

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks