Specification

Specification

Base framework

• INPUT:

  • classpath to built measured source
  • classpath to built generators
  • generator parameters
  • measured method informations

• measure annotated method

  • get data generator -> input data
  • decide the loop count
  • start a new JVM, warm it up, start testing and getting measured data
  • save measured data to disc

• provides library of input generators

Ant script

• parameter file
  • contains repository settings / source code location / classpath

• Description file generation

  • searches in source code / class files for anotated methods
  • contains two main sections
    • per each revision the info about the repo, list of methods with information about paramters and generators
    • the final formulas to evaluate

• Remove already measured functions (optional)

• Testing

  • for each revision checkout and build (there must be some build script)
    • for each function checkout and build of generators
    • deploy the jar to testing machine
    • run the base framework
    • collect data and save

• Results generation

  • evaluate each formula
  • generate all needed results
    • R script
    • XML output of basic statistics
    • simple graph for each function
    • generated HTML websites
    • need to sort results by importance

Eclipse plugin

• Java editor extension
  • add syntax highlighting
    • keywords
    • mark wrong syntax
  • intellisense
    • autocompletion of annotations
    • possibility to choose method to compare with from list of methods
  • jump to annotated method (or annotation of the method?) after click on it in the list of methods
  • add version of the measured method to the annotation
  • generate estimated parameter values
• dialog for simple formula generation
• spl results
  • connection profiles (hudson, ...)
    • multiple sources
    • create and edit dialog
  • download test-data & test-results
  • import local test-data & test-results
    • handled by plugin or copied by user into specified folder
  • listing of spl data
    • filtered dedicated tab by (connection profile / date / revision)
      • test-data - show graph
      • test-results - sort by measure of success / fail
      • spl formula on already measured test-data
        • evaluate or refuse due to lack of data
• performance progress for selected function
  • spl support for evaluating performance difference
    • find K that F1 is K times faster than F2
  • pick revisions (handled by spl)
    • since when (date / revision)
    • how often (date / revision)

Hudson plugin

• periodic launch
  • planning provides Hudson in job configuration
  • Ant script using Base framework to perform measurements
  • get results back to Hudson from testing server to be able to display them
• ensure that testing server has at most one measurement task running at a time
• per project configuration
  • which SCM is used (to be able to get specific revisions)
• display results
  • list of measured annotated methods
  • detail of each measured annotation with graphs of measured methods
  • compare with previous builds
• provide results for Eclipse (anything else in machine-readable format)
  • export all results as one zip package for offline usage
  • allow Eclipse to fetch latest results
    • using FTP

Regression testing

• extend SPL formulas
  • add revision identification to methods and generators

Case study

• find real-world Java project to annotate