Specification

Frantisek Haas Vojtech Horky Martin Lacina Jaroslav Kotrc Jiří Daniel
Attachments
spec.pdf (125457 bytes)

Specification

Base framework

  • INPUT:

    • classpath to built measured source
    • classpath to built generators
    • generator parameters
    • measured method informations
  • measure annotated method

    • get data generator -> input data
    • decide the loop count
    • start a new JVM, warm it up, start testing and getting measured data
    • save measured data to disc
  • provides library of input generators

Ant script

  • parameter file
    • contains repository settings / source code location / classpath
  • Description file generation

    • searches in source code / class files for anotated methods
    • contains two main sections
      • per each revision the info about the repo, list of methods with information about paramters and generators
      • the final formulas to evaluate
  • Remove already measured functions (optional)

  • Testing

    • for each revision checkout and build (there must be some build script)
      • for each function checkout and build of generators
      • deploy the jar to testing machine
      • run the base framework
      • collect data and save
  • Results generation

    • evaluate each formula
    • generate all needed results
      • R script
      • XML output of basic statistics
      • simple graph for each function
      • generated HTML websites
      • need to sort results by importance

Eclipse plugin

  • Java editor extension
    • add syntax highlighting
      • keywords
      • mark wrong syntax
    • intellisense
      • autocompletion of annotations
      • possibility to choose method to compare with from list of methods
    • jump to annotated method (or annotation of the method?) after click on it in the list of methods
    • add version of the measured method to the annotation
    • generate estimated parameter values
  • dialog for simple formula generation
  • spl results
    • connection profiles (hudson, ...)
      • multiple sources
      • create and edit dialog
    • download test-data & test-results
    • import local test-data & test-results
      • handled by plugin or copied by user into specified folder
    • listing of spl data
      • filtered dedicated tab by (connection profile / date / revision)
        • test-data - show graph
        • test-results - sort by measure of success / fail
        • spl formula on already measured test-data
          • evaluate or refuse due to lack of data
  • performance progress for selected function
    • spl support for evaluating performance difference
      • find K that F1 is K times faster than F2
    • pick revisions (handled by spl)
      • since when (date / revision)
      • how often (date / revision)

Hudson plugin

  • periodic launch
    • planning provides Hudson in job configuration
    • Ant script using Base framework to perform measurements
    • get results back to Hudson from testing server to be able to display them
  • ensure that testing server has at most one measurement task running at a time
  • per project configuration
    • which SCM is used (to be able to get specific revisions)
  • display results
    • list of measured annotated methods
    • detail of each measured annotation with graphs of measured methods
    • compare with previous builds
  • provide results for Eclipse (anything else in machine-readable format)
    • export all results as one zip package for offline usage
    • allow Eclipse to fetch latest results
      • using FTP

Regression testing

  • extend SPL formulas
    • add revision identification to methods and generators

Case study

  • find real-world Java project to annotate

Related

Wiki: Home