Menu

#4 WARC/KEYVAL input format brokenness and defaults

open-fixed
nobody
None
5
2012-02-12
2012-02-03
No

First of all, there is an inconsistency between individual jobs *and* between what some jobs' usage messages say.
* QueryTermCount claims it defaults to WARC, but it defaults to nothing and will break if no format is explicity specified
* TrecRun defaults to KEYVAL (even though its usage says it defaults to WARC, My patch in ticket 3483256 addresses the incorrect usage output)
* TrecRunBaselines defaults to KEYVAL (even though its usage says it defaults to WARC)

Second, the README.html does not correspond.
For example we're told to
"hadoop jar mirex-0.2.jar nl.utwente.mirex.TrecRun \
ClueWeb09_Anchors/* TrecOut wt09-topics.txt"
but if you run this you'll get NullPointers at nl.utwente.mirex.util.KeyValueInputFormat$KeyValueRecordReader.nextKeyValue(KeyValueInputFormat.java:70)
This command needs to have an explicit 'WARC' argument.

The best solution is probably to default everything to WARC, fix all usage messages, and then the instructions in the readme should also work.

Discussion

  • Dieter Plaetinck

    correction: QueryTermCount does default to WARC.

     
  • Robin Aly

    Robin Aly - 2012-02-12
    • status: open --> open-fixed
     

Log in to post a comment.