Menu ▾ ▴

#3 where to get wt09-topics.txt from?

open-fixed
nobody
None
5
2014-10-01
2012-02-02
Dieter Plaetinck
No

Hi,
in README.html the example QueryTermCount and TrecRun refer to the file wt09-topics.txt
Where should we get it from?

Discussion

  • Djoerd Hiemstra

    Djoerd Hiemstra - 2012-02-02

    The TREC topics can be found on the TREC website, under:
    Data -> Web Track -> TREC 2009 Web Track
    http://trec.nist.gov/data/web/09/wt09.topics.queries-only

    To use trec_eval (see README) you need the relevance judgements provided by TREC as well. These are not provided for trec_eval, and therefore need to be pre-processed at bit. Let me know if you have any more problems.

     

  • Djoerd Hiemstra

    Djoerd Hiemstra - 2012-02-02
    • status: open --> open-fixed
     

  • Dieter Plaetinck

    Dieter Plaetinck - 2012-02-03

    Hmm, I have the file in my current working directory, as well as uploaded to the DFS.
    Yet, hadoop still "cannot find it".

    $ wget http://trec.nist.gov/data/web/09/wt09.topics.queries-only -O wt09.topics.txt
    2012-02-03 10:51:43 (209 MB/s) - “wt09.topics.txt” saved [1104/1104]
    $ hadoop dfs -copyFromLocal wt09.topics.txt wt09.topics.txt
    $ hadoop dfs -ls
    Found 3 items
    drwxr-xr-x - dplaetin supergroup 0 2012-02-02 16:34 /user/dplaetin/ClueWeb09_Anchors
    drwxr-xr-x - dplaetin supergroup 0 2012-02-02 14:43 /user/dplaetin/ClueWeb09_English
    -rw-r--r-- 2 dplaetin supergroup 1104 2012-02-03 10:53 /user/dplaetin/wt09.topics.txt

    $ hadoop jar mirex-0.2.jar nl.utwente.mirex.TrecRun ClueWeb09_Anchors/* TrecOut wt09-topics.txt
    12/02/03 10:53:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    12/02/03 10:53:35 INFO mapred.JobClient: Cleaning up the staging area hdfs://n-0:9000/var/bomvl/tmp/mapred/staging/dplaetin/.staging/job_201202021421_0017
    12/02/03 10:53:35 ERROR security.UserGroupInformation: PriviledgedActionException as:dplaetin cause:java.io.FileNotFoundException: File does not exist: wt09-topics.txt
    Exception in thread "main" java.io.FileNotFoundException: File does not exist: wt09-topics.txt
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:517)
    at org.apache.hadoop.filecache.DistributedCache.getFileStatus(DistributedCache.java:185)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineTimestamps(TrackerDistributedCacheManager.java:700)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:763)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:655)
    at org.apache.hadoop.mapred.JobClient.access$300(JobClient.java:174)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:865)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:465)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)
    at nl.utwente.mirex.TrecRun.main(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

     

  • Dieter Plaetinck

    Dieter Plaetinck - 2012-02-03
    • status: open-fixed --> open
     

  • Dieter Plaetinck

    Dieter Plaetinck - 2012-02-03

    oops sorry. silly me, I used the wrong filename. works now!

    here's a patch for the README:

    From: Dieter Plaetinck <dieter.plaetinck@intec.ugent.be>
    Date: Fri, 3 Feb 2012 11:02:12 +0100
    Subject: [PATCH] document how to get the wt09-topics.txt file

    ---
    README.html | 9 +++++++++
    1 files changed, 9 insertions(+), 0 deletions(-)

    diff --git a/README.html b/README.html
    index a7f5517..2bdc2f3 100644
    --- a/README.html
    +++ b/README.html
    @@ -38,6 +38,15 @@ First, the warc inputfiles must be uploaded to the HDFS:

    </pre>

    +For the TREC tasks, you'll want to have the topics from the 2009 web track:
    +
    +<pre>
    +
    +% wget http://trec.nist.gov/data/web/09/wt09.topics.queries-only -O wt09-topics.txt
    +% hadoop dfs -copyFromLocal wt09.topics.txt wt09-topics.txt
    +
    +</pre>
    +
    <h2> Usage </h2>

    Run anchor text extraction as follows:
    --
    1.7.9

     

  • Robin Aly

    Robin Aly - 2012-02-12
    • status: open --> open-fixed
     

  • Nobody/Anonymous

    Nobody/Anonymous - 2012-10-13

    Well I sincerely liked studying it. This information provided by you is very useful for proper planning.
    news for cheaper burberry tobacco http://cheapburberrybagforyou.bloglove.cc/

     

Log in to post a comment.

veeam logo