Menu ▾ ▴

ConfigureSearch

Anonymous

Following the same logic of Nutchwax, the configuration files will be inside the JAR files, which requires that after changing configuration you must compile again.

Step-by-step

Edit file pwa-technologies/PwaArchive-access/projects/nutchwax/conf/wax-default.xml:

  • Change property collection.type according to the type of collection for indexing. This has implications mostly during the Linkdb and Index phases. There are three types of collections:
    • normal - collection from one crawl. It will be handled as one snapshot, where a version is identified by a URL.
    • multiple - collection from multiple crawls. It will be handled as multiple snapshots, where a version is identified by a URL and a day.
    • trec - collection from TREC (Text REtrieval Conference)
  • If normal or trec are selected then exit this page.

NOTE: you can change the values of the other properties if necessary.

  • If multiple is selected then the database parameters must be configured to create virtual snapshots:
    • Change property database.conection (e.g. //t2.tomba.fccn.pt/nutchwax)
    • Change property database.username (e.g. nutchwax)
    • Change property database.password (e.g. xxxxx)

Virtual snapshots guarantee that versions of documents link only to versions of other documents with the closeness timestamp. This is a requirement for link-based algorithms, such as Pagerank, to work well.

Prepare a PostgreSQL database if multiple is selected in the collection.type property:

  • Create database for the first time using it:
    • createdb nutchwax
    • ALTER USER nutchwax WITH PASSWORD 'xxxxxx';
  • Login in database:
    • psql

  • Create table and trigger: 

      DROP table files cascade;
  SET client_encoding TO 'LATIN1';

  CREATE LANGUAGE plpgsql;
  CREATE OR REPLACE FUNCTION before_insert() RETURNS trigger AS '
  DECLARE
   n integer;
  BEGIN
   IF tg_op = ''INSERT'' THEN
     select count(*) into n
      from files
      where date=new.date and url=new.url;  
      IF n > 0 THEN
        RETURN NULL;
      ELSE
        RETURN new; 
      END IF;   
   END IF;
  END
  ' LANGUAGE plpgsql;

  create table files
      (date TIMESTAMP, 
       url VARCHAR(4000),
       type VARCHAR(100),
       status INTEGER,
       size INTEGER NOT NULL,
       arcname VARCHAR(100) NOT NULL,
       PRIMARY KEY (url,date));

   CREATE TRIGGER before_insert_trigger BEFORE INSERT ON files
    FOR EACH ROW EXECUTE PROCEDURE before_insert();

  • Install heritrix-1.12.1 because of libraries to extract meta-data:

    • download heritrix-1.12.1.zip
    • unzip it to a heritrixDirectory directory
    • export HERITRIX_HOME=heritrixDirectory (e.g. /opt/searcher/heritrix-1.12.1)
    • export WAYBACK_HOME=waybackDirectory (e.g. /opt/searcher/pwa-technologies/PwaArchive-access/projects/wayback/wayback-webapp/target/wayback-1.2.1/WEB-INF/lib/)

  • Extract metadata from ARC files:

    • ${HERITRIX_HOME}/src/scripts/arcreader.sh /_directoryArcs_ /_directoryArcs_/stats.csv
    • ${HADOOP_HOME}/bin/hadoop jar pwa-technologies/PwaArchive-access/projects/nutchwax/nutchwax-job/target/nutchwax-job-0.11.0-SNAPSHOT.jar class org.apache.access.nutch.utils.UrlNormalizer /_directoryArcs_/stats.csv /_directoryArcs_/statsNormalized.csv 6 1

  • Load metadata into database:

    • psql 
        \COPY files FROM '/_directoryArcs_/statsNormalized.csv' DELIMITER ',' NULL AS '-' CSV

  • Create indexes: 

      CREATE INDEX type_index ON files(type);
  CREATE INDEX status_index ON files(status);
  CREATE INDEX url_index ON files USING hash(url);

Errors and debugging

  • Check collection.type configured:
    • ${HADOOP_HOME}/bin/hadoop jar pwa-technologies/PwaArchive-access/projects/nutchwax/nutchwax-job/target/nutchwax-job-0.11.0-SNAPSHOT.jar version
  • Test database:
    • test postmaster:

psql -h t7.tomba.fccn.pt -d nutchwax 

* run postmaster:

su service postgresql stop service postgresql start 

* accessing remotly to postgresql - add to file /data/postgres:/data/pg_hba.conf

           host all all 0.0.0.0/0 md5


* set /usr/local/pgsql/data/postgresql.conf:

           listen_addresses = '*'
           port = 5432

See http://www.cyberciti.biz/tips/postgres-allow-remote-access-tcp-connection.html for more info.

  • Query database just for debugging: 

       java -classpath pwa-technologies/PwaArchive-access/projects/nutchwax/nutchwax-job/target/nutchwax-job-0.11.0-SNAPSHOT.jar:~/.m2/repository/postgresql/postgresql/8.3-604.jdbc4/postgresql-8.3-604.jdbc4.jar org.archive.access.nutch.jobs.sql.SqlSearcher [database.connection] [database.user] [database.password] [URL] [timestamp]
   e.g. org.archive.access.nutch.jobs.sql.SqlSearcher //t7.tomba.fccn.pt/nutchwax nutchwax xxxxx http://jn.sapo.pt/robots.txt "20070831100222"

  • If postgres requires a superuser to create the before_insert() function, then you should:

    • su
    • su postgres
    • psql 
         ALTER USER nutchwax WITH CREATEUSER CREATEDB;
   \du
Auth0 Logo