Filtering special files or folders

Status: Abandoned

Brought to you by: csaltos

Filtering special files or folders

Forum: indexing and crawling

Creator: Nobody/Anonymous

Created: 2006-02-17

Updated: 2013-04-09

Nobody/Anonymous - 2006-02-17

How can I filter a path, a file or an url containing special phrases, so that it not will be added to the index?
For example I want to crawl the site

www.example.com

and want to exclude:
- the complete Folder
www.example.com/content/privat
- and url which contain:
* sendit *

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- netzgoetter - 2006-02-17
  
  I have the same questions.
  
  How can I configure it?
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Darin Kuntze - 2006-02-24
  
  Oxyus uses regex to specify which content is indexed. The config file is located WEB-INF/conf/initial_pages.conf.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - netzgoetter - 2006-02-24
    
    I understood so far that the initial_pages.conf is used for the defintion of the start crawl url.
    
    What is the syntax for an entry in initial_pages.conf to filter some urls.
    
    Example:
    
    http://www.example.com
    
    Starts the crawl at that point. How can I exclude for example urls that contain the folder cgi?
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Log in to post a comment.