Relative URL are not indexed

Help
2010-09-16
2012-09-13
  • Richard SINELLE

    Richard SINELLE - 2010-09-16

    Hi all,

    I make some test on Open Search Server v1.2 - developer - rev 861 - build 408

    I have created a web site with Liferay and tried to index the content.

    I have created an index with the default web template Crawler.

    In some pages, I have some relative link like :

    We hope you will enjoy our new web site and that your visit today will be the first of many more to come.

    The link is never indexed by the crawler.

    I found a workaround that is to add a complete url. So the like become

    We hope you will enjoy our new web site and that your visit today will be the first of many more to come.

    Is it normal to index only complete url ?

    or should I need to change the default configuration of the Web Crawler ?

    Thanls for your help

     
  • Richard SINELLE

    Richard SINELLE - 2010-09-16

    To cmplete the description, the configuration is !

     
  • Naveen A.N

    Naveen A.N - 2010-09-19

    can u please check the robots.txt file has the user agent is allowed to crawl
    the site..

     
  • Richard SINELLE

    Richard SINELLE - 2010-09-20

    My robots.txt :

    User-agent: *

    Disallow:

    and I add the meta tag in each page :

    <meta name="robots" content="index, follow">

     
  • Richard SINELLE

    Richard SINELLE - 2010-10-06

    Hi

    I find the answer to my question. It's a bug because I use to my test
    environment an uri with port.

    In LinkUtils.changePathUrl line : 42, you forget to add the character ":" to
    the new uri parameter.

    The code is

    if (url.getPort() != -1)

    newUri.append(url.getPort());

    and must be

    if (url.getPort() != -1) {

    newUri.append(":");

    newUri.append(url.getPort());

    }

    Thanks

     
  • Emmanuel Keller

    Emmanuel Keller - 2010-10-11

    Hi Richard,

    Thank you for your contribution. The source code you submitted has been
    approved and integrated in the 1.2 branch.

    Regards,

    Emmanuel Keller.

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks