I want create a '1.5' follow mode. I want just check external URL in website. But i don't want crawler, crawl other website. Where should I change the code?
For example i crawl current-domain.com, if crawler find other-domain.com, it check if domain return 200 http code. But it doesn't crawl other-domain.com.
It's possible ?
Thank in advance.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi, i need information :)
I want create a '1.5' follow mode. I want just check external URL in website. But i don't want crawler, crawl other website. Where should I change the code?
For example i crawl current-domain.com, if crawler find other-domain.com, it check if domain return 200 http code. But it doesn't crawl other-domain.com.
It's possible ?
Thank in advance.
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Maybe use referer ?
Last edit: Anonymous 2016-02-15
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Please help me, i have try in urlMatchesRules function :
if ($this->general_follow_mode == 0 ) { if($this->CurrentDocumentInfo->referer_url != "") { $pos = strpos($this->CurrentDocumentInfo->referer_url, "website-to-crawl.com"); if ( $pos <= 1 ) { return false; } } }but doesn't work
Last edit: Anonymous 2016-02-16