Prevent crawling same URL with different query-parts twice
Status: Beta
Brought to you by:
huni
Menu
▾
▴
#12 Prevent crawling same URL with different query-parts twice
Add the ability to tell the crawler that i.e. this URL with different query-parts only should get requested
once:
http://mysite.com/show.php?order_by=date
http://mysite.com/show.php?order_by=name
See this forum-thread and marlars suggested solution:
https://sourceforge.net/projects/phpcrawl/forums/forum/307696/topic/5199468
Discussion
Anonymous
View and moderate all "feature-requests Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Feature Requests"
Is it possible now?
View and moderate all "feature-requests Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Feature Requests"
xdfgdfgdfgdfgdfgdf