I just added a condition for the content type that should be text/html and the http status code which has to be 200. That works very vell. If the condition fails I just put a return -1 and the page is not crawled.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hello
I have a problem with the crawl, because I can not exclude some files. In code I reject CSS files, but see by yourself in the picture:
I don't know what to do, perhaps you have some ideas. Thanks in advance!
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
I just added a condition for the content type that should be text/html and the http status code which has to be 200. That works very vell. If the condition fails I just put a return -1 and the page is not crawled.