Lately, because of a change in vendors, we have been noticing a rise traffic from the Xenu User agent. I noticed that its listed under "Unknown browsers" instead of the identified "Robots/Spiders". Upon editing robots.pm to add in the the changes to be able to identify the UA, i took notice that it should have already been identified by robots.pm
Default regex to identify bot from robots.pm:
Regex i added to identify bot:
'Xenu_Link_Sleuth', (thinking it may be a case sensitivity issue)
'Xenu_Link_Sleuth_1.3c', (a second attempt to identify it being more specific)
Nothing appears to be catching catching and identifying this UA.
Log in to post a comment.