#201 Modification of python-urllib regex in robots.pm

open
nobody
None
1
2013-07-31
2013-07-31
Julien Lefeuvre
No

Hi,

As i was using awstats to gather statistics on a project using a REST api that is used by several clients using project-ruby, project-python, ... as user agents, i realized that "project-python" was not showing up as a browser and was getting parsed as a python-urllib robot.

Indeed, the robots.pm regex for python-urllib is 'python'. As python-urllib default user agent is "Python-urllib/", i attached a patch that modifies the 'python' regex to 'python-urllib'. This is just a suggestion though, i did not check older python versions to know if the python-urllib user-agent was always that precise.

Thanks for this nice tool,
Cheers,

Julien

1 Attachments

Discussion