As i was using awstats to gather statistics on a project using a REST api that is used by several clients using project-ruby, project-python, ... as user agents, i realized that "project-python" was not showing up as a browser and was getting parsed as a python-urllib robot.
Indeed, the robots.pm regex for python-urllib is 'python'. As python-urllib default user agent is "Python-urllib/<version>", i attached a patch that modifies the 'python' regex to 'python-urllib'. This is just a suggestion though, i did not check older python versions to know if the python-urllib user-agent was always that precise.
Thanks for this nice tool,
Log in to post a comment.