Hi Martin,

1. I can't speak for anyone else, but as long as you aren't trying to use what you find here for the purpose of creating a malicious crawler/bot, I don't think there is a problem.
(The fact that you are asking shows that is not what you are trying to do.)

2. I am not sure that this is the best place. Most of the information here is pretty general.

3. You can see all of the filters that come with fail2ban in the GitHub repository: https://github.com/fail2ban/fail2ban/tree/master/config/filter.d
On the wiki, have a look the the Configuration category: http://www.fail2ban.org/wiki/index.php/Category:Configuration
It includes suggestions for filters that are not included.

On Wed, Jun 26, 2013 at 12:22 PM, Martin York <martin@moz.com> wrote:

Hi all,

I work for Moz.com we provide analytical information about the web; and as such we have a set of crawlers to generate the information we need.

We strive very hard to make sure the crawlers are polite and don't stress particular domains. But recently we had a complain that our crawler was hitting a "known attack vector" on a web server. We have remedied this particular situation but would like to try and automatically exclude paths that are considered harmful.

After some research that has brought us to your wiki.
So my question is:

1) Do you mind us (Moz) using information we find here.
    We do not want to digg through your wiki if you have ethical issues with crawlers.
    Some people do and we understand this and will move on to other resources.
    If there are legal issues (licensing) I am sure that is something we can talk about.
2) Is this a good location to find this type of information.
3) Is there anywhere in particular on your site I should start reading.

Any help appreciated


This SF.net email is sponsored by Windows:

Build for Windows Store.

Fail2ban-users mailing list