Hi all,


I work for Moz.com we provide analytical information about the web; and as such we have a set of crawlers to generate the information we need.

We strive very hard to make sure the crawlers are polite and don't stress particular domains. But recently we had a complain that our crawler was hitting a "known attack vector" on a web server. We have remedied this particular situation but would like to try and automatically exclude paths that are considered harmful.

After some research that has brought us to your wiki.
So my question is:

1) Do you mind us (Moz) using information we find here.
    We do not want to digg through your wiki if you have ethical issues with crawlers.
    Some people do and we understand this and will move on to other resources.
    If there are legal issues (licensing) I am sure that is something we can talk about.
2) Is this a good location to find this type of information.
3) Is there anywhere in particular on your site I should start reading.


Any help appreciated

Martin.