Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#1 the ability to parse a "do not follow" file

open
nobody
None
5
2003-03-26
2003-03-26
Krusty D Klown
No

Hi. I am using this to see who my site is indirectly
linked to (through links that are on pages I am linked
to). I would like the ability to supply a text file of
domains and IP's that the Lech shouldn't follow. This
would eliminate the Lech from following links to many
ad servers that are linked to pages, linked to me.
There are sites that regularly update ad server IP and
URL lists. I would like to parse these lists so I
don't download doubleclick.net or adfarm.com by
following links off of my site.

Otherwise, WebLech is pretty cool. Fairly stable also,
for Alpha code.

Discussion