Allow redirect by single iframe / frameset on page
Status: Beta
Brought to you by:
huni
As discussed in: http://sourceforge.net/p/phpcrawl/discussion/307696/thread/6bf703b0/
Several websites I'm trying to crawl use multiple URL's for their website. On some of those, the page is actually a HTML page that contains an iframe or a frameset which loads the content from the actual URL. Since I don't want to crawl unrelated websites, I disabled cross-domain crawling. However, in the cases where the first content retrieved only contains a single frameset or iframe (except other required html, such as html, head, body tags, etc), this URL should be followed.
Anonymous