Menu

#20 Allow redirect by single iframe / frameset on page

open
nobody
5
2013-05-29
2013-05-29
MadEgg
No

As discussed in: http://sourceforge.net/p/phpcrawl/discussion/307696/thread/6bf703b0/

Several websites I'm trying to crawl use multiple URL's for their website. On some of those, the page is actually a HTML page that contains an iframe or a frameset which loads the content from the actual URL. Since I don't want to crawl unrelated websites, I disabled cross-domain crawling. However, in the cases where the first content retrieved only contains a single frameset or iframe (except other required html, such as html, head, body tags, etc), this URL should be followed.

Discussion

Anonymous
Anonymous

Add attachments
Cancel