IP Range crawling?

Help
Anonymous
2014-04-26
2014-04-29

  • Anonymous
    2014-04-26

    I would like to crawl an IP Range, eg an entire country. Can I do this easily with phpcrawl?

     
    Last edit: gyuri79 2014-04-26

  • Anonymous
    2014-04-29

    Yes and no ;)

    Of course you can loop over a list of IPs and start a crawling-instance for every IP.

    But in most cases you wont't get any proper content from webservers if you use the IP as host/domainname (which has nothing to do with phpcrawl, just a general fact).

    E.g.:
    The server of alternate.de (just came into my mind) has the IP 195.234.50.235.
    But if you do a http-request for http://195.234.50.235, you don't get anything from the server (just try in you browser), because the hostname is missing in your request.

     


Anonymous


Cancel   Add attachments