Is it possible that in your case the crawler visited http://www.tnumc.org/404.cfm during the crawling-process before?
If so it won't request the same URL a second time, every URL will only get requestes once.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It doesn't appear to follow 302's, only 301's… shouldn't it follow them just like a 301?
Thanks,
Patrick
Hi Patrick,
just did a quick test with the URL you wrote to me, everything looks ok form over here, the crawler follows the 302-redirect, here is my output:
Page requested: http://www.tnumc.org/custpage.cfm/frm/73429/sec_id/73429 (302)
Referer-page:
Content received: 0 bytes
Page requested: http://www.tnumc.org/404.cfm (200)
Referer-page: http://www.tnumc.org/custpage.cfm/frm/73429/sec_id/73429
Content received: 21053 bytes
Is it possible that in your case the crawler visited http://www.tnumc.org/404.cfm during the crawling-process before?
If so it won't request the same URL a second time, every URL will only get requestes once.