Some webistes use a sepearate site for setting a session-cookie (like "http://www.bobbyberkhome.com").
For the first request a redirect to another site is responded. This other site sets a session-cookie and then redirects BACK to
the entry page. But the crawler already requested this site and doesn't request it again and sthe crawling-process stops.