#32 REDIR of robots.txt leads to no fetching

[Anonymous]

If the attempt to fetch the robots.txt file comes back with
a REDIR, the rootnode does not get into the worklist, so
nothing for that site gets harvested.

My guess is that the block of code after the REDIR section in
Controller.pm sub done function needs to get executed before
returning.

Note there is a FIXME comment just before the code in question.