[htdig] robots.txt not recognized by htdig-3.2.0b5

[Rick <lis...@ds...>]

Hello htdiggers ...

Just installed htdig-3.2.0b5, but it has some
difficulty reading this robots.txt file ...

	User-agent: grub-client
	Disallow: /
	User-agent: *
	Disallow:

I believe that the robots.txt syntax is correct.
Release 3.1.6 has no problem with it.

Any ideas/solutions/workarounds ?