This is an excellent crawler :)
Support for gzip encoded content would be a good idea for those cases where there bandwidth is the bottleneck, rather than processing time.
Anonymous
You seem to have CSS turned off. Please don't fill out this field.
Hi,
yes, it's a good point. Will put it onto the featrue-request list.
Thanks!
This is an excellent crawler :)
Support for gzip encoded content would be a good idea for those cases where there bandwidth is the bottleneck, rather than processing time.
Hi,
yes, it's a good point. Will put it onto the featrue-request list.
Thanks!