Re: [Xmltv-devel] new grabber for www.digiturk.com.tr

[<hon...@gm...>]

On Tue, 18 Feb 2014 15:57:40 +0000 (UTC), Diguen wrote:
> 
> I've worked on code a bit more.
[...]
> 
> I used LWP Simple for fetchig programme details pages, it is much more
> faster now.

Yes it would be quicker using LWP by you shouldn't do that. The point from my earlier post was that site scrapers should *not* try and get pages as fast as they can.

There are website grabbers out there who pride themselves on being "very fast" and "fetch a whole days listings in under a minute" but that's a rookie's mistake.  That behaviour is highly likely to annoy the website owner and you are likely to be banned, and probably get everyone else banned too.  Don't do it.

Especially when there's no need for it - simply run the grabber as a scheduled background job and it doesn't matter how long it takes; the data will be there waiting for you when you're ready.

Geoff