On Sun, Jan 29, 2012 at 7:00 PM, Aristotle Pagaltzis <pagaltzis@gmx.de> wrote:
* Lars Lindner <lars.lindner@gmail.com> [2012-01-29 00:45]:
> On Sun, Jan 29, 2012 at 12:11 AM, Aristotle Pagaltzis <pagaltzis@gmx.de>wrote:
> > As an aside, I think this is also how the feed update schedule
> > should be calculated, automatically.
>
> Given that we do use Etags and Last-Modified? So exchanging "I know
> for certain there is nothing new" for "There is probably nothing new,
> and I saved some small requests...".

I subscribe a number of feeds that update only every couple of months.
Polling them every hour even with conditional requests is a non-trivial
waste of resources. I have manually configured many of them to use
a slower interval, and I have the default interval set to 6 hours, but
it would be nice if Liferea could do something smarter and more flexible
than that automatically.

At the very least, also, it should add some random scatter to the update
schedule. When you open Liferea after it hadnít run for a while, it will
realise there are 50 feeds to update, and updates them all at once Ė
which wouldnít be a problem if it was just that one time, but Liferea
then schedules the next update of those 50 feeds all at the same time as
well, so they will forever thenceforth update all at once. And if many
of them update regularly, then every time is freeze time.

But getting the statistics of random scattering right so it will really
disperse mass events like that, and effectively, is somewhat tricky.

I once wrote that aggregators should do exponential back-off while they
find no new items in a feed, maybe up to a limit. Thatís also an option.

Maybe some combination of algorithms is a better idea. Maybe back off up
to a limit informed by the average+stddev of the feedís update rate. You
donít have to use the numbers the formulś produce directly either, you
can take them as just a factor.

*Anything* even slightly less primitive than a fixed interval schedule
would be better. And, you could remove a configuration option.

Other users wouldn't agree with removing the configuration option.
You know I'm always happy to remove options :-)

Besides you are aware of the TTL standard in RSS and Atom which
feed authors should use for rarely updated feeds. So for your use case
I'd say the author should provide a reasonable update interval and
not the aggregator trying to guess it.

Regards,
Lars