Liferea

On sam, 2005-05-28 at 22:48 +0200, Lars Lindner wrote:
> Hi everyone!
> 
> My ideas for 0.9.3:
> 
> 

May i suggest a small (i don't know how small it is ;) ) improvement:
As people are syndicated to more and more planet it is not uncommon to
have 2 or three time the same posting on the list coming for example
from three different planets.

This is really annoying, and beside that the timestamp differs a bit,
and maybe the formatting.

There should be a way liferea recognize dups and only show the first (in
timestamp) one, for example.

Duplicate feed could maybe be detected by hashing the text or a small
portion of the text (without any markup and ignoring space or somthing
like that) or any other technique i'm not aware to determine text
similarities in a smart way.

Maybe this is already planned, but it was just a thought..

Raf

Short answer is "maybe soon".

I've been thinking about this option some. The best way to tell if two
items are duplicates is based on their unique item IDs. Nearly every
RSS item is tagged with a (hopefully) unique identifier. The planets,
when they take items from other feeds, do not change the
identifiers. So, one can comparie the identifier strings of multiple
items to see if they are the same.

This feature requires significant parts of Liferea to be rewritten. I
am planning to (sometime after 1.0 is released) to re-write the
backend of Liferea to store feeds in a sqlite database instead of the
XML cache files. At that point, it will be easy to implement this
feature.

When viewing an item, I think that it would be best to display a list
of the feeds in which the item also appears. One problem with using
the IDs is that it allows one feed to "spoof" items of the other
feed, so if you view an item in the errant feed, it will mark a
choosen item in an unrelated feed as unread.

One problem is the best way to handle item updates. Feed A publishes
item 1. Liferea gets the item from planet B. User marks item as read
in planet B. Liferea downloads updated version of item from feed
A. How does it tell that the item has been updated?

Or maybe it should verify that the contents of the items match exactly
and the unique IDs match exactly?

-Nathan

On Sun, May 29, 2005 at 12:39:29AM +0200, Rapha=EBl Slinckx wrote:
> On sam, 2005-05-28 at 22:48 +0200, Lars Lindner wrote:
> > Hi everyone!
> >=20
> > My ideas for 0.9.3:
> >=20
> >=20
>=20
> May i suggest a small (i don't know how small it is ;) ) improvement:
> As people are syndicated to more and more planet it is not uncommon to
> have 2 or three time the same posting on the list coming for example
> from three different planets.
>=20
> This is really annoying, and beside that the timestamp differs a bit,
> and maybe the formatting.
>=20
> There should be a way liferea recognize dups and only show the first (i=
n
> timestamp) one, for example.
>=20
> Duplicate feed could maybe be detected by hashing the text or a small
> portion of the text (without any markup and ignoring space or somthing
> like that) or any other technique i'm not aware to determine text
> similarities in a smart way.
>=20
> Maybe this is already planned, but it was just a thought..
>=20
> Raf
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.Net email is sponsored by Yahoo.
> Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
> Search APIs Find out how you can build Yahoo! directly into your own
> Applications - visit http://developer.yahoo.net/?fr=3Doffad-ysdn-ostg-q=
22005
> _______________________________________________
> Liferea-devel mailing list
> Liferea-devel@...
> https://lists.sourceforge.net/lists/listinfo/liferea-devel

* Nathan Conrad <conrad@...> [2005-05-29 01:20]:
> When viewing an item, I think that it would be best to display
> a list of the feeds in which the item also appears. One problem
> with using the IDs is that it allows one feed to "spoof" items
> of the other feed, so if you view an item in the errant feed,
> it will mark a choosen item in an unrelated feed as unread.

We have been discussing this particular case on the Atom working
group mailing list a lot. The short answer is that there is no
way to do this reliably that does not involve signatures; but
it’s going to be a while before people start signing their feed
entries.

The longer answer is that it’s a question of trust, and what you
do depends on whether you trust a source that it is providing
entries republished from another source; f.ex the user can
express that he trusts a Planet to be republishing entries from
some source, and if he trusts multiple Planets for the same
sources, then they can cancel each other’s entries. If OTOH an
entry appears with the same ID as an entry from another feed,
this is reason to become paranoid. In that case, if it’s a
republisher, and the entry you received differs from the one you
got from the original source, then you can refresh the original
feed and check what that looks like.

In any case, this is difficult territory.

Regards,
-- 
Aristotle

* Lars Lindner <lars.lindner@...> [2005-05-28 23:25]:
> - Minimize red feed retrieval error boxes (make them a small 
>   header with a more link that expands to the full text). The
>   direct display of pretty technical parser errors is low
>   usability.

I agree, but please don’t make the full error message imprecise
or cumbersome to retrieve. I had alerted various people to
problems in their feeds or sometimes used the information to
write a filter to fix the feed. Please keep it easy to find out
what’s wrong. As long as that is ensured, I will be happy to see
this done better.

Anyway.

Here’s another suggestion: Atom 1.0 is imminent. The discussion
was declared done three days ago, and now the editors are making
the last changes that were discussed to the draft. Within a few
weeks the RFC will be published. On the whole, the format feels
very much like Atom 0.3, but it’s also quite different in many
details. -draft-08 of the spec has some minor aspects that will
change significantly for the final RFC (multiple authors allowed
being the biggest, I think), so I don’t know if it’s really
helpful as a starting point for implementation. But you probably
want to at least keep your eyes out for the RFC.

Regards,
-- 
Aristotle

* A. Pagaltzis <pagaltzis@...> [2005-05-29 02:45]:
> The longer answer is that it’s a question of trust, and what you
> do depends on whether you trust a source that it is providing
> entries republished from another source; f.ex the user can
> express that he trusts a Planet to be republishing entries from
> some source, and if he trusts multiple Planets for the same
> sources, then they can cancel each other’s entries. If OTOH an
> entry appears with the same ID as an entry from another feed,
> this is reason to become paranoid.

What I mean is, of course, if an entry in one feed has the same ID
as an entry in another, but the user has not declared that he
trusts one feed to be republishing entries from the other, then
you should be paranoid.

> In that case, if it’s a republisher, and the entry you received
> differs from the one you got from the original source, then you
> can refresh the original feed and check what that looks like.

And by this I mean:

* feed f1 has entry e1 with ID xyz
* feed f2 has an entry e2 different from e1, but also with ID xyz

In this case you could refresh f1 to check if e1 changed, and
then decide what to do based on that. If after the refresh you
find that e1 and e2 are identical, then you could just let one
cancel the other out. If they are not, the user should possibly
be alerted to decide how to proceed. Or you could not tell the
user at all, and just display the entry as if nothing special was
going on. Or only show an informational box on the entry that
notifies the user, but does not require any interaction. Or…

Regards,
-- 
Aristotle

On Sun, 2005-05-29 at 02:52 +0200, A. Pagaltzis wrote:
> * Lars Lindner <lars.lindner@...> [2005-05-28 23:25]:
> > - Minimize red feed retrieval error boxes (make them a small=20
> >   header with a more link that expands to the full text). The
> >   direct display of pretty technical parser errors is low
> >   usability.
>=20
> I agree, but please don=E2=80=99t make the full error message imprecise
> or cumbersome to retrieve. I had alerted various people to
> problems in their feeds or sometimes used the information to
> write a filter to fix the feed. Please keep it easy to find out
> what=E2=80=99s wrong. As long as that is ensured, I will be happy to see
> this done better.
>=20
> Anyway.
>=20
> Here=E2=80=99s another suggestion: Atom 1.0 is imminent. The discussion
> was declared done three days ago, and now the editors are making
> the last changes that were discussed to the draft. Within a few
> weeks the RFC will be published. On the whole, the format feels
> very much like Atom 0.3, but it=E2=80=99s also quite different in many
> details. -draft-08 of the spec has some minor aspects that will
> change significantly for the final RFC (multiple authors allowed
> being the biggest, I think), so I don=E2=80=99t know if it=E2=80=99s real=
ly
> helpful as a starting point for implementation. But you probably
> want to at least keep your eyes out for the RFC.

Hi,

I'm actually doing some work on the pie_entry and pie_feed files now :)
Basically because I was stupid enough to create a feed with -draft-08
just to find that Liferea hated it :)
So I guess best would be to=20
a) make it work with -draft-08
b) as soon as the final RFC hits my mailbox, start working on polishing=20
   the changed parts for 1.0

--
Andreas

On dim, 2005-05-29 at 03:00 +0200, A. Pagaltzis wrote:
> * A. Pagaltzis <pagaltzis@...> [2005-05-29 02:45]:
>=20
> What I mean is, of course, if an entry in one feed has the same ID
> as an entry in another, but the user has not declared that he
> trusts one feed to be republishing entries from the other, then
> you should be paranoid.
>=20

As you said there is no reliable way to detect this, so i think liferea
should not do anything to try to detect it, it will be hackish at more.

This particular case should not interfere on the descussion on how to
display multiple similar feeds.

> > In that case, if it=FFs a republisher, and the entry you received
> > differs from the one you got from the original source, then you
> > can refresh the original feed and check what that looks like.
>=20
> And by this I mean:
>=20
> * feed f1 has entry e1 with ID xyz
> * feed f2 has an entry e2 different from e1, but also with ID xyz

What i suggest to detect same unique id (that's weird :) ) on different
posts, is to also take account of the title of the post. These two
should be enough to determine if two items are identical or not. Maybe
even the first of last characters.

For the UI, i don't want to see there are multiple same item, otherwise
it's just like it is now. So i suggest in the unread vFolder only one
item appear maybe with an icon indicating that it is has duplicates, or
maybe with an arrow so you can expand and see the duplicate items.

Raf

Am Sonntag, den 29.05.2005, 12:02 +0200 schrieb Raphaël Slinckx:
> On dim, 2005-05-29 at 03:00 +0200, A. Pagaltzis wrote:
> > * A. Pagaltzis <pagaltzis@...> [2005-05-29 02:45]:
> > 
> > What I mean is, of course, if an entry in one feed has the same ID
> > as an entry in another, but the user has not declared that he
> > trusts one feed to be republishing entries from the other, then
> > you should be paranoid.
> > 
> 
> As you said there is no reliable way to detect this, so i think liferea
> should not do anything to try to detect it, it will be hackish at more.
> 
> This particular case should not interfere on the descussion on how to
> display multiple similar feeds.

This is correct, but both ideas share a similar implementation
with in my opininion quite some effort and significant runtime
overhead to match newly downloaded items against all cached
items. Of course this will be faster with a DB backend, but
even if it would take parts seconds for each item it sums up
when updating for example 5 feeds resulting in 30 new items.
Also this could cause pretty CPU intense operations.

I think such specialised data-mining features are have a bad
development+runtime effort versus usefulness ratio.

> > > In that case, if it’s a republisher, and the entry you received
> > > differs from the one you got from the original source, then you
> > > can refresh the original feed and check what that looks like.
> > 
> > And by this I mean:
> > 
> > * feed f1 has entry e1 with ID xyz
> > * feed f2 has an entry e2 different from e1, but also with ID xyz
> 
> What i suggest to detect same unique id (that's weird :) ) on different
> posts, is to also take account of the title of the post. These two
> should be enough to determine if two items are identical or not. Maybe
> even the first of last characters.
> 
> For the UI, i don't want to see there are multiple same item, otherwise
> it's just like it is now. So i suggest in the unread vFolder only one
> item appear maybe with an icon indicating that it is has duplicates, or
> maybe with an arrow so you can expand and see the duplicate items.

I fear I might disappoint you now, but my experience
is that you cannot maintain this feature complexity level
with two developers currently spending about 1 hour a day.
My mantra: the feature set and complexity must match the 
development ressources. 

Lars

On dim, 2005-05-29 at 13:10 +0200, Lars Lindner wrote:
> > 
> > For the UI, i don't want to see there are multiple same item, otherwise
> > it's just like it is now. So i suggest in the unread vFolder only one
> > item appear maybe with an icon indicating that it is has duplicates, or
> > maybe with an arrow so you can expand and see the duplicate items.
> 
> I fear I might disappoint you now, but my experience
> is that you cannot maintain this feature complexity level
> with two developers currently spending about 1 hour a day.
> My mantra: the feature set and complexity must match the 
> development ressources. 


I perfectly understand this, cause i'm facing the same problem with my
various projects, it was just throwing an idea, not knowing how hard it
would be to implement.

That being said, if anyone comfortable with C could add that to
liferea :)

Raf

* Andreas Kohn <andreas@...> [2005-05-29 10:25]:
> Basically because I was stupid enough to create a feed with
> -draft-08 just to find that Liferea hated it :)

Heh, it is *called* do-not-deploy, dude. :-)

> So I guess best would be to 
> a) make it work with -draft-08
> b) as soon as the final RFC hits my mailbox, start working on
>    polishing the changed parts for 1.0

That sounds fine. Thinking about it, there are some changes in
1.0 that might constitute architectural changes in an aggregator
that follows the Atom model (as I said: multiple authors will be
allowed), but Liferea doesn’t, so I suppose implementing
-draft-08 and incrementally updating for 1.0 would work.

If you want, I can comb the atom-syntax list to see exactly which
changes had consensus since -draft-08 was posted, so that you
have an idea which things not to make too difficult to change?

Regards,
-- 
Aristotle

* Lars Lindner <lars.lindner@...> [2005-05-29 13:20]:
> This is correct, but both ideas share a similar implementation
> with in my opininion quite some effort and significant runtime
> overhead to match newly downloaded items against all cached
> items.

Not if all you are looking for is the GUI or the atom:id, rather
than some computed value such as a checksum. Though even in that
case, you could store the computed checksum in the backend files
and just run a lookup.

> Of course this will be faster with a DB backend, but even if it
> would take parts seconds for each item it sums up when updating
> for example 5 feeds resulting in 30 new items. Also this could
> cause pretty CPU intense operations.

With a DB backend it will, in fact, be trivial. It actually takes
only a single query for any number of new items if you check them
all at once.

I don’t see how this can be at all CPU-intensive.

> > What i suggest to detect same unique id (that's weird :) ) on
> > different posts, is to also take account of the title of the
> > post. These two should be enough to determine if two items
> > are identical or not. Maybe even the first of last
> > characters.

I already said that. It’s necessary only once you detect
duplicate IDs anyway.

> > For the UI, i don't want to see there are multiple same item,
> > otherwise it's just like it is now. So i suggest in the
> > unread vFolder only one item appear maybe with an icon
> > indicating that it is has duplicates, or maybe with an arrow
> > so you can expand and see the duplicate items.

If Liferea’s backend had been written item-centric rather than
feed-centric, any solution would have been trivial, in fact.

What I really like is the feature some other aggregators offer,
which keep a history of *all* versions of any entry, and you can
page through them, or even pull up diffs. Doing that cross-feed
would be difficult with the way Liferea is set up, though.

Regards,
-- 
Aristotle

On Sun, 2005-05-29 at 16:17 +0200, A. Pagaltzis wrote:
> * Andreas Kohn <andreas@...> [2005-05-29 10:25]:
> > Basically because I was stupid enough to create a feed with
> > -draft-08 just to find that Liferea hated it :)
>=20
> Heh, it is *called* do-not-deploy, dude. :-)

Gotcha...I'm used to skip over these sections where they say "don't use,
it's just for fun" :)

> > So I guess best would be to=20
> > a) make it work with -draft-08
> > b) as soon as the final RFC hits my mailbox, start working on
> >    polishing the changed parts for 1.0
>=20
> That sounds fine. Thinking about it, there are some changes in
> 1.0 that might constitute architectural changes in an aggregator
> that follows the Atom model (as I said: multiple authors will be
> allowed), but Liferea doesn=E2=80=99t, so I suppose implementing
> -draft-08 and incrementally updating for 1.0 would work.
>=20
> If you want, I can comb the atom-syntax list to see exactly which
> changes had consensus since -draft-08 was posted, so that you
> have an idea which things not to make too difficult to change?

I have to read both things anyway (my test-feed was hacked by just
reading the parts I needed), but yeah, such a change-list could be
helpful.

Andreas
--=20