Re: ChangeLog mining doesn't work

[Ethan A M. <sf...@us...>]

On Friday, 13 October, 2017 12:40:48 Eric S. Raymond wrote:
> Alas, automated ChangeLog mining to fill in author slots doesn't work
> well enough to be usable.
> 
> The algorithm I ended up implementing  digests the ChangeLogs into a
> set of tuples each consiring of a date stamp (with no time part), a
> contributor ID, and a set of paths.  It then walks through the entry
> list, looking for commits that match the path set.  Then it filters
> for commits close in time to the entry timestamp.
> 
> "Close" is by default 12 hours to either side of the commit stamp. 

I can believe that 12 hours from the nominal date stamp misses many
of my commits, given that if I label an entry with today's date
anything I do in the afternoon probably gets a next-day timestamp
because of the time-zone difference.  

Even so that seems like too low a hit rate on the match-up.
My experience when I've tried to match a reported bug to a likely
culpable patch by taking the date in the ChangeLog and manually 
searching for a matching repository commit date in "cvs view"
there is a high chance of finding it easily (same-day or next-day
match). Certainly I've not seen a 99% failure rate.

Wait, hang on - what do you mean by "contributor ID"?
Many of the patches I commit are from other contributors.
My name does not appear in the ChangeLog entry. The CVS repository
presumably lists me as the source of the commit and offers no hint
that the ChangeLog entry shows someone else as the contributor.
I think you can only match the filename and the commit time
(and with some human intelligence the correlation between the
description in the ChangeLog and the short text in the commit
message).  As a sanity check or additional guide it might be
reasonable to use a mapping of contributor name/email -> likely committer.

Another thought... What do you mean by "match the path set"?
CVS commits occur one-by-one, so there isn't really a "set" to search for.
I can easily believe that the set of filenames listed in the ChangeLog
omits some that were committed by at the same time by mistake or
intent (trivial change to comment or some such).  I think it makes sense 
only to search for a match to one file at a time, not a set of files.

Could there be an issue with mis-match of branch identifiers?
I.e. a patch is often applied to the main (development) branch first,
and then applied much later to the current release branch.
The ChangeLog entries and commit messages are often identical
but for the time stamp.  Could your automated search be confused by
finding a potential match in one branch but then comparing it to
a timestamp from a different branch?

	Ethan


> The
> window can be extended on the future side to allow for delay in
> merging patches.
> 
> $ reposurgeon "read ." "changelogs"
> reposurgeon: Fills 123 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 36"
> reposurgeon: Fills 468 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 72"
> reposurgeon: Fills 498 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 120"
> reposurgeon: Fills 515 of 49627 authorship slots from 4915 ChangeLog entries
> 
> The argument of changelogs is a count of hours to extend the
> closeness window by. As expected, increasing the window yields
> more matches.
> 
> However, even with a very long window the match rate never goes above
> 1% of commits. That is noise level.
> 
> I think the reason is hinted at by the large disparity (about 10:1)
> between commit cliques and ChangeLog entries.  What this tells us is
> that a typical ChangeLog entry corresponds not to one commit clique
> but to several.  There's no algorithmic way to know what the boundaries
> are.
> 
> If such annotations are going to be made they will need a human eye
> and hand comparing at each of 4916 ChangeLog entries against the
> commit history.
> 
> That could be done with reposurgeon; I ran some numbers and it's
> probably about 80 hours of hand-work. No thanks...
>