gnuplot-beta

ChangeLog mining doesn't work

[<es...@th...>]

Alas, automated ChangeLog mining to fill in author slots doesn't work
well enough to be usable.

The algorithm I ended up implementing  digests the ChangeLogs into a
set of tuples each consiring of a date stamp (with no time part), a
contributor ID, and a set of paths.  It then walks through the entry
list, looking for commits that match the path set.  Then it filters
for commits close in time to the entry timestamp.

"Close" is by default 12 hours to either side of the commit stamp.  The
window can be extended on the future side to allow for delay in
merging patches.

$ reposurgeon "read ." "changelogs"
reposurgeon: Fills 123 of 49627 authorship slots from 4915 ChangeLog entries
$ reposurgeon "read ." "changelogs 36"
reposurgeon: Fills 468 of 49627 authorship slots from 4915 ChangeLog entries
$ reposurgeon "read ." "changelogs 72"
reposurgeon: Fills 498 of 49627 authorship slots from 4915 ChangeLog entries
$ reposurgeon "read ." "changelogs 120"
reposurgeon: Fills 515 of 49627 authorship slots from 4915 ChangeLog entries

The argument of changelogs is a count of hours to extend the
closeness window by. As expected, increasing the window yields
more matches.

However, even with a very long window the match rate never goes above
1% of commits. That is noise level.

I think the reason is hinted at by the large disparity (about 10:1)
between commit cliques and ChangeLog entries.  What this tells us is
that a typical ChangeLog entry corresponds not to one commit clique
but to several.  There's no algorithmic way to know what the boundaries
are.

If such annotations are going to be made they will need a human eye
and hand comparing at each of 4916 ChangeLog entries against the
commit history.

That could be done with reposurgeon; I ran some numbers and it's
probably about 80 hours of hand-work. No thanks...
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

A true libertarian supports free enterprise, opposes big business;
supports local self-government, opposes the nation-state; supports the
National Rifle Association, opposes the Pentagon.	-- Edward Abbey

Re: ChangeLog mining doesn't work

[Daniel J S. <dan...@ie...>]

On 10/13/2017 11:40 AM, Eric S. Raymond wrote:
> Alas, automated ChangeLog mining to fill in author slots doesn't work
> well enough to be usable.
> 
> The algorithm I ended up implementing  digests the ChangeLogs into a
> set of tuples each consiring of a date stamp (with no time part), a
> contributor ID, and a set of paths.  It then walks through the entry
> list, looking for commits that match the path set.  Then it filters
> for commits close in time to the entry timestamp.
> 
> "Close" is by default 12 hours to either side of the commit stamp.  The
> window can be extended on the future side to allow for delay in
> merging patches.
> 
> $ reposurgeon "read ." "changelogs"
> reposurgeon: Fills 123 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 36"
> reposurgeon: Fills 468 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 72"
> reposurgeon: Fills 498 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 120"
> reposurgeon: Fills 515 of 49627 authorship slots from 4915 ChangeLog entries
> 
> The argument of changelogs is a count of hours to extend the
> closeness window by. As expected, increasing the window yields
> more matches.
> 
> However, even with a very long window the match rate never goes above
> 1% of commits. That is noise level.
> 
> I think the reason is hinted at by the large disparity (about 10:1)
> between commit cliques and ChangeLog entries.  What this tells us is
> that a typical ChangeLog entry corresponds not to one commit clique
> but to several.  There's no algorithmic way to know what the boundaries
> are.

The necessary information is in the ChangeLog diff hunks of the 
repository itself.


> If such annotations are going to be made they will need a human eye
> and hand comparing at each of 4916 ChangeLog entries against the
> commit history.

No, see the automated utility I mentioned, which I believe is accurate 
and reasonable.  The list ends up being 6132 entries.  That suggests 
that on average 20% of the time the maintainers went back to a previous 
ChangeLog entry making changes.

Dan

Re: ChangeLog mining doesn't work

[Ethan A M. <sf...@us...>]

On Friday, 13 October, 2017 12:40:48 Eric S. Raymond wrote:
> Alas, automated ChangeLog mining to fill in author slots doesn't work
> well enough to be usable.
> 
> The algorithm I ended up implementing  digests the ChangeLogs into a
> set of tuples each consiring of a date stamp (with no time part), a
> contributor ID, and a set of paths.  It then walks through the entry
> list, looking for commits that match the path set.  Then it filters
> for commits close in time to the entry timestamp.
> 
> "Close" is by default 12 hours to either side of the commit stamp. 

I can believe that 12 hours from the nominal date stamp misses many
of my commits, given that if I label an entry with today's date
anything I do in the afternoon probably gets a next-day timestamp
because of the time-zone difference.  

Even so that seems like too low a hit rate on the match-up.
My experience when I've tried to match a reported bug to a likely
culpable patch by taking the date in the ChangeLog and manually 
searching for a matching repository commit date in "cvs view"
there is a high chance of finding it easily (same-day or next-day
match). Certainly I've not seen a 99% failure rate.

Wait, hang on - what do you mean by "contributor ID"?
Many of the patches I commit are from other contributors.
My name does not appear in the ChangeLog entry. The CVS repository
presumably lists me as the source of the commit and offers no hint
that the ChangeLog entry shows someone else as the contributor.
I think you can only match the filename and the commit time
(and with some human intelligence the correlation between the
description in the ChangeLog and the short text in the commit
message).  As a sanity check or additional guide it might be
reasonable to use a mapping of contributor name/email -> likely committer.

Another thought... What do you mean by "match the path set"?
CVS commits occur one-by-one, so there isn't really a "set" to search for.
I can easily believe that the set of filenames listed in the ChangeLog
omits some that were committed by at the same time by mistake or
intent (trivial change to comment or some such).  I think it makes sense 
only to search for a match to one file at a time, not a set of files.

Could there be an issue with mis-match of branch identifiers?
I.e. a patch is often applied to the main (development) branch first,
and then applied much later to the current release branch.
The ChangeLog entries and commit messages are often identical
but for the time stamp.  Could your automated search be confused by
finding a potential match in one branch but then comparing it to
a timestamp from a different branch?

	Ethan


> The
> window can be extended on the future side to allow for delay in
> merging patches.
> 
> $ reposurgeon "read ." "changelogs"
> reposurgeon: Fills 123 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 36"
> reposurgeon: Fills 468 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 72"
> reposurgeon: Fills 498 of 49627 authorship slots from 4915 ChangeLog entries
> $ reposurgeon "read ." "changelogs 120"
> reposurgeon: Fills 515 of 49627 authorship slots from 4915 ChangeLog entries
> 
> The argument of changelogs is a count of hours to extend the
> closeness window by. As expected, increasing the window yields
> more matches.
> 
> However, even with a very long window the match rate never goes above
> 1% of commits. That is noise level.
> 
> I think the reason is hinted at by the large disparity (about 10:1)
> between commit cliques and ChangeLog entries.  What this tells us is
> that a typical ChangeLog entry corresponds not to one commit clique
> but to several.  There's no algorithmic way to know what the boundaries
> are.
> 
> If such annotations are going to be made they will need a human eye
> and hand comparing at each of 4916 ChangeLog entries against the
> commit history.
> 
> That could be done with reposurgeon; I ran some numbers and it's
> probably about 80 hours of hand-work. No thanks...
> 

Re: ChangeLog mining doesn't work

[Eric S. R. <es...@th...>]

Ethan A Merritt <sf...@us...>:
> I can believe that 12 hours from the nominal date stamp misses many
> of my commits, given that if I label an entry with today's date
> anything I do in the afternoon probably gets a next-day timestamp
> because of the time-zone difference.  
> 
> Even so that seems like too low a hit rate on the match-up.
> My experience when I've tried to match a reported bug to a likely
> culpable patch by taking the date in the ChangeLog and manually 
> searching for a matching repository commit date in "cvs view"
> there is a high chance of finding it easily (same-day or next-day
> match). Certainly I've not seen a 99% failure rate.
> 
> Wait, hang on - what do you mean by "contributor ID"?

In this entry

2009-03-12  Ethan A Merritt  <merritt@u.washington.edu>

	* src/fit.c (fit_command): Replace bogus initialization of dummy_token[]
	with explicit declaration.  Bug #2657599 

the contributor-ID is everything on the header line after the date stamp.

> I think you can only match the filename and the commit time

And that's the code I wrote does.  The committer-ID isn't used to *find*
a match between log entry and commit, it's used to fill the author field
of the commit *if an entry match on timestamp and pathset is found*.

> Another thought... What do you mean by "match the path set"?
> CVS commits occur one-by-one, so there isn't really a "set" to search for.
> I can easily believe that the set of filenames listed in the ChangeLog
> omits some that were committed by at the same time by mistake or
> intent (trivial change to comment or some such).  I think it makes sense 
> only to search for a match to one file at a time, not a set of files.

Under that assumption the entire procedure is doomed.

The heavy lifting in a CVS-to-git conversion is precisely the part where
you recognize CVS single-file commits and group them into git changesets.
You do this by recognizing identical committer metadata and comments
and timestamps that are within a certain window of time difference, usually
15 minutes.

By the time the Changelog-mining code sees the repository those changesets
have already been grouped.  What you want to do is set authorship for
the *changesets*, not the compoment per-file commits that don't
actually exist any more.

What I've found out is that the ChangeLogs themselves are not sufficient
for that.  Daniel Sebald is working on a different approach based on examining
diffs that he thinks might have better results.

> Could there be an issue with mis-match of branch identifiers?
> I.e. a patch is often applied to the main (development) branch first,
> and then applied much later to the current release branch.
> The ChangeLog entries and commit messages are often identical
> but for the time stamp.  Could your automated search be confused by
> finding a potential match in one branch but then comparing it to
> a timestamp from a different branch?

It looks for every potential match that is close enough in time,
so one of those would probably get filled in but not the other.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.