Re: ChangeLog mining doesn't work

[Eric S. R. <es...@th...>]

Ethan A Merritt <sf...@us...>:
> I can believe that 12 hours from the nominal date stamp misses many
> of my commits, given that if I label an entry with today's date
> anything I do in the afternoon probably gets a next-day timestamp
> because of the time-zone difference.  
> 
> Even so that seems like too low a hit rate on the match-up.
> My experience when I've tried to match a reported bug to a likely
> culpable patch by taking the date in the ChangeLog and manually 
> searching for a matching repository commit date in "cvs view"
> there is a high chance of finding it easily (same-day or next-day
> match). Certainly I've not seen a 99% failure rate.
> 
> Wait, hang on - what do you mean by "contributor ID"?

In this entry

2009-03-12  Ethan A Merritt  <merritt@u.washington.edu>

	* src/fit.c (fit_command): Replace bogus initialization of dummy_token[]
	with explicit declaration.  Bug #2657599 

the contributor-ID is everything on the header line after the date stamp.

> I think you can only match the filename and the commit time

And that's the code I wrote does.  The committer-ID isn't used to *find*
a match between log entry and commit, it's used to fill the author field
of the commit *if an entry match on timestamp and pathset is found*.

> Another thought... What do you mean by "match the path set"?
> CVS commits occur one-by-one, so there isn't really a "set" to search for.
> I can easily believe that the set of filenames listed in the ChangeLog
> omits some that were committed by at the same time by mistake or
> intent (trivial change to comment or some such).  I think it makes sense 
> only to search for a match to one file at a time, not a set of files.

Under that assumption the entire procedure is doomed.

The heavy lifting in a CVS-to-git conversion is precisely the part where
you recognize CVS single-file commits and group them into git changesets.
You do this by recognizing identical committer metadata and comments
and timestamps that are within a certain window of time difference, usually
15 minutes.

By the time the Changelog-mining code sees the repository those changesets
have already been grouped.  What you want to do is set authorship for
the *changesets*, not the compoment per-file commits that don't
actually exist any more.

What I've found out is that the ChangeLogs themselves are not sufficient
for that.  Daniel Sebald is working on a different approach based on examining
diffs that he thinks might have better results.

> Could there be an issue with mis-match of branch identifiers?
> I.e. a patch is often applied to the main (development) branch first,
> and then applied much later to the current release branch.
> The ChangeLog entries and commit messages are often identical
> but for the time stamp.  Could your automated search be confused by
> finding a potential match in one branch but then comparing it to
> a timestamp from a different branch?

It looks for every potential match that is close enough in time,
so one of those would probably get filled in but not the other.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.