Re: Release timestamps for archaic tarballs

[Daniel J S. <dan...@ie...>]

On 10/16/2017 12:03 AM, Eric S. Raymond wrote:
> Daniel J Sebald <dan...@ie...>:
>> If you provide the git repository from the frozen CVS repository, I can
>> clone the repository and generate the authorship map then post that file
>> somewhere.  Alternatively, you can compile the utility and follow the
>> directions described in "git_changelog_author --help".
> 
> It may be superfluous now.  I started thinking about what you and
> broeker were telling me about analyzing diffs and ended up using a similar
> technique to implement parsing each ChangeLog blog for authorship info to
> apply to the clique it's in.
> 
> I'll interleave a description of my algorithm with yours.
> 
>> Let me describe the algorithm, for Ethan's benefit, then give an example.
>>
>> The utility is quite simple.  It's main approach is: for each changeset
>> where there is a modification to ChangeLog
>>
>> MAIN ALGORITHM:
>>
>> Search for the first change in ChangeLog diff-hunk.  From that point search
>> backward for the first authorship line (which typically is the current line
>> because a new FSF entry is being added to the ChangeLog).
> 
> Yup, this is my first stage as well.  Because I'm looking at the blob
> (the entire ChangeLog state associated with the changeset) rather than
> a diff, I can just grab the top (most recent) entry header
> 
>> EXCEPTIONS:
>>
>> 1) The authorship line cannot be a subtraction, i.e., can't have a minus
>> sign as the first character.  The reason is that is typically a scenario
>> with a wholesale swap of the ChangeLog file, for example if ChangeLog is
>> renamed ChangeLog.1 and a new blank ChangeLog is created, or it might be a
>> corrected authorship line.  Such situations are left attributed to the
>> committer, usually Ethan (sfeam).
> 
> I'm not checking for this case specially.  Probably the right thing to do
> is not analyze if *all* the file paths in the commit are ChangeLogs.
> 
> I just implemented this.
> 
>> 2) If the ChangeLog file is the *only* file that changes in the changeset,
>> it is left out of the authorship map file.  These are typically cases where
>> Ethan goes back to clean up some comments.  In such scenarios it is likely
>> that the above algorithm searches backward to some FSF author line that
>> really isn't pertinent.  It might make sense in some cases because it goes
>> back to the original author.  But it could also be random.  My thinking is
>> that it is best to leave clean-up modifications attributed to the committer.
> 
> Above rule filters out these too.
> 
>> To my way of thinking, the one problem this won't catch is where the
>> original ChangeLog entry had an error in the authorship line itself and then
>> Ethan went back and corrected the authorship line, because such a scenario
>> will have '-' in the first column of the first changed line and a '+' in the
>> first column of the second changed line.  This is discarded, hence the
>> original incorrect authorship (misspelled, wrong email, whatever) will
>> remain in the authorship map.  If such a scenario exists in the map and
>> someone finds the uncorrected authorship in the future, as Mojca pointed
>> out, that can be fixed by hand.
> 
> Yeah, this can't realistically be detected in the metadata representation I'm
> traversing, either.

OK, we're on the same page.  There are all sorts of conditions I could 
imagine having accounted for, but I think at some point its covering 
corner cases that become increasingly unlikely.  We'll have opportunity 
to review how well the authorship has worked out.


>> I used word-count utility to check how many lines are in each file. What the
>> above is telling us is that there are 6037 changesets in which the ChangeLog
>> was modified in which '-' was not in the first column of the authorship
>> line.  Of those, 405 (i.e., 6037 - 5632) changesets involved only the
>> ChangeLog.
> 
> In 13202 commits, my technique finds 8245 ChangeLog blobs of which it uses
> 8222.  So all but 25 ChangeLogs yield usable attributions, and my method
> picks up about 1200 more attributions than yours.

But I'm not using a good translation; it's Mojca's automated cvs-import 
repository which we know has issues.  That repository is pretty far 
behind the current CVS repository, and doesn't pass its checks.  I.e.,

sebald@ ~/test_repository $ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (78081/78081), done.
dangling blob e06a11799a6b06633e506611b184d15d31b71103
dangling blob 13ba2e4924d917df123084aa2ebb0931f9afd8b0

It's got dangling blobs; that can't be good.


> In 3137 cases the deduced author is different from the committer.  That's
> 23% of commits, which is a completely reasonable density of patches
> contributed by non-devs.

I couldn't determine the percent without some extra effort because the 
committer (domain) here is sfeam, markisch, broeker, etc., while the 
author (range) is the full names of those individuals.

Dan