Menu
▾
▴
gnuplot-beta
|
From: <es...@th...> - 2017-10-13 12:56:43
|
You haven't heard from me in a couple of days because I decided to
build a ChangeLog parser into reposurgeon and have been working on
that.
It's not an easy problem, and I have doubts that the information
extracted that way will either cover a lot of commits or be of
useful quality.
Here's what my documentation now says about the feature:
Mine the latest version of every ChangeLog file for authorship data.
Assume such files have basenames beginning with 'ChangeLog', and that
they are in the format used by FSF projects: entry header lines begin
with YYYY-MM-DD and are followed by a fullname/address, entry file
path lists in the text are delimited by '*' and ':' and each list
always begins first on a line after whitespace.
The logic attempts to match commits by path manifest and date; the
ChangeLog from which an entry derives is added to its manifest.
Because of time zone issues, each entry date is allowed to match
commit dates 12 hours earlier or later in UTC.
To allow for lag in applying inbound patches, the forward window can be
extended by a length of time given as a command-line argument in hours;
the default is 0. Note that setting this high is likely to cause false
matches, especially on single-file commits.
When an commit matches an entry, the attribution and the date from the
entry are copied to the author list of the commit. This can happen
more than once per entry. Author attributions generated this way will
always have a timestamp of 00:00:00 with an offset of +0000.
These attributions are marginally better than nothing but should not
be greatly trusted.
Alas, the problems here are fundamental, stemming from inadequate data
representations, and cannot be fixed with clever programming.
I have written the ChangeLog parser stage and am working on the
logic to match ChangeLog entries to commits.
By the way, while working on this I found a bad date 2206-07-27
in term/lua/ChangeLog. That probably wants to be 2006; somebody
should fix it.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
A true libertarian supports free enterprise, opposes big business;
supports local self-government, opposes the nation-state; supports the
National Rifle Association, opposes the Pentagon. -- Edward Abbey
|
|
From: Daniel J S. <dan...@ie...> - 2017-10-13 17:08:45
|
On 10/13/2017 07:56 AM, Eric S. Raymond wrote: > You haven't heard from me in a couple of days because I decided to > build a ChangeLog parser into reposurgeon and have been working on > that. > > It's not an easy problem, and I have doubts that the information > extracted that way will either cover a lot of commits or be of > useful quality. I have done such a thing with the post git-converted repository already. Code is here: https://sourceforge.net/p/gnuplot/patches/763/ I believe it is rather accurate, more accurate than the method you are describing which attempts to pull info from the ChangeLog entries alone. The methodology in the link above has the added, important detail (particular to the gnuplot project) of working from the diff hunks of the ChangeLog file itself for every git-translated changeset. It hinges on the fact that Ethan, et al. have consistently modified the proper location in the ChangeLog file for every CVS modification they've made. It's that action over the years which makes it accurate, i.e., the method traces the incremental changes made by the maintainers. Parsing the ChangeLog on its own won't work very well because, as I pointed out, the entries have been generated piecemeal. Ethan, et al. would often make a mod that goes back several days or possibly weeks pertaining to some checkin attributed to various contributors. Chronological order is not maintained; the mod maintainers make can be weeks apart from the date listed in the ChangeLog entry. That's why relying on the particular mod the maintainers made is better. The utility generates a list similar to the following: 87ca52fd56cf60be023642ad2664390affd5d896 2017-09-03 Daniel J Sebxxx <xxxxxx@xxxxxx> 778fe3d377116dc6d8c4f855372795bea5339e62 2017-09-03 Ethan A Merrxxx <xxxxxx@xxxxxx> c67c414e1604b20c562451ca6a2d6e9dca71561f 2017-09-02 Ethan A Merrxxx <xxxxxx@xxxxxx> 357f0cf68d691f3f76aade64053a6791f2e28686 2017-09-01 Bastian Maerkixxx <xxxxxx@xxxxxx> 984f9ad6c62f1c3ade9a41b46b5e034a2cb01c0a 2017-08-31 Martin Satuxxx <xxxxxx@xxxxxx> a0b903cbcae615cef9c0c0e3942c78ab150e3c82 2017-08-24 Ethan A Merrxxx <xxxxxx@xxxxxx> I pointed out in a previous email that the date in the above list is of no value for the git repository. There will not be good correlation between the date above and the actual date of the git changeset associated with 87ca52fd56cf60be023642ad2664390affd5d896, for example. All that needs to be done is some automated git command that will associate the author information in the above list with the gitID changeset in the list. > By the way, while working on this I found a bad date 2206-07-27 > in term/lua/ChangeLog. That probably wants to be 2006; somebody > should fix it. Ethan will have to fix that, thanks. Dan |
|
From: Eric S. R. <es...@th...> - 2017-10-13 18:22:02
|
Daniel J Sebald <dan...@ie...>: > On 10/13/2017 07:56 AM, Eric S. Raymond wrote: > >You haven't heard from me in a couple of days because I decided to > >build a ChangeLog parser into reposurgeon and have been working on > >that. > > > >It's not an easy problem, and I have doubts that the information > >extracted that way will either cover a lot of commits or be of > >useful quality. > > I have done such a thing with the post git-converted repository already. > Code is here: > > https://sourceforge.net/p/gnuplot/patches/763/ > > I believe it is rather accurate, more accurate than the method you are > describing which attempts to pull info from the ChangeLog entries alone. > > The methodology in the link above has the added, important detail > (particular to the gnuplot project) of working from the diff hunks of the > ChangeLog file itself for every git-translated changeset. It hinges on the > fact that Ethan, et al. have consistently modified the proper location in > the ChangeLog file for every CVS modification they've made. It's that action > over the years which makes it accurate, i.e., the method traces the > incremental changes made by the maintainers. > > Parsing the ChangeLog on its own won't work very well because, as I pointed > out, the entries have been generated piecemeal. Ethan, et al. would often > make a mod that goes back several days or possibly weeks pertaining to some > checkin attributed to various contributors. Chronological order is not > maintained; the mod maintainers make can be weeks apart from the date listed > in the ChangeLog entry. > > That's why relying on the particular mod the maintainers made is better. > > The utility generates a list similar to the following: > > 87ca52fd56cf60be023642ad2664390affd5d896 2017-09-03 Daniel J Sebxxx > <xxxxxx@xxxxxx> > 778fe3d377116dc6d8c4f855372795bea5339e62 2017-09-03 Ethan A Merrxxx > <xxxxxx@xxxxxx> > c67c414e1604b20c562451ca6a2d6e9dca71561f 2017-09-02 Ethan A Merrxxx > <xxxxxx@xxxxxx> > 357f0cf68d691f3f76aade64053a6791f2e28686 2017-09-01 Bastian Maerkixxx > <xxxxxx@xxxxxx> > 984f9ad6c62f1c3ade9a41b46b5e034a2cb01c0a 2017-08-31 Martin Satuxxx > <xxxxxx@xxxxxx> > a0b903cbcae615cef9c0c0e3942c78ab150e3c82 2017-08-24 Ethan A Merrxxx > <xxxxxx@xxxxxx> > > I pointed out in a previous email that the date in the above list is of no > value for the git repository. There will not be good correlation between > the date above and the actual date of the git changeset associated with > 87ca52fd56cf60be023642ad2664390affd5d896, for example. All that needs to be > done is some automated git command that will associate the author > information in the above list with the gitID changeset in the list. I see your point. Unfortunately, I can't integrate this technique into my conversion pipeline as-is, because (a) it relies, as you note, on a regularity in metadata specific to your project, and (b) My tools know nothing about SHA-1 IDs! Those are specific to an instantiated repository's hash chains. However, there may be a way around this. Can you generate your report with the hashes replaced by action stamps? An action-stamp is a kind of commit ID my tools understand. It has this form: <committer-id!yyyy-mm-ddThh:mm:ssZ> That is, it consists of a committer ID email address, followed by an RFC3339-format representation of the commit date. These are not necessarily unique per repository, but they usually are. This command will generate an action stamp from a SHA-1 ID: git log --format='<%cE!%cIZ>' -1 $1 -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> My work is funded by the Internet Civil Engineering Institute: https://icei.org Please visit their site and donate: the civilization you save might be your own. |
|
From: Daniel J S. <dan...@ie...> - 2017-10-13 18:44:19
|
On 10/13/2017 01:21 PM, Eric S. Raymond wrote: > Daniel J Sebald <dan...@ie...>: >> On 10/13/2017 07:56 AM, Eric S. Raymond wrote: >>> You haven't heard from me in a couple of days because I decided to >>> build a ChangeLog parser into reposurgeon and have been working on >>> that. >>> >>> It's not an easy problem, and I have doubts that the information >>> extracted that way will either cover a lot of commits or be of >>> useful quality. >> >> I have done such a thing with the post git-converted repository already. >> Code is here: >> >> https://sourceforge.net/p/gnuplot/patches/763/ >> >> I believe it is rather accurate, more accurate than the method you are >> describing which attempts to pull info from the ChangeLog entries alone. >> >> The methodology in the link above has the added, important detail >> (particular to the gnuplot project) of working from the diff hunks of the >> ChangeLog file itself for every git-translated changeset. It hinges on the >> fact that Ethan, et al. have consistently modified the proper location in >> the ChangeLog file for every CVS modification they've made. It's that action >> over the years which makes it accurate, i.e., the method traces the >> incremental changes made by the maintainers. >> >> Parsing the ChangeLog on its own won't work very well because, as I pointed >> out, the entries have been generated piecemeal. Ethan, et al. would often >> make a mod that goes back several days or possibly weeks pertaining to some >> checkin attributed to various contributors. Chronological order is not >> maintained; the mod maintainers make can be weeks apart from the date listed >> in the ChangeLog entry. >> >> That's why relying on the particular mod the maintainers made is better. >> >> The utility generates a list similar to the following: >> >> 87ca52fd56cf60be023642ad2664390affd5d896 2017-09-03 Daniel J Sebxxx >> <xxxxxx@xxxxxx> >> 778fe3d377116dc6d8c4f855372795bea5339e62 2017-09-03 Ethan A Merrxxx >> <xxxxxx@xxxxxx> >> c67c414e1604b20c562451ca6a2d6e9dca71561f 2017-09-02 Ethan A Merrxxx >> <xxxxxx@xxxxxx> >> 357f0cf68d691f3f76aade64053a6791f2e28686 2017-09-01 Bastian Maerkixxx >> <xxxxxx@xxxxxx> >> 984f9ad6c62f1c3ade9a41b46b5e034a2cb01c0a 2017-08-31 Martin Satuxxx >> <xxxxxx@xxxxxx> >> a0b903cbcae615cef9c0c0e3942c78ab150e3c82 2017-08-24 Ethan A Merrxxx >> <xxxxxx@xxxxxx> >> >> I pointed out in a previous email that the date in the above list is of no >> value for the git repository. There will not be good correlation between >> the date above and the actual date of the git changeset associated with >> 87ca52fd56cf60be023642ad2664390affd5d896, for example. All that needs to be >> done is some automated git command that will associate the author >> information in the above list with the gitID changeset in the list. > > I see your point. Unfortunately, I can't integrate this technique into > my conversion pipeline as-is, because > > (a) it relies, as you note, on a regularity in metadata specific to your > project, and > > (b) My tools know nothing about SHA-1 IDs! Those are specific to an > instantiated repository's hash chains. Yes, it's particular to the repository; no transferring of lists across platforms, conversions, etc. > However, there may be a way around this. > > Can you generate your report with the hashes replaced by action stamps? > > An action-stamp is a kind of commit ID my tools understand. It has > this form: > > <committer-id!yyyy-mm-ddThh:mm:ssZ> > > That is, it consists of a committer ID email address, followed by an > RFC3339-format representation of the commit date. > > These are not necessarily unique per repository, but they usually are. > > This command will generate an action stamp from a SHA-1 ID: > > git log --format='<%cE!%cIZ>' -1 $1 Oh, OK. Yeah, I've found a command that does it without even having to change the code (posted at the patch-tracker): git log --format='commit <%cE!%cIZ>' -p --unified=50 ChangeLog > ChangeLog.diff git_changelog_author ChangeLog.diff > author.txt The "commit" in the first column is what the utility is searching for. So the list then looks like this: <sfeam!2017-09-04T05:57:37+00:00Z> 2017-09-03 Daniel J Sebxxx <xxxxxx@xxxxxx> <sfeam!2017-09-03T23:28:50+00:00Z> 2017-09-03 Ethan A Merrxxx <xxxxxx@xxxxxx> <sfeam!2017-09-03T03:35:30+00:00Z> 2017-09-02 Ethan A Merrxxx <xxxxxx@xxxxxx> <markisch!2017-09-01T05:21:00+00:00Z> 2017-09-01 Bastian Maerkixxx <xxxxxx@xxxxxx> <sfeam!2017-08-31T23:03:55+00:00Z> 2017-08-31 Martin Satuxxx <xxxxxx@xxxxxx> <sfeam!2017-08-30T20:30:12+00:00Z> 2017-08-24 Ethan A Merrxxx <xxxxxx@xxxxxx> which has the person who committed the changeset and author. Nice. Dan |
|
From: Eric S. R. <es...@th...> - 2017-10-13 19:11:16
|
Daniel J Sebald <dan...@ie...>: > Oh, OK. Yeah, I've found a command that does it without even having to > change the code (posted at the patch-tracker): > > git log --format='commit <%cE!%cIZ>' -p --unified=50 ChangeLog > > ChangeLog.diff > git_changelog_author ChangeLog.diff > author.txt > > The "commit" in the first column is what the utility is searching for. So > the list then looks like this: > > <sfeam!2017-09-04T05:57:37+00:00Z> 2017-09-03 Daniel J Sebxxx > <xxxxxx@xxxxxx> > <sfeam!2017-09-03T23:28:50+00:00Z> 2017-09-03 Ethan A Merrxxx > <xxxxxx@xxxxxx> > <sfeam!2017-09-03T03:35:30+00:00Z> 2017-09-02 Ethan A Merrxxx > <xxxxxx@xxxxxx> > <markisch!2017-09-01T05:21:00+00:00Z> 2017-09-01 Bastian Maerkixxx > <xxxxxx@xxxxxx> > <sfeam!2017-08-31T23:03:55+00:00Z> 2017-08-31 Martin Satuxxx <xxxxxx@xxxxxx> > <sfeam!2017-08-30T20:30:12+00:00Z> 2017-08-24 Ethan A Merrxxx > <xxxxxx@xxxxxx> > > which has the person who committed the changeset and author. Nice. OK, I can easily massage that report into a set of reposurgeon commands that will patch in the author fields. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> My work is funded by the Internet Civil Engineering Institute: https://icei.org Please visit their site and donate: the civilization you save might be your own. |
|
From: Ethan A M. <sf...@us...> - 2017-10-13 18:50:56
|
On Friday, 13 October, 2017 08:56:36 Eric S. Raymond wrote: > By the way, while working on this I found a bad date 2206-07-27 > in term/lua/ChangeLog. That probably wants to be 2006; somebody > should fix it. There you open a can of worms I'd just as soon leave closed. That file no longer exists in the set of files obtained by generic cvs checkout. Yes it can be found in the repository as a "dead file", but I don't think it's worth my time to go mucking about in the graveyard of dead files to correct typos. Ethan |
|
From: Eric S. R. <es...@th...> - 2017-10-13 19:11:51
|
Ethan A Merritt <sf...@us...>: > On Friday, 13 October, 2017 08:56:36 Eric S. Raymond wrote: > > By the way, while working on this I found a bad date 2206-07-27 > > in term/lua/ChangeLog. That probably wants to be 2006; somebody > > should fix it. > > There you open a can of worms I'd just as soon leave closed. > That file no longer exists in the set of files obtained by generic > cvs checkout. Yes it can be found in the repository as a "dead file", > but I don't think it's worth my time to go mucking about in the > graveyard of dead files to correct typos. OK. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> My work is funded by the Internet Civil Engineering Institute: https://icei.org Please visit their site and donate: the civilization you save might be your own. |
|
From: Daniel J S. <dan...@ie...> - 2017-10-13 20:14:57
|
On 10/13/2017 01:48 PM, Ethan A Merritt via gnuplot-beta wrote: > On Friday, 13 October, 2017 08:56:36 Eric S. Raymond wrote: > > By the way, while working on this I found a bad date 2206-07-27 > > in term/lua/ChangeLog. That probably wants to be 2006; somebody > > should fix it. > > There you open a can of worms I'd just as soon leave closed. > That file no longer exists in the set of files obtained by generic > cvs checkout. Yes it can be found in the repository as a "dead file", > but I don't think it's worth my time to go mucking about in the > graveyard of dead files to correct typos. > > Ethan What files no longer exist? The 2206 currently appears in the file ChangeLog.1 of the current CVS repository. If you were to change that file, I don't think it would have any ramifications on the git conversion. That date info wouldn't be used for anything. The 2206 still appears in the historical repository record; no changing that. But again, those dates aren't used for anything in the conversion scheme. I would say that making any changes to ChangeLog files--typos, dates, etc.--is better done before the git conversion than after. Do we really want to be doing such changes under the new git repository? I can imagine regrouping the ChangeLog files, even deleting the ChangeLog files if later you think git log -p ChangeLog | less is sufficient for searching old ChangeLog mods. (Can always retrieve old ChangeLog files for local use.) But to be cleaning up ChangeLog files that are sort of deprecated under the new source control seems odd. It might be better freezing them at that point. Dan |
