gnuplot-beta

Roundup of GNUPLOT conversion issues

[<es...@th...>]

The latest GNUPLOT conversion-machinery tarball, available as usual at

    wget http://www.catb.org/~esr/gnuplot-conversion.tar.gz

cleans up a number of minor issues, including:

(1) Patching out CVS release numbers in comments.

(2) Better explanatory text and comments for the very early section of
the repository.

(3) Patch that incorrect author date due to a ChangeLog typo.

I shall round up my responses to all recent traffic and then propose
what to do next.

Bastian Märkisch:
>Unfortunately, the last problem we discussed is not solved, though:
>If there's an insertion into the ChangeLog below an existing author line,
>the current algorithm picks the author line _below_ instead of the one
>_above_. This happens quite frequently and currently still leads to hundreds
>of wrong attributions.
>For examples try:
>  git log --author=Ethan --committer=Bastian --oneline | wc -l

I still need an example pair of Changelogs with a difference that
triggers this bug. Given that, I can (a) fix the bug, and (b) add it
to my regression tedt so we can be certain it does not recur.

Daniel Sebald, Fri Oct 27 12:50:51 2017
>                                                   These files with the
>wrong RCSid and the demo-file discrepancies and the backward-progressing
>copyright dates reside in the master branch *until* some changeset resolves
>them.  In the case of 3.7.x branch, there are a hundred "corrupted" files
>still, but shortly after 3.7.0 was completed there was a mass restructuring
>which moved files like alloc.c to a new subdirectory.  That action resulted in
>the correct files going forward, such that 4.x and 5.x series all match
>well...except for the 4.0 shim that Eric noted, probably a vestige still of
>what I described.

>So, we need to go back to the start of the repository and make sure those
>first few commits, the "Import of beta 340", etc. are done properly such that
>something like the following set of files all match in terms of stamped
>revisions and branches:

>How should we proceed Eric?

What I'm hearing is, exact for the demo shim I added, all the remaining
differences early in branches other than 3.7.x are incorrect RCS IDs.

What I think is that trying to get the RCS IDs perfect would be wasted
effort.  One of my last patches removes all of those, anyway.  It's
time to remember that the purpose of the repository transition is to
support development going forward, not to have a theoretically perfect
and literal transcription of the CVS history.

We wouldn't even *want* that kind of 'perfection' - the old-style
change comments wouldn't play well with git log, for one thing.

Daniel J Sebald, Sat Oct 28 14:48:42 2017
> I've dug into the actual ./reconvert process a bit.  I'm bewildered by some
> fundamental things regarding the CVS repository on
> 
> :pserver:ano...@gn...:/cvsroot/gnuplot
> 
> I assume that this repository is the same that is used to get the CVS copy
> used in the ./reconvert process
> 
> cvssync -c gnuplot.cvs.sourceforge.net:/cvsroot/gnuplot gnuplot
> mv gnuplot gnuplot-cvs
> 
> Those are the same repository, correct?

Yes.

> OK, I've been comparing the gnuplot-git repository against "cvs update
> -rGNUPLOT_3_7_0" and so on.  As I pointed out I've often gotten good matches
> in terms of code mods, with the exception being all these RCSid strings.

There is, however, that known problem with the 3.7.x tip revision, which has
non-RCSid differences between the tip and the tarball.

For reference, here is HBB's explanation.

    3.7.1 has one check-in to ChangeLog that's tagged but not tarred, and an extra
    pair of quotes around a line in configure.in that tarred but not tagged:

    diff -uwrp -x CVS cvs-3.7.1/ChangeLog gnuplot-3.7.1/ChangeLog
    --- cvs-3.7.1/ChangeLog 1999-11-01 13:33:32.000000000 +0100
    +++ gnuplot-3.7.1/ChangeLog     1999-10-21 16:15:07.000000000 +0200
    @@ -1,7 +1,3 @@
    -1999-11-01  Berthold Hoellmann  <ho...@ge...>
    -
    -       * docs/Makefile.in: Fix pdf target for compiling outside source dir.
    -
     1999-10-21  Lars Hecking  <lhe...@nm...>

	    * configure.in: Fix compile/link on NeXT without side effects ...
    diff -uwrp -x CVS cvs-3.7.1/configure gnuplot-3.7.1/configure
    --- cvs-3.7.1/configure 1999-10-27 13:00:15.000000000 +0200
    +++ gnuplot-3.7.1/configure     1999-11-07 16:57:22.000000000 +0100
    @@ -2591,7 +2591,7 @@ fi
     fi


    -if test $ac_cv_lib_nsl_gethostbyname = no; then
    +if test "$ac_cv_lib_nsl_gethostbyname" = no; then
       echo $ac_n "checking for gethostbyname in -lbsd""... $ac_c" 1>&6
     echo "configure:2597: checking for gethostbyname in -lbsd" >&5
     ac_lib_var=`echo bsd'_'gethostbyname | sed 'y%./+-%__p_%'`
    diff -uwrp -x CVS cvs-3.7.1/configure.in gnuplot-3.7.1/configure.in
    --- cvs-3.7.1/configure.in      1999-10-22 20:29:24.000000000 +0200
    +++ gnuplot-3.7.1/configure.in  1999-11-04 18:21:00.000000000 +0100
    @@ -115,7 +115,7 @@ AC_SUBST(GNUPLOT_X11)
     AC_PATH_XTRA

     dnl Needed for LynxOS until AC_PATH_XTRA is fixed
    -if test $ac_cv_lib_nsl_gethostbyname = no; then
    +if test "$ac_cv_lib_nsl_gethostbyname" = no; then
       AC_CHECK_LIB(bsd, gethostbyname, X_EXTRA_LIBS="$X_EXTRA_LIBS -lbsd")
     fi

    diff -uwrp -x CVS cvs-3.7.1/term/tgif.trm gnuplot-3.7.1/term/tgif.trm
    --- cvs-3.7.1/term/tgif.trm     1998-12-16 20:48:20.000000000 +0100
    +++ gnuplot-3.7.1/term/tgif.trm 1998-12-16 20:48:20.000000000 +0100
    @@ -371,7 +371,7 @@ TERM_PUBLIC void TGIF_init()
	 fprintf(gpoutfile, "\
     %%TGIF 2.15-p7\n\

    state(%d,30,%u,0,0,%u,16,1,9,1,1,0,0,0,0,1,0,'%s',0,%u,0,0,1,10,0,0,1,1,0,16,0,0,1,
    +1,1).\n\
    -%%\n%% @(#)$Header: /home/hbbro/prg/gp/backup/cvs/gnuplot/term/tgif.trm,v
    1.10 1998/12/16 19:48:20 lhecking Exp $\n%% %%W%%\n%%\n\
    +%%\n%% @(#)$Header: /var/tmp/CVSROOT/gnuplot/term/tgif.trm,v 1.10 1998/12/16
    19:48:20 lhecking Exp $\n%% %%W%%\n%%\n\
     page(1,\"\").\n",
		TgifPortrait ? 0 : 1, uActResolution, uActZoom, sActFont,
    uActFontSize);
	 eTgifState = NEWPOLY;

Dan again:
> Why are these strings not matching what is listed in the gnuplot-cvs files?
> 
> Perhaps this has something to do with this
> 
> # In order not to introduce noise into tree comparisons, you must specify
> the
> # argument 'finish' to do the part of the conversion that involves checking
> # in the FAQ and patching out the RCS cookies
> [snip]
> #
> # Also requires patch files COOKIEPATCH
> #
> 
> Is there something about the file COOKIEPATCH possibly missing some fixes
> needed for files tucked away in the Attic?

No, that's not it at all.  All COOKIEPATCH does is *remove all RCS cookies
from the tree* (well,except for one diff in the doc directory where I
couldn't figure out how to rip them out without damaging the diff).

> I agree these RCSid aren't that important, but it sure would be a nice
> feeling to have a match between, say, "cvs update -rXYZ" and "git checkout
> XYZ" from remote locations.

Too little gain for too much effort.  Once SourceForge shuts down CVS,
nobody will ever care about "cvs update -rXYZ" again.

Hans-Bernhard Bröker <HBB...@t-...>:
> >On the other hand, I'm not getting
> >a good feeling from Dan's reports on his attempts to disentangle that branch.
> >We know it's corrupt - doesn't have the right tip state.
> 
> I'm not quite convinced Dan's analysis is correct.  The branch isn't
> corrupt, it was just set up somewhat strangely at the time.

It's corrupt in the sense that because of the vendor-branch confusion we
not only can't guarantee that the gitspace sequence of commits really
matches the CVS development history, we can't even get the right tip
state out of the conversion.

The extent to which this is a CVS problem versus a git problem versus
a cvs-fast-export problem is really only of theoretical interest.

I've spent three weeks on this conversion and my patience for
wandering further off into the weeds is evaporating.  It only
lasted this long because getting ChangeLog analysis right seemed
like a thing worth doing.

Can we please refocus on *finishing* this with changes that actually add
value for forward development?

> Vendor branches with other branches partially rooted in them cannot be
> collapsed or ignored.  Nor can they be treated completely like ordinary
> branches (or conversion tools would have been doing that since day one).

I know.  Vendor branches have been a huge pain in the ass on other
conversions I've done, too.

Here is what I suggest we do:

1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
the archived 3.7.[0123] trees in sequence (I now have them all).

2. Fix any remaining deficiences in ChangeLog scanning if possible.

3. Fill in the empty log comments that are left.

4. Wrap it up and call it done.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Morality is always the product of terror; its chains and
strait-waistcoats are fashioned by those who dare not trust others,
because they dare not trust themselves, to walk in liberty.
	-- Aldous Huxley 

Re: Roundup of GNUPLOT conversion issues

[Daniel J S. <dan...@ie...>]

On 10/29/2017 12:30 AM, Eric S. Raymond wrote:
> The latest GNUPLOT conversion-machinery tarball, available as usual at
> 
>      wget http://www.catb.org/~esr/gnuplot-conversion.tar.gz
> 
> cleans up a number of minor issues, including:
> 
> (1) Patching out CVS release numbers in comments.
> 
> (2) Better explanatory text and comments for the very early section of
> the repository.
> 
> (3) Patch that incorrect author date due to a ChangeLog typo.
> 
> I shall round up my responses to all recent traffic and then propose
> what to do next.

I've been making some ./reconvert mods, supplying all the branch points 
as determined with cvs2git.  However, the branches at the start of the 
repository for importing beta340, etc. are going to need some work on 
your part.

I'll describe in a follow-up email.  But wait for me to get a diff file 
to you and we can put my mods in place.


> Bastian Märkisch:
>> Unfortunately, the last problem we discussed is not solved, though:
>> If there's an insertion into the ChangeLog below an existing author line,
>> the current algorithm picks the author line _below_ instead of the one
>> _above_. This happens quite frequently and currently still leads to hundreds
>> of wrong attributions.
>> For examples try:
>>   git log --author=Ethan --committer=Bastian --oneline | wc -l
> 
> I still need an example pair of Changelogs with a difference that
> triggers this bug. Given that, I can (a) fix the bug, and (b) add it
> to my regression tedt so we can be certain it does not recur.
> 
> Daniel Sebald, Fri Oct 27 12:50:51 2017
>>                                                    These files with the
>> wrong RCSid and the demo-file discrepancies and the backward-progressing
>> copyright dates reside in the master branch *until* some changeset resolves
>> them.  In the case of 3.7.x branch, there are a hundred "corrupted" files
>> still, but shortly after 3.7.0 was completed there was a mass restructuring
>> which moved files like alloc.c to a new subdirectory.  That action resulted in
>> the correct files going forward, such that 4.x and 5.x series all match
>> well...except for the 4.0 shim that Eric noted, probably a vestige still of
>> what I described.
> 
>> So, we need to go back to the start of the repository and make sure those
>> first few commits, the "Import of beta 340", etc. are done properly such that
>> something like the following set of files all match in terms of stamped
>> revisions and branches:
> 
>> How should we proceed Eric?
> 
> What I'm hearing is, exact for the demo shim I added, all the remaining
> differences early in branches other than 3.7.x are incorrect RCS IDs.
> 
> What I think is that trying to get the RCS IDs perfect would be wasted
> effort.  One of my last patches removes all of those, anyway.  It's
> time to remember that the purpose of the repository transition is to
> support development going forward, not to have a theoretically perfect
> and literal transcription of the CVS history.

I'm interested in retaining history.  Why shape it into a different 
history that no one is familiar with?  I'm fine with the RCS IDs not 
matching--Ethan (with my help), can branch a stub off of a release with 
bad RCS IDs and then put in the actual release code.  That will then 
produce the actual release code in the repository--for the price of just 
a few diffs.


> We wouldn't even *want* that kind of 'perfection' - the old-style
> change comments wouldn't play well with git log, for one thing.

Projects like Octave use FSF format

http://hg.savannah.gnu.org/hgweb/octave/rev/1680d425bb38

and it produces a nice log which highlights with color in the pager.


> Hans-Bernhard Bröker <HBB...@t-...>:
>>> On the other hand, I'm not getting
>>> a good feeling from Dan's reports on his attempts to disentangle that branch.
>>> We know it's corrupt - doesn't have the right tip state.
>>
>> I'm not quite convinced Dan's analysis is correct.  The branch isn't
>> corrupt, it was just set up somewhat strangely at the time.
> 
> It's corrupt in the sense that because of the vendor-branch confusion we
> not only can't guarantee that the gitspace sequence of commits really
> matches the CVS development history, we can't even get the right tip
> state out of the conversion.

cvs2git seems to handle the vendor-branch via a branch-and-merge 
approach.  (It only goes for a year or so, right?)  We'll come back to 
this, as I think it is the missing piece we need to deal with still.


> The extent to which this is a CVS problem versus a git problem versus
> a cvs-fast-export problem is really only of theoretical interest.
> 
> I've spent three weeks on this conversion and my patience for
> wandering further off into the weeds is evaporating.  It only
> lasted this long because getting ChangeLog analysis right seemed
> like a thing worth doing.

Yes, ChangeLog analysis is good.


> Can we please refocus on *finishing* this with changes that actually add
> value for forward development?

The only major thing still is that first dozen or so commits at the 
start.  I'm not sure, but that may be cvs-fast-export dependent. 
Otherwise, I'm getting the hang of reposurgeon and ./reconvert, so I can 
deal with verifying correctness of branches and rebuilding.


>> Vendor branches with other branches partially rooted in them cannot be
>> collapsed or ignored.  Nor can they be treated completely like ordinary
>> branches (or conversion tools would have been doing that since day one).
> 
> I know.  Vendor branches have been a huge pain in the ass on other
> conversions I've done, too.
> 
> Here is what I suggest we do:
> 
> 1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
> the archived 3.7.[0123] trees in sequence (I now have them all).

This is more work than just moving the branch to the correct location. 
cvs2git has chosen the branch point that I deduced by version/dates 
within the files, i.e., "New file."  I'll work this out.


> 2. Fix any remaining deficiences in ChangeLog scanning if possible.
> 
> 3. Fill in the empty log comments that are left.

We'll see if Ethan wants to do this.  He and I can split the work, but 
only if it is something easy to do with reposurgeon.

Dan

Re: Roundup of GNUPLOT conversion issues

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> I'm interested in retaining history.  Why shape it into a different history
> that no one is familiar with?

Seems to me like the release tarballs are the best-known history there is.

> >We wouldn't even *want* that kind of 'perfection' - the old-style
> >change comments wouldn't play well with git log, for one thing.
> 
> Projects like Octave use FSF format
> 
> http://hg.savannah.gnu.org/hgweb/octave/rev/1680d425bb38
> 
> and it produces a nice log which highlights with color in the pager.

That isn't actually FSF format at all.  It adopts two practices the
FSF never had: (1) having a headline separated by a blank line from the
body of the comment, and (2) restricting the length of that line so it
will fit in an 80-column window with 4 leading spaces to spare.

What I sean by "old-style" is what was commonly done in CVS and
Subversion.  That is, paragraphs of running text *without* a separated
headline (and often with no internal line breaks). Those don't play
well with git.

> >It's corrupt in the sense that because of the vendor-branch confusion we
> >not only can't guarantee that the gitspace sequence of commits really
> >matches the CVS development history, we can't even get the right tip
> >state out of the conversion.
> 
> cvs2git seems to handle the vendor-branch via a branch-and-merge approach.
> (It only goes for a year or so, right?)  We'll come back to this, as I think
> it is the missing piece we need to deal with still.

Does cvs2git get the tip content of 3.7.x right - that is, coincident with the
3.7.3 tarball?

> >1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
> >the archived 3.7.[0123] trees in sequence (I now have them all).
> 
> This is more work than just moving the branch to the correct location.
> cvs2git has chosen the branch point that I deduced by version/dates within
> the files, i.e., "New file."  I'll work this out.

I'm guessing you mean

<1998-04-11T19:43:11Z!lhe...@nm...>

which is the earliest commit with a "New file." comment. The commands you want
would probably be 

/Initial import of 3.6 beta340./ assign root37
@min(/New file.$/),<root37> reparent --use-order --rebase

except I suspect you want to sctually do this:

/Windows linestyle fix/ assign root37

and then lose the preceding reimports.

> >3. Fill in the empty log comments that are left.
> 
> We'll see if Ethan wants to do this.  He and I can split the work, but only
> if it is something easy to do with reposurgeon.

You two don't have to drive reposurgeon st all to do this.  Just fill in
comments in the enclosed file and send it back to me.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: One branchpoint discrepancy

[Hans-Bernhard B. <HBB...@t-...>]

Am 30.10.2017 um 20:30 schrieb Daniel J Sebald:

>  I'd sort of like to put effort into the right place.  I've cloned the
> cvs-fast-export utility and I'm willing to help on matters, so if its 
> possible I wonder if we could enhance the merging aspect of that utility.

I rather doubt any good will come from that.  Nobody understands that 
utility anywhere near well enough to be able to modify it on short 
notice without risking total break-down.  At this point it is, 
essentially, magic.

> Yes, there is a bit of ambiguity in merges with CVS, it just isn't 100% 
> accurate to identify exactly what the programmer had checked out when 
> compiling and subsequently did a checkin.

Not really, because there _are_ no recorded merges in CVS.  If merges 
happen, they do so in somebody's working copy.  To the repository, they 
only ever appear as check-ins, without any indication whether the new 
content was created by some kind of merge, or by just writing it manually.

> There might be some ambiguity about the state of other files at this 
> point, but I would say simply recall the BETA_344_989422 state from CVS 
> and call that the state for the merge and generate all git diffs 
> accordingly, perhaps that's not the exact methodology.

That won't work at all.  The primary conflict is that CVS has individual 
branch structures for every member file, whereas git branches the entire 
repository.  Those two concepts don't mix and match.  BETA_344_989422 
may be the right join point for this particular file, but that's most 
likely the _only_ file for which that's the case.  Basically every 
single tag ever made in CVS can contain one or more file joins from the 
vendor branch onto the trunk.  Some are still waiting to happen.

Normal CVS repositories would have every single file starting off at 
1.1.  I.e. the first tag would be on 1.1 revisions of every file, and 
all development would start from there.  The conversion tools have no 
problem at all with this set-up.

But our repository was started by a "cvs import", and received some 
further imports after that, and that changes everything.  It means that 
all our original files started at revision 1.1.1.1, and progressed along 
that 1.1.1.* branch, until they were first modified.  None of them ever 
got a tag on it 1.1 revision --- 1.1. was really never used for anything.

Every time a file that was on the vendor branch until that point (and 
remember, for some files that still hasn't happened today!) is checked 
in, that particular file is essentially merged over from the vendor 
branch onto the trunk --- but in the archive this merge appears as an 
ordinary check-in of a revision 1.2.

E.g. even though in a RCS revision tree, it appears like this:

	1.1 --*---> 1.2
	       \
	        +-> 1.1.1.1 --> 1.1.1.2

the real sequence of active revisions for gnuplot.rot is:

	1.1.1.1 --> 1.1.1.2 --> 1.2

Other files have different sequences, and different points in time at 
which they made their transition 1.1.1.1 --> 1.1.1.2, or 1.1.1.{n} --> 
1.2.  Some even have branches inside the vendor branch (1.1.1.2.2.1)

The vendor branch may best be grafted into the trunk _before_ the 1.1 
initial import, and replace that entirely.  I.e. we might imagine that 
the above sequence was transformed into

	1.0 --> 1.1 --> 1.2

Files with more stuff going on in their vendor branch would have to dip 
into negative numbers, i.e.

	1.1.1.1 --> 1.1.1.2 +-> 1.1.1.3 --> 1.1.1.4 --> 1.2 ...
	                    |
	                    +-> 1.1.1.2.2.1

would (imaginatively) turn into

	1.(-2)  --> 1.(-1)  +-> 1.0 ------> 1.1 ------> 1.2 ...
  	                    |
                             +-> 1.(-1).2.1

Realistically all the revision numbers in the entire archive would have 
to be shifted up such that the chain really does start at 1.1:

	1.1  -----> 1.2 --*---> 1.3 ------> 1.4 ------> 1.5 ...
  	                   \
                             +-> 1.2.2.1

Unfortunately, the way RCS ,v files are organized, this shift can only 
be performed by parsing and re-encoding every revision on the vendor 
branch.  (The direction the diffs are recorded is from the head all the 
way down to 1.1, and from there _up_ along the vendor branch).  And 
because of the way CVS uses RCS ,v files, every one of them has to be 
transformed individually.

_That_ is the transformation that needs to be done in order for 
conversion tools not to have any problems with the vendor branch.  And 
because the transformation differs for every RCS archive, it has to be 
done either directly on the CVS repository, or the importer has to 
pretend it had happened that way.

Let me reiterate: to the best of my understanding, no process working on 
an already converted git repository has any realistic chance to perform 
this operation correctly.  It has to be done on the CVS import side.

Re: One branchpoint discrepancy

[Eric S. R. <es...@th...>]

Hans-Bernhard Bröker <HBB...@t-...>:
> Am 30.10.2017 um 20:30 schrieb Daniel J Sebald:
> 
> > I'd sort of like to put effort into the right place.  I've cloned the
> >cvs-fast-export utility and I'm willing to help on matters, so if its
> >possible I wonder if we could enhance the merging aspect of that utility.
> 
> I rather doubt any good will come from that.  Nobody understands that
> utility anywhere near well enough to be able to modify it on short notice
> without risking total break-down.  At this point it is, essentially, magic.

Sadly, HBB is correct.  The way I modified Keith's original code was by
working from the outside in - there is a hard core, now in merge.c, that nobody
understands. 

> That won't work at all.  The primary conflict is that CVS has individual
> branch structures for every member file, whereas git branches the entire
> repository.  Those two concepts don't mix and match.  BETA_344_989422 may be
> the right join point for this particular file, but that's most likely the
> _only_ file for which that's the case.  Basically every single tag ever made
> in CVS can contain one or more file joins from the vendor branch onto the
> trunk.  Some are still waiting to happen.
> 
> Normal CVS repositories would have every single file starting off at 1.1.
> I.e. the first tag would be on 1.1 revisions of every file, and all
> development would start from there.  The conversion tools have no problem at
> all with this set-up.
> 
> But our repository was started by a "cvs import", and received some further
> imports after that, and that changes everything.  It means that all our
> original files started at revision 1.1.1.1, and progressed along that
> 1.1.1.* branch, until they were first modified.  None of them ever got a tag
> on it 1.1 revision --- 1.1. was really never used for anything.
> 
> Every time a file that was on the vendor branch until that point (and
> remember, for some files that still hasn't happened today!) is checked in,
> that particular file is essentially merged over from the vendor branch onto
> the trunk --- but in the archive this merge appears as an ordinary check-in
> of a revision 1.2.
>
> E.g. even though in a RCS revision tree, it appears like this:
> 
> 	1.1 --*---> 1.2
> 	       \
> 	        +-> 1.1.1.1 --> 1.1.1.2
> 
> the real sequence of active revisions for gnuplot.rot is:
> 
> 	1.1.1.1 --> 1.1.1.2 --> 1.2
> 
> Other files have different sequences, and different points in time at which
> they made their transition 1.1.1.1 --> 1.1.1.2, or 1.1.1.{n} --> 1.2.  Some
> even have branches inside the vendor branch (1.1.1.2.2.1)
> 
> The vendor branch may best be grafted into the trunk _before_ the 1.1
> initial import, and replace that entirely.  I.e. we might imagine that the
> above sequence was transformed into
> 
> 	1.0 --> 1.1 --> 1.2
> 
> Files with more stuff going on in their vendor branch would have to dip into
> negative numbers, i.e.
> 
> 	1.1.1.1 --> 1.1.1.2 +-> 1.1.1.3 --> 1.1.1.4 --> 1.2 ...
> 	                    |
> 	                    +-> 1.1.1.2.2.1
> 
> would (imaginatively) turn into
> 
> 	1.(-2)  --> 1.(-1)  +-> 1.0 ------> 1.1 ------> 1.2 ...
>  	                    |
>                             +-> 1.(-1).2.1
> 
> Realistically all the revision numbers in the entire archive would have to
> be shifted up such that the chain really does start at 1.1:
> 
> 	1.1  -----> 1.2 --*---> 1.3 ------> 1.4 ------> 1.5 ...
>  	                   \
>                             +-> 1.2.2.1
> 
> Unfortunately, the way RCS ,v files are organized, this shift can only be
> performed by parsing and re-encoding every revision on the vendor branch.
> (The direction the diffs are recorded is from the head all the way down to
> 1.1, and from there _up_ along the vendor branch).  And because of the way
> CVS uses RCS ,v files, every one of them has to be transformed individually.
> 
> _That_ is the transformation that needs to be done in order for conversion
> tools not to have any problems with the vendor branch.  And because the
> transformation differs for every RCS archive, it has to be done either
> directly on the CVS repository, or the importer has to pretend it had
> happened that way.
> 
> Let me reiterate: to the best of my understanding, no process working on an
> already converted git repository has any realistic chance to perform this
> operation correctly.  It has to be done on the CVS import side.

This is good analysis.  You have described the problem more clearly than
*I* undetstood it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: One branchpoint discrepancy

[Daniel J S. <dan...@ie...>]

On 10/31/2017 12:11 PM, Eric S. Raymond wrote:
> Hans-Bernhard Bröker <HBB...@t-...>:
>> Am 30.10.2017 um 20:30 schrieb Daniel J Sebald:
>>
>>> I'd sort of like to put effort into the right place.  I've cloned the
>>> cvs-fast-export utility and I'm willing to help on matters, so if its
>>> possible I wonder if we could enhance the merging aspect of that utility.
>>
>> I rather doubt any good will come from that.  Nobody understands that
>> utility anywhere near well enough to be able to modify it on short notice
>> without risking total break-down.  At this point it is, essentially, magic.
> 
> Sadly, HBB is correct.  The way I modified Keith's original code was by
> working from the outside in - there is a hard core, now in merge.c, that nobody
> understands.

I've looked at the code.  Very clean and well-written, actually, but I 
see the limitations.

Dan

Re: Roundup of GNUPLOT conversion issues

[Hans-Bernhard B. <HBB...@t-...>]

Am 29.10.2017 um 13:45 schrieb Eric S. Raymond:
> Daniel J Sebald <dan...@ie...>:
> 
> Does cvs2git get the tip content of 3.7.x right - that is, coincident with the
> 3.7.3 tarball?

Please note that the tip of the 3.7.* branch is not coincident with the 
3.7.3 tag or tarball.  Some further changes were made on the branch 
after 3.7.3.

The same goes for all of our release branches.  They all received a 
couple more changes after the last release was made from them.  So none 
of the branch tips coincides to any released tarball.  Comparisons 
really would have to be made to cvs checkouts of the branches, instead.

>>> 1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
>>> the archived 3.7.[0123] trees in sequence (I now have them all).

That would be slightly wrong because 3.7.0 is not on the branch.  It's 
on the trunk.

Re: One branchpoint discrepancy

[Daniel J S. <dan...@ie...>]

On 10/30/2017 05:52 PM, Hans-Bernhard Bröker wrote:
> Am 30.10.2017 um 20:30 schrieb Daniel J Sebald:
> 
>>  I'd sort of like to put effort into the right place.  I've cloned the
>> cvs-fast-export utility and I'm willing to help on matters, so if its 
>> possible I wonder if we could enhance the merging aspect of that utility.
> 
> I rather doubt any good will come from that.  Nobody understands that 
> utility anywhere near well enough to be able to modify it on short 
> notice without risking total break-down.  At this point it is, 
> essentially, magic.

It doesn't look like a very big program.


>> Yes, there is a bit of ambiguity in merges with CVS, it just isn't 
>> 100% accurate to identify exactly what the programmer had checked out 
>> when compiling and subsequently did a checkin.
> 
> Not really, because there _are_ no recorded merges in CVS.  If merges 
> happen, they do so in somebody's working copy.  To the repository, they 
> only ever appear as check-ins, without any indication whether the new 
> content was created by some kind of merge, or by just writing it manually.

I didn't state that right; I should have said "implied merges" or 
"psuedo-merges", i.e., the merges placed in the git translation by the 
conversion tool.  And the way something like cvs2git has an implied 
history for constructing those merges comes from the fact Lars placed 
what looks like three dozen or more tags in the work.  Without all those 
tags, it would be as you described below, simply a trail of numbers.


>> There might be some ambiguity about the state of other files at this 
>> point, but I would say simply recall the BETA_344_989422 state from 
>> CVS and call that the state for the merge and generate all git diffs 
>> accordingly, perhaps that's not the exact methodology.
> 
> That won't work at all.  The primary conflict is that CVS has individual 
> branch structures for every member file, whereas git branches the entire 
> repository.  Those two concepts don't mix and match.  BETA_344_989422 
> may be the right join point for this particular file, but that's most 
> likely the _only_ file for which that's the case.

It's not.  I believe there are several files that fall in that same 
category.  I can search for them, to verify, but that's a lot of work. 
But in terms of principle, generally BETA_344_989422 can appear in 
multiple files in the same way.  I sent screenshots of what cvs2git 
produces for merges, and it seems to produce a good job of matching CVS 
updates to various versions.  Here are the sorts of statements cvs2git 
is making:

"This commit was generated by cvs2svn to compensate for changes in r48, 
which included commits to RCS files with non-trunk default branches."

     BETA_344_989422:1.1.1.2  (this tag is in the trunk, but it's 
referencing a version that is in a branch different than the trunk)
     BETA_344:1.1.1.2         (this tag is in the branch)
     BETA_343_980416:1.1.1.1



>  Basically every 
> single tag ever made in CVS can contain one or more file joins from the 
> vendor branch onto the trunk.  Some are still waiting to happen.

That's correct, one or more.  And in every one of those merges back to 
the trunk (in cvs2git) there are probably a half dozen files that have 
that same pattern of being referenced by BETA_344_989422.  It's a 
subgroup of at least all these files I listed previously:

amiga.c
Copyright
ctrl87.c
ctrl87.h
demo/gnuplot.rot
docs/gpcard.tex
docs/old/ChangeLog.old  [This is RCSid diff, does not count]
docs/old/makefile.r     [Ditto]
docs/old/README.3p5     [Ditto]
fnproto.h
intergra.x11
os9.c
win/wgnuplib.c
win/wprinter.c
win/wresourc.h

and then there are a bunch of ancillary changes that are being made to 
many other files that keep the psuedo-merge in sync.


> Normal CVS repositories would have every single file starting off at 
> 1.1.  I.e. the first tag would be on 1.1 revisions of every file, and 
> all development would start from there.  The conversion tools have no 
> problem at all with this set-up.
> 
> But our repository was started by a "cvs import", and received some 
> further imports after that, and that changes everything.  It means that 
> all our original files started at revision 1.1.1.1, and progressed along 
> that 1.1.1.* branch, until they were first modified.  None of them ever 
> got a tag on it 1.1 revision --- 1.1. was really never used for anything.
> 
> Every time a file that was on the vendor branch until that point (and 
> remember, for some files that still hasn't happened today!) is checked 
> in, that particular file is essentially merged over from the vendor 
> branch onto the trunk --- but in the archive this merge appears as an 
> ordinary check-in of a revision 1.2.
> 
> E.g. even though in a RCS revision tree, it appears like this:
> 
>      1.1 --*---> 1.2
>             \
>              +-> 1.1.1.1 --> 1.1.1.2
> 
> the real sequence of active revisions for gnuplot.rot is:
> 
>      1.1.1.1 --> 1.1.1.2 --> 1.2
> 
> Other files have different sequences, and different points in time at 
> which they made their transition 1.1.1.1 --> 1.1.1.2, or 1.1.1.{n} --> 
> 1.2.  Some even have branches inside the vendor branch (1.1.1.2.2.1)

Correct, but the psuedo-merges don't really care about all the different 
times, just all the files associated with a particular tag, in this case 
BETA_344_989422.  I'm not sure, but I think cvs2git in this case makes a 
rough estimate of the time which it assigns to the merge, like maybe 
halfway between the latest age of any file and the next modification. 
When a psuedo-merge took place exactly isn't important, just that it 
fits sequentially in the proper location (i.e., it must have happened 
between A and B is the important part).


> The vendor branch may best be grafted into the trunk _before_ the 1.1 
> initial import, and replace that entirely.  I.e. we might imagine that 
> the above sequence was transformed into
> 
>      1.0 --> 1.1 --> 1.2

Yes, there are multiple ways to imagine this, but I think from a 
conversion tool's standpoint it has to pick the scenario, e.g., we're 
going to assume this is a master branch and then follow all these 
branches and merges according to the tags.  It's similar to what I said 
early on that even with git, once the heads of branches are merged, it's 
sort of difficult to figure out in hindsight which particular branch was 
associated with which head at the time.


> Files with more stuff going on in their vendor branch would have to dip 
> into negative numbers, i.e.
> 
>      1.1.1.1 --> 1.1.1.2 +-> 1.1.1.3 --> 1.1.1.4 --> 1.2 ...
>                          |
>                          +-> 1.1.1.2.2.1
> 
> would (imaginatively) turn into
> 
>      1.(-2)  --> 1.(-1)  +-> 1.0 ------> 1.1 ------> 1.2 ...
>                           |
>                              +-> 1.(-1).2.1
> 
> Realistically all the revision numbers in the entire archive would have 
> to be shifted up such that the chain really does start at 1.1:
> 
>      1.1  -----> 1.2 --*---> 1.3 ------> 1.4 ------> 1.5 ...
>                          \
>                              +-> 1.2.2.1
> 
> Unfortunately, the way RCS ,v files are organized, this shift can only 
> be performed by parsing and re-encoding every revision on the vendor 
> branch.  (The direction the diffs are recorded is from the head all the 
> way down to 1.1, and from there _up_ along the vendor branch).  And 
> because of the way CVS uses RCS ,v files, every one of them has to be 
> transformed individually.

Maybe that is what cvs2git is doing.  As I said, it takes 45 minutes. 
But gosh the tree-structure and tags of cvs-fast-export matches cvs2git 
so well.  I think it is a simple matter of cvs-fast-export not 
recognizing it has to do a merge in those half dozen locations due to a 
cross-branch reference to a version number.  If reposurgeon could make 
those connections, it would be nice, but I don't think reposurgeon works 
that way, just rebasing.


> _That_ is the transformation that needs to be done in order for 
> conversion tools not to have any problems with the vendor branch.  And 
> because the transformation differs for every RCS archive, it has to be 
> done either directly on the CVS repository, or the importer has to 
> pretend it had happened that way.
> 
> Let me reiterate: to the best of my understanding, no process working on 
> an already converted git repository has any realistic chance to perform 
> this operation correctly.  It has to be done on the CVS import side.

But I said that.  cvs-fast-export has access to all that original 
information; it's the place this sort of thing should be addressed.

I suggest trying cvs2git and then explore the various changesets for the 
repository, see if it makes sense, and keep in mind this idea that the 
abundance of tags from Lars is the added information that's giving these 
merges an implied structure.

I'm willing to help, I identified all the branch points and agree with 
what cvs2git is producing in terms of psuedo-merges, tested against CVS 
etc..  I'm willing to look into cvs-fast-export (as I see it, git-wise, 
merges and branches are very similar except the branch case is like 
having one of the bases of a merge be empty, so there might not need to 
be too much code-writing).  But I didn't make the call on 
cvs-fast-export, so I think the decision of what to do here is up to 
someone else at this point.

Dan

Re: One branchpoint discrepancy

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> >I rather doubt any good will come from that.  Nobody understands that
> >utility anywhere near well enough to be able to modify it on short notice
> >without risking total break-down.  At this point it is, essentially,
> >magic.
> 
> It doesn't look like a very big program.

No, it doesn't.  What used to be one large lump of incomprehensibility
has shrunk in size and apparent complexity as I carved pieces off the
outside, narrowing their interfaces to the central core.

Those outside bits are indeed pretty small and comprehensible, now.
The most notable success was a clean refactor of the front-end code
for digesting RCS masters that enabled it to use thread-per-master
parallelism with a re-entrant parser.  That's how the program went
from respectably fast to ridiculously fast.

Unfortunately, there remain two knots of mystery in the code that
nobody has been able to crack.  The greater one one is the actual
changeset generation in merge.c - the black hole of incomprehension is
the functions merge_to_changesets() and merge_branches(), which solve
the general changeset-synthesis problem HBB accurately described.  The
lesser mystery is cvs_master_patch_vendor_branch(), which is probably
where our problem is.

Over the last five years about six pretty sharp hackers have tried
to comprehend these well enough to substantially modify them.  The two
who came closest to succeeding were probably Lawrence Hygate and
myself, but all of us ultimately failed. The central merge code and
the vendor-branch handling are not much less of a black box than when
Keith Packard wrote them.

Yes, this seriously sucks.  It means that when cvs-fast-export fails
there isn't much recourse.  But...I triaged several CVS lifters when I
was qualifying front ends for reposurgeon; I ended up adopting this
one (and heavily modifying it) because it sucked the least. Which is
to say it handles the largest range of cases without crashing or
spewing nonsense.

Also, fast actually matters.  cvs2git probably comes the closest to
cvs-fast-export in terms of correctness and robustness across weird
cases.  But it's so slow that the kind of iterative refinement I'm
doing by repeatedly tweaking the reconvert script is not really
practical - I belive you noticed 45 minutes of lag, as opposed to 20
seconds on my desktop.  That's a problem that's getting worse over
time, because the small, clean CVS repos have already been moved.
Increasingly it's only the large, old, nasty ones that are left.

When I wrote about this adverse-selection problem in 2014 I actually
cited GNUPLOT as an example:

     http://esr.ibiblio.org/?p=6216

So...Daniel, the problem is really difficult, and not one I think we
can afford to block the GNUPLOT conversion on when CVS support has a
hard drop-dead date.

That said, you might bring a new perspective to the problem.  I ran
out of possibilities years back; if you can actually improve the
vendor-branch handling I will be *very* impressed.

> Maybe that is what cvs2git is doing.  As I said, it takes 45 minutes. But
> gosh the tree-structure and tags of cvs-fast-export matches cvs2git so well.
> I think it is a simple matter of cvs-fast-export not recognizing it has to
> do a merge in those half dozen locations due to a cross-branch reference to
> a version number.  If reposurgeon could make those connections, it would be
> nice, but I don't think reposurgeon works that way, just rebasing.

That is correct.  By the time reposurgeon sees the stream, the information
required to do vendor-branch surgery is gone.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Eric S. R. <es...@th...>]

Hans-Bernhard Bröker <HBB...@t-...>:
> Am 29.10.2017 um 13:45 schrieb Eric S. Raymond:
> >Daniel J Sebald <dan...@ie...>:
> >
> >Does cvs2git get the tip content of 3.7.x right - that is, coincident with the
> >3.7.3 tarball?
> 
> Please note that the tip of the 3.7.* branch is not coincident with the
> 3.7.3 tag or tarball.  Some further changes were made on the branch after
> 3.7.3.

I do not see any in gitspace. How would I retrieve or list them in CVS? 

> The same goes for all of our release branches.  They all received a couple
> more changes after the last release was made from them.  So none of the
> branch tips coincides to any released tarball.  Comparisons really would
> have to be made to cvs checkouts of the branches, instead.

Which just brings up the more general question: how do I see these in CVS?

> >>>1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
> >>>the archived 3.7.[0123] trees in sequence (I now have them all).
> 
> That would be slightly wrong because 3.7.0 is not on the branch.  It's on
> the trunk.

OK. No one has told me where yet, though.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Hans-Bernhard B. <HBB...@t-...>]

Am 29.10.2017 um 17:05 schrieb Eric S. Raymond:
> Hans-Bernhard Bröker <HBB...@t-...>:
>> Am 29.10.2017 um 13:45 schrieb Eric S. Raymond:
>>> Daniel J Sebald <dan...@ie...>:
>>>
>>> Does cvs2git get the tip content of 3.7.x right - that is, coincident with the
>>> 3.7.3 tarball?
>>
>> Please note that the tip of the 3.7.* branch is not coincident with the
>> 3.7.3 tag or tarball.  Some further changes were made on the branch after
>> 3.7.3.
> 
> I do not see any in gitspace. How would I retrieve or list them in CVS?

The pedestrian way would be to just check out a working copy on the 
branch, and one of the last release tag from the same branch, and 
compare those.  In a generated reposurgeon Makefile setup:

   make gnuplot-{Release_5_0_7,branch-5-0-stable}-checkout
   diff -u gnuplot-{Release_5_0_7,branch-5-0-stable}-checkout/ChangeLog

Or one can look at a "cvs log ChangeLog" or the ChangeLog,v archive file 
itself, and check if there are revisions 1.{N}.2.{M} with N the same as 
that of the last release tag, and M bigger than that of the same tag.

>>>>> 1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
>>>>> the archived 3.7.[0123] trees in sequence (I now have them all).
>>
>> That would be slightly wrong because 3.7.0 is not on the branch.  It's on
>> the trunk.
> 
> OK. No one has told me where yet, though.

The GNUPLOT_RELEASE_3_7_0 tag is genuine.

Re: Roundup of GNUPLOT conversion issues

[Eric S. R. <es...@th...>]

Hans-Bernhard Bröker <HBB...@t-...>:
> >>Please note that the tip of the 3.7.* branch is not coincident with the
> >>3.7.3 tag or tarball.  Some further changes were made on the branch after
> >>3.7.3.
> >
> >I do not see any in gitspace. How would I retrieve or list them in CVS?
> 
> The pedestrian way would be to just check out a working copy on the branch,
> and one of the last release tag from the same branch, and compare those.  In
> a generated reposurgeon Makefile setup:

OK, I have verified that I have those post-3.7.3 commits in the conversion.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Daniel J S. <dan...@ie...>]

On 10/29/2017 10:20 AM, Hans-Bernhard Bröker wrote:
> Am 29.10.2017 um 13:45 schrieb Eric S. Raymond:
>> Daniel J Sebald <dan...@ie...>:
>>
>> Does cvs2git get the tip content of 3.7.x right - that is, coincident 
>> with the
>> 3.7.3 tarball?
> 
> Please note that the tip of the 3.7.* branch is not coincident with the 
> 3.7.3 tag or tarball.  Some further changes were made on the branch 
> after 3.7.3.
> 
> The same goes for all of our release branches.  They all received a 
> couple more changes after the last release was made from them.  So none 
> of the branch tips coincides to any released tarball.  Comparisons 
> really would have to be made to cvs checkouts of the branches, instead.

Correct.  I'm not sure, but it may be that the tarball doesn't exactly 
match CVS either.  It could have been that Lars tweaked some files on 
his system just prior to making the tarball.

In any case, its a very easy matter in git to checkout the converted 
release version as a new branch, wipe all the files in the directory 
(except .git directory of course!), unpack the tarball, press the 
"reload" button of qgit, gitg, or git-gui, write a commit message and 
create a commit... and there you have "GNUPLOT_RELESE_X_Y_Z-tarball" as 
a little stub branch.  The diffs are relatively small so won't take much 
size in the repository.


>>>> 1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
>>>> the archived 3.7.[0123] trees in sequence (I now have them all).
> 
> That would be slightly wrong because 3.7.0 is not on the branch.  It's 
> on the trunk.

This is correct.  GNUPLOT_RELEASE_3_7_0 is on the trunk while the other 
GNUPLOT_RELEASE_3_7_x are on the branch.  I'm attaching screenshots of 
the cvs2git tree and the cvs-fast-export tree after I've made the rebase 
with reposurgeon.  The GNUPLOT_RELEASE_3_7_0 synthetic commit was done 
by cvs-fast-export, not reposurgeon.  So, cvs-fast-export got the 
tag/branch correct but didn't realize that's where the pertinent branch 
should go.

Dan

Re: Roundup of GNUPLOT conversion issues

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> >That would be slightly wrong because 3.7.0 is not on the branch.  It's on
> >the trunk.
> 
> This is correct.  GNUPLOT_RELEASE_3_7_0 is on the trunk while the other
> GNUPLOT_RELEASE_3_7_x are on the branch.  I'm attaching screenshots of the
> cvs2git tree and the cvs-fast-export tree after I've made the rebase with
> reposurgeon.  The GNUPLOT_RELEASE_3_7_0 synthetic commit was done by
> cvs-fast-export, not reposurgeon.  So, cvs-fast-export got the tag/branch
> correct but didn't realize that's where the pertinent branch should go.

Right, because that commit represents a broken (incomplete) tag set.
cvs-fast-export ignores those as branch point candidates because
there's no way to know that the checkout at that point is correct.

Which actually gives me an idea for a feature.  I could add a switch to
whitelist incomplete tags.

Here is a useful thing you can do while I try to fix the changelog scanning.
Modify reconvert so it doesn't delete that one branchlet, reparent the
3.7.x branch onto it, and compare the result at tip.

If there are no differences other than RCS IDs, I would whitelist the tag.
At that point we could consider the 3.7.x branch salvaged. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Daniel J S. <dan...@ie...>]

On 10/29/2017 11:05 AM, Eric S. Raymond wrote:
> Hans-Bernhard Bröker <HBB...@t-...>:
>> Am 29.10.2017 um 13:45 schrieb Eric S. Raymond:
>>> Daniel J Sebald <dan...@ie...>:
>>>
>>> Does cvs2git get the tip content of 3.7.x right - that is, coincident with the
>>> 3.7.3 tarball?
>>
>> Please note that the tip of the 3.7.* branch is not coincident with the
>> 3.7.3 tag or tarball.  Some further changes were made on the branch after
>> 3.7.3.
> 
> I do not see any in gitspace. How would I retrieve or list them in CVS?
> 
>> The same goes for all of our release branches.  They all received a couple
>> more changes after the last release was made from them.  So none of the
>> branch tips coincides to any released tarball.  Comparisons really would
>> have to be made to cvs checkouts of the branches, instead.
> 
> Which just brings up the more general question: how do I see these in CVS?

I don't think the exact tarballs are there.  I sort of have a 
recollection of whenever a release was to be made, the tarball required 
some extra effort on someone's part.


>>>>> 1. Drop the 3.7.x branch.  Replace it with a branch consisting of just
>>>>> the archived 3.7.[0123] trees in sequence (I now have them all).
>>
>> That would be slightly wrong because 3.7.0 is not on the branch.  It's on
>> the trunk.
> 
> OK. No one has told me where yet, though.

cvs-fast-export seems to have gotten it right.  This is why I restored 
tags/branches in ./reconvert, to get greater detail.

Dan

Re: One branchpoint discrepancy

[Daniel J S. <dan...@ie...>]

On 10/31/2017 02:02 PM, Eric S. Raymond wrote:
> Daniel J Sebald <dan...@ie...>:
>>> I rather doubt any good will come from that.  Nobody understands that
>>> utility anywhere near well enough to be able to modify it on short notice
>>> without risking total break-down.  At this point it is, essentially,
>>> magic.
>>
>> It doesn't look like a very big program.
> 
> No, it doesn't.  What used to be one large lump of incomprehensibility
> has shrunk in size and apparent complexity as I carved pieces off the
> outside, narrowing their interfaces to the central core.
> 
> Those outside bits are indeed pretty small and comprehensible, now.
> The most notable success was a clean refactor of the front-end code
> for digesting RCS masters that enabled it to use thread-per-master
> parallelism with a re-entrant parser.  That's how the program went
> from respectably fast to ridiculously fast.
> 
> Unfortunately, there remain two knots of mystery in the code that
> nobody has been able to crack.  The greater one one is the actual
> changeset generation in merge.c - the black hole of incomprehension is
> the functions merge_to_changesets() and merge_branches(), which solve
> the general changeset-synthesis problem HBB accurately described.  The
> lesser mystery is cvs_master_patch_vendor_branch(), which is probably
> where our problem is.

Yes, I started out looking at merge.c and merge_branches().  It walks 
backward from the start of heads of file branches stitching them 
together to create a git branch.  I printed out the number of branches:

find . -name '*,v' -print | cvs-fast-export --reposurgeon > /dev/null
MERGED SOME BRANCHES 1257
MERGED SOME BRANCHES 338
MERGED SOME BRANCHES 407
cvs-fast-export: warning - branch point branch-pre-3-7-1 -> import-1.1.1 
matched by date
MERGED SOME BRANCHES 672
cvs-fast-export: warning - branch point branch-4-2-stable -> 
import-1.1.1 matched by date
MERGED SOME BRANCHES 740
cvs-fast-export: warning - branch point branch-4-4-stable -> 
import-1.1.1 matched by date
MERGED SOME BRANCHES 795
cvs-fast-export: warning - branch point branch-4-6-stable -> 
import-1.1.1 matched by date
MERGED SOME BRANCHES 805
cvs-fast-export: warning - branch point branch-5-0-stable -> 
import-1.1.1 matched by date
MERGED SOME BRANCHES 789
MERGED SOME BRANCHES 553
cvs-fast-export: warning - branch point column -> import-1.1.1 matched 
by date
MERGED SOME BRANCHES 544
cvs-fast-export: warning - branch point branch-4-0-stable -> 
import-1.1.1 matched by date
MERGED SOME BRANCHES 529
cvs-fast-export: warning - branch point pm3d -> import-1.1.1 matched by date
MERGED SOME BRANCHES 507
cvs-fast-export: warning - branch point axis_branch_base -> import-1.1.1 
matched by date
cvs-fast-export: no commitids before 2017-10-31T04:47:09Z.

Those number of branches suggests to me the first step in this process 
is to build a branch/linked-list for all individual files, i.e., the 
file.ext,v files as the source.  Then to build a branch means "merging" 
all those in a meaningful way to create the git branch--something 
involving "clique" determines the root of the branch.  Having reached 
that point, then the git branch is attached to its parent, i.e., branch 
point.

The issue is that there is no consideration in this clique process that 
includes the symbol "version stamps" identifying cross-branch file use. 
(Instead it is solely based upon date, which means cvs-fast-export 
concludes so many of these branches go back to the import/vendor branch 
as the branchpoint.  See warnings above about dates matching.)  The 
information about branches there, all neatly saved in the branch C++ 
structures; it's just not used.  I really like the C++ structure of the 
code, e.g., the way that CVS numbers are stored as little objects for 
which A-to-B comparisons can be done easily.  It's just a question of 
wanting to set about doing that.

However, the sense in which branch_merge() considers "merge" isn't the 
same notion as a git "merge".


> Also, fast actually matters.  cvs2git probably comes the closest to
> cvs-fast-export in terms of correctness and robustness across weird
> cases.  But it's so slow that the kind of iterative refinement I'm
> doing by repeatedly tweaking the reconvert script is not really
> practical - I belive you noticed 45 minutes of lag, as opposed to 20
> seconds on my desktop.  That's a problem that's getting worse over
> time, because the small, clean CVS repos have already been moved.
> Increasingly it's only the large, old, nasty ones that are left.

Yes, cvs2git is slow, sort of brute force creating some giant blob.

Dan

Re: One branchpoint discrepancy

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> Those number of branches suggests to me the first step in this process is to
> build a branch/linked-list for all individual files, i.e., the file.ext,v
> files as the source.  Then to build a branch means "merging" all those in a
> meaningful way to create the git branch--something involving "clique"
> determines the root of the branch.  Having reached that point, then the git
> branch is attached to its parent, i.e., branch point.

That is correct.  For branch-merging purposes, a "clique" is a set of file
mods with identical metadata (committer and comment) and dates within a certain
time fuzz in seconds of each other (default 300 seconds).

The logic walks down branches from tip to base trying to identify cliques
which it makes into changesets.

> The issue is that there is no consideration in this clique process that
> includes the symbol "version stamps" identifying cross-branch file use.

That is also correct.  

>                I really like the C++ structure of the code, e.g., the way
> that CVS numbers are stored as little objects for which A-to-B comparisons
> can be done easily.  It's just a question of wanting to set about doing
> that.

Yeah, thank Keith for that, then me.  He went partway down the road of making
those things objectlets; I saw where he was going and continued it to its
logical conclusion.  I have no doubt he would have done the same, but he
abandoned the code pretty immediately and in a semi-unfinished state
as soon as he got the X repos moved.

As a result, the core algorithm and overall architecture are Keith's, but the
prettiness is often me.  I did it in self-defense - refactoring the code
to make it readable was the only way I could find to understand it.

> However, the sense in which branch_merge() considers "merge" isn't the same
> notion as a git "merge".

Yes.  I should think about renaming so the term "merge" isn't used, to avoid
that confusion.  Perhaps "collate"?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Hans-Bernhard B. <HBB...@t-...>]

Am 29.10.2017 um 20:31 schrieb Daniel J Sebald:

> I don't think the exact tarballs are there.  I sort of have a 
> recollection of whenever a release was to be made, the tarball required 
> some extra effort on someone's part.

Not back then they didn't.  The tarballs can be generated directly from 
a fresh working copy, just by running

	./configure && make dist

But all that effectively does is make a version-named copy of the source 
tree without "CVS" directories and .cvsignore files, and tarball that.

Any actual differences between that and the official release tarball are 
down to either

a) auto-tools of different versions being re-run at 'make' time, or
b) last-minute fixes applied directly to the tarball, manually

Only when we switched to automake, dropped all generated files from the 
repository, and started using the 'prepare' script instead, did tarball 
contents start to be really different from fresh working copies of their 
corresponding CVS tags.

Re: Roundup of GNUPLOT conversion issues

[Daniel J S. <dan...@ie...>]

On 10/29/2017 03:48 PM, Hans-Bernhard Bröker wrote:
> Am 29.10.2017 um 20:31 schrieb Daniel J Sebald:
> 
>> I don't think the exact tarballs are there.  I sort of have a 
>> recollection of whenever a release was to be made, the tarball 
>> required some extra effort on someone's part.
> 
> Not back then they didn't.  The tarballs can be generated directly from 
> a fresh working copy, just by running
> 
>      ./configure && make dist
> 
> But all that effectively does is make a version-named copy of the source 
> tree without "CVS" directories and .cvsignore files, and tarball that.
> 
> Any actual differences between that and the official release tarball are 
> down to either
> 
> a) auto-tools of different versions being re-run at 'make' time, or
> b) last-minute fixes applied directly to the tarball, manually
> 
> Only when we switched to automake, dropped all generated files from the 
> repository, and started using the 'prepare' script instead, did tarball 
> contents start to be really different from fresh working copies of their 
> corresponding CVS tags.

Thanks for the -I '\$[A-Z]*[a-z]*', it's a major help.  I've done a 
comparison of cvs2git and CVS at GNUPLOT_RELEASE_3_7_3 and get agreement 
aside from the cruft of empty CVS directories an so on.  There are these 
few variations on the RCS-style lines:

diff -ur '--exclude=.git' -I '\$[A-Z]*[a-z]*' 
/home/sebald/gnuplot/gnuplot/gnuplot/docs/old/ChangeLog.old 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/ChangeLog.old
--- /home/sebald/gnuplot/gnuplot/gnuplot/docs/old/ChangeLog.old 
2017-10-29 15:27:21.998077850 -0500
+++ 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/ChangeLog.old 
2017-10-28 22:02:43.908845219 -0500
@@ -8,10 +8,7 @@
   * working. This means that the revision numbers in the individual files
   * do not agree with the one in the log. I hope to fix this some time 
soon.
   *
- * $Log: ChangeLog.old,v $
- * Revision 1.1  1998/11/16 13:07:33  lhecking
- * Moved from version.c.
- *
+ * $Log$
   *
   * I have moved these entries from version.c. All future code changes
   * should be logged in ChangeLog. Lars Hecking
Only in /home/sebald/gnuplot/gnuplot/gnuplot/docs/old: CVS
diff -ur '--exclude=.git' -I '\$[A-Z]*[a-z]*' 
/home/sebald/gnuplot/gnuplot/gnuplot/docs/old/makefile.r 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/makefile.r
--- /home/sebald/gnuplot/gnuplot/gnuplot/docs/old/makefile.r	2002-01-26 
12:55:02.000000000 -0600
+++ 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/makefile.r 
2017-10-29 15:49:42.034090965 -0500
@@ -1,10 +1,7 @@

-# $Id: makefile.r,v 1.1.2.1 2002/01/26 18:55:02 lhecking Exp $
-#
-# $Log: makefile.r,v $
-# Revision 1.1.2.1  2002/01/26 18:55:02  lhecking
-# Support for pdf and W3C Scalable Vector Graphics output.
+# $Id$
  #
+# $Log$
  # Revision 1.1  1998/12/09 17:24:30  lhecking
  # Moved from ../..
  #
diff -ur '--exclude=.git' -I '\$[A-Z]*[a-z]*' 
/home/sebald/gnuplot/gnuplot/gnuplot/docs/old/README.3p5 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/README.3p5
--- /home/sebald/gnuplot/gnuplot/gnuplot/docs/old/README.3p5	2017-10-29 
15:27:22.006077850 -0500
+++ 
/home/sebald/gnuplot/git_translation/cvs2git_test/clonedgit/myproject/gnuplot/docs/old/README.3p5 
2017-10-28 22:02:43.912845219 -0500
@@ -1,12 +1,9 @@
  This is a bugfix to version 3.4.


-# $Id: README.3p5,v 1.1 1998/11/16 13:01:23 lhecking Exp $
-#
-# $Log: README.3p5,v $
-# Revision 1.1  1998/11/16 13:01:23  lhecking
-# Moved from top level dir.
+# $Id$
  #
+# $Log$
  # Revision 1.1  1993/09/27 17:07:30  alex
  # gnuplot 3.5 release
  #


The cvs-fast-export result, with reposurgeon rebase of the branch to the 
same point of the cvs2git branch doesn't match so well.  Here are the 
files that differ:

amiga.c
Copyright
ctrl87.c
ctrl87.h
demo/gnuplot.rot
docs/gpcard.tex
docs/old/ChangeLog.old  [This is RCSid diff, does not count]
docs/old/makefile.r     [Ditto]
docs/old/README.3p5     [Ditto]
fnproto.h
intergra.x11
os9.c
win/wgnuplib.c
win/wprinter.c
win/wresourc.h

In qgit applied to cvs2git translation, I can trace the changes in these 
files back to the changeset "Import of beta 343." (amiga.c, Copyright, 
et al.) and "Import of beta 344." (demo/gnuplot.rot).  I believe what we 
are missing is as simple as merging those utility branches back into 
master branch at the start of the repository.

Dan

Re: Roundup of GNUPLOT conversion issues

[Hans-Bernhard B. <HBB...@t-...>]

Am 29.10.2017 um 22:41 schrieb Daniel J Sebald:

> Thanks for the -I '\$[A-Z]*[a-z]*', it's a major help.  I've done a 
> comparison of cvs2git and CVS at GNUPLOT_RELEASE_3_7_3 and get agreement 
> aside from the cruft of empty CVS directories an so on.  There are these 
> few variations on the RCS-style lines:
[...]

Those are caused by the $Log$ keyword, which expands differently from 
all the other RCS/SCCS keywords, so it cannot be filtered out by diff.

One branchpoint discrepancy

[Daniel J S. <dan...@ie...>]

I've skimmed through branchpoints for both cvs-fast-export/reposurgeon 
(i.e., current ./reconvert) and cvs2git.  I found one disagreement, the 
branch-4-6-stable branchpoint.

In cvs2git branch-4-6-stable emanates from

     update Bruce Ravel's contact info
     sfeam<>
     11/22/11 11:12 PM

In fast-cvs-export/reposurgeon we've chosen

     Try to allow for coordinate offset caused by scrolling (Firefox, 
Opera, chrome?)
     Ethan A Merritt<xxxxxx@xxxxxx>
     Peter<ploxxxx@pixxxx>

which is one changeset prior to "update Bruce Ravel's contact info".

The interesting thing is that just like cvs2git, cvs-fast-export chose 
to tag

[gnuplot-4-6-alpha]  update Bruce Ravel's contact info

but failed to make that branch connection.  Anyway, this necessitates 
the attached change to ./reconvert.

Dan

Re: One branchpoint discrepancy

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> I've skimmed through branchpoints for both cvs-fast-export/reposurgeon
> (i.e., current ./reconvert) and cvs2git.  I found one disagreement, the
> branch-4-6-stable branchpoint.
> 
> In cvs2git branch-4-6-stable emanates from
> 
>     update Bruce Ravel's contact info
>     sfeam<>
>     11/22/11 11:12 PM
> 
> In fast-cvs-export/reposurgeon we've chosen
> 
>     Try to allow for coordinate offset caused by scrolling (Firefox, Opera,
> chrome?)
>     Ethan A Merritt<xxxxxx@xxxxxx>
>     Peter<ploxxxx@pixxxx>
> 
> which is one changeset prior to "update Bruce Ravel's contact info".
> 
> The interesting thing is that just like cvs2git, cvs-fast-export chose to
> tag
> 
> [gnuplot-4-6-alpha]  update Bruce Ravel's contact info
> 
> but failed to make that branch connection.  Anyway, this necessitates the
> attached change to ./reconvert.
> 
> Dan

Thanks, applied to my reconvert master.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: One branchpoint discrepancy

[Daniel J S. <dan...@ie...>]

On 10/30/2017 12:09 PM, Eric S. Raymond wrote:
> Daniel J Sebald <dan...@ie...>:
[snip]
>> The interesting thing is that just like cvs2git, cvs-fast-export chose to
>> tag
>>
>> [gnuplot-4-6-alpha]  update Bruce Ravel's contact info
>>
>> but failed to make that branch connection.  Anyway, this necessitates the
>> attached change to ./reconvert.
>>
>> Dan
> 
> Thanks, applied to my reconvert master.

OK, thanks.

I'm pretty sure the one major thing we have to resolve is this group of 
CVS checkins at the very start of the repository.  All those changes 
have to be merged into the master branch.  If we can address that, all 
will work out with no need for adjustments.

I'd sort of like to put effort into the right place.  I've cloned the 
cvs-fast-export utility and I'm willing to help on matters, so if its 
possible I wonder if we could enhance the merging aspect of that utility.

Yes, there is a bit of ambiguity in merges with CVS, it just isn't 100% 
accurate to identify exactly what the programmer had checked out when 
compiling and subsequently did a checkin.  But I can see the logic that 
cvs2git is following and there is some sense to it.

RULE: Basically, a merge should be declared whenever a symbol that is 
assigned to a higher-level branch (let's say trunk) references a version 
number that is associated with a lower-level branch.  Let's use 
gnuplot.rot as an example, as that is the one that Eric has applied a 
shim for.  The change associated with that shim is version 1.1.1.2:

1.1.1.2
log
@Import of beta 344.
@
text
@a0 6
# HBB: revised open-ended animation routine. Used to just turn
etc.

OK, so let's look at the CVS gnuplot.rot,v file for the symbols and 
version 1.1.1.2.  I'm including a screenshot of the qgit display of 
cvs-fast-export's output at that important would-be merge point.  Here 
are the contiguous three symbols in which 1.1.1.2 first appears, from 
the gnuplot.rog,v file:

	BETA_344_989422:1.1.1.2
	BETA_344:1.1.1.2
	BETA_343_980416:1.1.1.1

Refer to the screenshot, and notice that BETA_344 is associated with the 
branch (red), and then the first symbol after that which references 
1.1.1.2 and *appears in the trunk* (look at the screenshot, black) is 
the synthetic commit BETA_344_989422:1.1.1.2.  OK, so by the rule listed 
above, this should be declared a merge.  Now, gnuplot.rot is not the 
only file that falls in this changeset, as there are probably several 
other files in which a branch is first referenced by BETA_344_989422.

There might be some ambiguity about the state of other files at this 
point, but I would say simply recall the BETA_344_989422 state from CVS 
and call that the state for the merge and generate all git diffs 
accordingly, perhaps that's not the exact methodology.  But the general 
idea is those changes from the branch have to get merged in at some 
point if they are used by the trunk, and their first use in the trunk is 
the logical place to create a merge.  (Furthermore, in this case, but 
not of general relevance, is the fact that the comments for changesets 
surrounding BETA_344_989422 suggest that indeed things are being "merged".)

So, to summarize, looking at that screenshot PNG, we should have a merge 
connecting BETA_344 (the first reference to gnuplot.rot's 1.1.1.2 in the 
*branch*) to BETA_344_989422 (the first reference to gnuplot.rot's 
1.1.1.2 in the *trunk*).  That merge might sweep in other files, but 
whenever the rule hypothesis for any file appears, there has to be a merge.

I'll be away for most of the day, but think over the amount of effort it 
would be to address this in cvs-fast-export source code.  I'm willing to 
help if I can.  Otherwise, maybe there is some other creative way in 
reposurgeon to force a merge of the branch modifications.  (For example, 
I imagined rebasing the master on top of that red branch with Import XYZ 
changesets, but again that's effort that really isn't the best way to 
solve this.)

Dan

Re: Roundup of GNUPLOT conversion issues

[Eric S. R. <es...@th...>]

Hans-Bernhard Bröker <HBB...@t-...>:
> Am 29.10.2017 um 20:31 schrieb Daniel J Sebald:
> 
> >I don't think the exact tarballs are there.  I sort of have a recollection
> >of whenever a release was to be made, the tarball required some extra
> >effort on someone's part.
> 
> Not back then they didn't.  The tarballs can be generated directly from a
> fresh working copy, just by running
> 
> 	./configure && make dist
> 
> But all that effectively does is make a version-named copy of the source
> tree without "CVS" directories and .cvsignore files, and tarball that.
> 
> Any actual differences between that and the official release tarball are
> down to either
> 
> a) auto-tools of different versions being re-run at 'make' time, or
> b) last-minute fixes applied directly to the tarball, manually
> 
> Only when we switched to automake, dropped all generated files from the
> repository, and started using the 'prepare' script instead, did tarball
> contents start to be really different from fresh working copies of their
> corresponding CVS tags.

Can you identify when that switchover was?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.

Re: Roundup of GNUPLOT conversion issues

[Hans-Bernhard B. <HBB...@t-...>]

Am 01.11.2017 um 14:34 schrieb Eric S. Raymond:
> Hans-Bernhard Bröker <HBB...@t-...>:

>> Only when we switched to automake, dropped all generated files from the
>> repository, and started using the 'prepare' script instead, did tarball
>> contents start to be really different from fresh working copies of their
>> corresponding CVS tags.
> 
> Can you identify when that switchover was?
> 
The change to automake happened very soon after the directory structure 
change, on 1999-03-28.  'prepare' exists since 2001-06-09T19:15:55.  The 
drop of auto-generated files was on 2003-01-08 (as per state 'dead' in 
the Attic/,v files, and a commit mail in in the gnuplot-cvs archives).

The distance between fresh working copies and tarballs really only 
jumped up at the 2003-01-08 step, though.