Re: One branchpoint discrepancy

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> >I rather doubt any good will come from that.  Nobody understands that
> >utility anywhere near well enough to be able to modify it on short notice
> >without risking total break-down.  At this point it is, essentially,
> >magic.
> 
> It doesn't look like a very big program.

No, it doesn't.  What used to be one large lump of incomprehensibility
has shrunk in size and apparent complexity as I carved pieces off the
outside, narrowing their interfaces to the central core.

Those outside bits are indeed pretty small and comprehensible, now.
The most notable success was a clean refactor of the front-end code
for digesting RCS masters that enabled it to use thread-per-master
parallelism with a re-entrant parser.  That's how the program went
from respectably fast to ridiculously fast.

Unfortunately, there remain two knots of mystery in the code that
nobody has been able to crack.  The greater one one is the actual
changeset generation in merge.c - the black hole of incomprehension is
the functions merge_to_changesets() and merge_branches(), which solve
the general changeset-synthesis problem HBB accurately described.  The
lesser mystery is cvs_master_patch_vendor_branch(), which is probably
where our problem is.

Over the last five years about six pretty sharp hackers have tried
to comprehend these well enough to substantially modify them.  The two
who came closest to succeeding were probably Lawrence Hygate and
myself, but all of us ultimately failed. The central merge code and
the vendor-branch handling are not much less of a black box than when
Keith Packard wrote them.

Yes, this seriously sucks.  It means that when cvs-fast-export fails
there isn't much recourse.  But...I triaged several CVS lifters when I
was qualifying front ends for reposurgeon; I ended up adopting this
one (and heavily modifying it) because it sucked the least. Which is
to say it handles the largest range of cases without crashing or
spewing nonsense.

Also, fast actually matters.  cvs2git probably comes the closest to
cvs-fast-export in terms of correctness and robustness across weird
cases.  But it's so slow that the kind of iterative refinement I'm
doing by repeatedly tweaking the reconvert script is not really
practical - I belive you noticed 45 minutes of lag, as opposed to 20
seconds on my desktop.  That's a problem that's getting worse over
time, because the small, clean CVS repos have already been moved.
Increasingly it's only the large, old, nasty ones that are left.

When I wrote about this adverse-selection problem in 2014 I actually
cited GNUPLOT as an example:

     http://esr.ibiblio.org/?p=6216

So...Daniel, the problem is really difficult, and not one I think we
can afford to block the GNUPLOT conversion on when CVS support has a
hard drop-dead date.

That said, you might bring a new perspective to the problem.  I ran
out of possibilities years back; if you can actually improve the
vendor-branch handling I will be *very* impressed.

> Maybe that is what cvs2git is doing.  As I said, it takes 45 minutes. But
> gosh the tree-structure and tags of cvs-fast-export matches cvs2git so well.
> I think it is a simple matter of cvs-fast-export not recognizing it has to
> do a merge in those half dozen locations due to a cross-branch reference to
> a version number.  If reposurgeon could make those connections, it would be
> nice, but I don't think reposurgeon works that way, just rebasing.

That is correct.  By the time reposurgeon sees the stream, the information
required to do vendor-branch surgery is gone.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

My work is funded by the Internet Civil Engineering Institute: https://icei.org
Please visit their site and donate: the civilization you save might be your own.