Menu
▾
▴
preparing the repository move to gitlab
|
From: Bruno H. <br...@cl...> - 2018-04-15 22:55:01
|
Hi all, Thank you for the discussion and arguments re: "sourceforge miseries". The outcome of that discussion was a plan to move to a hg repository (on savannah.gnu.org) immediately and then to a git repository (on gitlab.com) afterwards. I couldn't resist but tried to do the conversion to git already. Like Sam predicted, it was less complicated than the conversion cvs->hg in 2011. We can now bypass the intermediate savannah step. So, I created a git repository from the hg repository [1], set up a group on gitlab.com [2], and set up a project 'clisp' inside this group [3]. What you can do now: * Please look at the git repository, to find mistakes that I overlooked. Look at everything you care about - from the branch structure to the email addresses. - Download [1]. $ mkdir clisp-git $ cd clisp-git $ tar xf .../clisp-initial-git.tar $ git checkout master * Register at gitlab.com if you intend to commit changes or edit the wiki. Then tell me your user name (in private mail), so that I can add you to the clisp group. Bruno [1] https://www.dropbox.com/s/fzrphkpzx12p9pv/clisp-initial-git.tar?dl=0 [2] https://gitlab.com/gnu-clisp [3] https://gitlab.com/gnu-clisp/clisp =============================================================================== Conversion of a hg repository to git ==================================== 1) Web search Experience report: https://hackernoon.com/migrating-140-000-commits-from-mercurial-to-git-5cf46f134261 Conversion using 'fast-export': https://stackoverflow.com/questions/16037787/convert-mercurial-project-to-git https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git https://www.tikalk.com/posts/2017/06/12/convert-mercurial-project-to-git/ https://www.daharveyjr.com/convert-a-mercurial-repository-to-git-using-hg-fast-export/ Conversion using 'HG-Git': https://stackoverflow.com/questions/10710250/converting-mercurial-folder-to-a-git-repository?noredirect=1&lq=1 http://hg-git.github.io/ https://www.mercurial-scm.org/wiki/HgGit 2) Try 'fast-export' approach: https://github.com/frej/fast-export http://repo.or.cz/w/fast-export.git have identical code. So, $ git clone https://github.com/frej/fast-export or $ git clone git://repo.or.cz/fast-export.git $ mkdir repo-git $ cd repo-git $ git init $ ../fast-export/hg-fast-export.sh -r ../HGREPO/clisp Issued 16738 commands git-fast-import statistics: --------------------------------------------------------------------- Alloc'd objects: 120000 Total objects: 115922 ( 18111 duplicates ) blobs : 53419 ( 17647 duplicates 30623 deltas of 52665 attempts) trees : 45816 ( 464 duplicates 41286 deltas of 41753 attempts) commits: 16687 ( 0 duplicates 0 deltas of 0 attempts) tags : 0 ( 0 duplicates 0 deltas of 0 attempts) Total branches: 86 ( 91 loads ) marks: 1048576 ( 16687 unique ) atoms: 3163 Memory total: 7985 KiB pools: 2360 KiB objects: 5625 KiB --------------------------------------------------------------------- pack_report: getpagesize() = 4096 pack_report: core.packedGitWindowSize = 1073741824 pack_report: core.packedGitLimit = 8589934592 pack_report: pack_used_ctr = 4463 pack_report: pack_mmap_calls = 558 pack_report: pack_open_windows = 5 / 5 pack_report: pack_mapped = 4425919816 / 4425919816 --------------------------------------------------------------------- 3) Create authors.txt (using ChangeLog and mail archives). $ ../fast-export/hg-fast-export.sh -r ../HGREPO/clisp -A ../authors.txt Result: 4 GB. After 'git repack -a -d': 4 GB After 'git gc --prune=now': 4 GB Tags: clisp_2_25_1-2001-04-06 clisp_2_25-2001-03-15 clisp_2_26-2001-05-23 clisp_2_27-2001-07-17 clisp_2_28-2002-03-03 clisp_2_30-2002-09-15 clisp_2_31-2003-09-01 clisp_2_32-2003-12-29 clisp_2_33_1-2004-05-22 clisp_2_33-2004-03-17 clisp_2_33_2-2004-06-02 clisp_2_33_84-2005-07-07 clisp_2_34-2005-07-20 clisp_2_35-2005-08-29 clisp_2_36-2005-12-04 clisp_2_37-2006-01-02 clisp_2_38-2006-01-24 clisp_2_39-2006-07-16 clisp_2_40-2006-09-23 clisp_2_41-2006-10-13 clisp_2_42-2007-10-16 clisp_2_43-2007-11-18 clisp_2_44-2008-02-02 clisp_2_45-2008-05-15 clisp_2_46-2008-07-02 clisp_2_47-2008-10-23 clisp_2_48-2009-07-28 clisp_2_49-2010-07-07 clisp_2_49_50-2017-03-19 clisp_2_49_51-2017-04-17 clisp_2_49_60-2017-06-25 clisp_2_49_90-2018-02-11 clisp_2_49_92-2018-02-18 devel ffcall-1-10 ffcall-1-8 ffcall-1-9 libcharset_1_0 libcharset_1_1 libiconv_1_6_1 libiconv_1_7 post-threads1-merge post-threads-merge pre-threads-merge release-1999-01-08 release-1999-07-22 release-2000-03-06 snap-1998-07-30 snap-1998-08-05 snap-1998-08-12 snap-1998-08-29 Branches: clisp_2_29-2002-07-25 clisp_2_33-patched clisp-2.50 jitc master release-2000-Feb texlive2017 thread2 threads threads1 unlabeled-1.10.2 unlabeled-1.1367.2 unlabeled-1.1367.2.1.2 unlabeled-1.1367.2.1.2.3.2 unlabeled-1.16.2 unlabeled-1.17.2 unlabeled-1.17.2.1.2 unlabeled-1.174.2 unlabeled-1.174.2.1.2 unlabeled-1.177.2 unlabeled-1.18.2 unlabeled-1.20.2 unlabeled-1.22.2 unlabeled-1.241.2 unlabeled-1.25.2 unlabeled-1.259.2 unlabeled-1.259.2.1.2 unlabeled-1.259.2.1.2.1.4 unlabeled-1.33.2 unlabeled-1.54.2 unlabeled-1.54.2.1.2 unlabeled-1.55.2 unlabeled-1.70.2 unlabeled-1.70.2.2.2 unlabeled-1.71.2 4) Find large blobs: https://stackoverflow.com/questions/10622179/how-to-find-identify-large-files-commits-in-git-history https://gist.github.com/magnetikonline/dd5837d597722c9c2d5dfa16d8efe5b9 Found 209 copies of impnotes.html, each ca. 4 MB large -> 800 MB of impnotes.html (uncompressed). Found 9901 copies of src/ChangeLog, each ca. 1 MB large on average -> 10 GB of src/ChangeLog (uncompressed). 5) What if I eliminate the ChangeLog? $ java -jar ~/Downloads/bfg-1.13.0.jar --delete-files ChangeLog /media/develdata/devel/CLISP/clisp-git2 Result: 4.2 GB After 'git repack -a -d': 809 MB After 'git gc --prune=now': 809 MB After 'git reflog expire --expire=now --all && git gc --prune=now --aggressive': 51 MB 6) What if I eliminate a dummy file instead? $ java -jar ~/Downloads/bfg-1.13.0.jar --delete-files xyzzy /media/develdata/devel/CLISP/clisp-git2 Result: 4 GB After 'git repack -a -d': 3.9 GB After 'git gc --prune=now': 3.9 GB After 'git reflog expire --expire=now --all && git gc --prune=now --aggressive': 55 MB 7) What is the effect of just 'git gc --prune=now --aggressive' ? Result: 55 MB 8) Create branches.txt and tags.txt. $ ../fast-export/hg-fast-export.sh -r ../HGREPO/clisp -A ../authors.txt -B ../branches.txt -T ../tags.txt 9) Note: The tags.txt file is ignored. The output ends with: Exporting tag [devel] at [hg r0] [git :1] Exporting tag [snap-1998-07-30] at [hg r24] [git :25] Exporting tag [snap-1998-08-05] at [hg r39] [git :40] Exporting tag [snap-1998-08-12] at [hg r54] [git :55] Exporting tag [snap-1998-08-29] at [hg r71] [git :72] Exporting tag [release-1999-01-08] at [hg r365] [git :366] Exporting tag [release-1999-07-22] at [hg r1194] [git :1195] Exporting tag [release-2000-03-06] at [hg r1238] [git :1239] Exporting tag [libcharset_1_0] at [hg r1958] [git :1959] Exporting tag [libiconv_1_6_1] at [hg r1959] [git :1960] Exporting tag [clisp_2_25-2001-03-15] at [hg r1983] [git :1984] Exporting tag [libcharset_1_1] at [hg r2375] [git :2376] Exporting tag [libiconv_1_7] at [hg r2399] [git :2400] Exporting tag [clisp_2_25_1-2001-04-06] at [hg r3749] [git :3750] Exporting tag [clisp_2_26-2001-05-23] at [hg r3750] [git :3751] Exporting tag [clisp_2_27-2001-07-17] at [hg r3751] [git :3752] Exporting tag [clisp_2_28-2002-03-03] at [hg r3752] [git :3753] Exporting tag [ffcall-1-8] at [hg r3753] [git :3754] Exporting tag [clisp_2_30-2002-09-15] at [hg r3954] [git :3955] Exporting tag [clisp_2_31-2003-09-01] at [hg r5494] [git :5495] Exporting tag [clisp_2_32-2003-12-29] at [hg r5534] [git :5535] Exporting tag [clisp_2_33-2004-03-17] at [hg r6010] [git :6011] Exporting tag [ffcall-1-9] at [hg r6011] [git :6012] Exporting tag [clisp_2_33_1-2004-05-22] at [hg r6564] [git :6565] Exporting tag [clisp_2_33_2-2004-06-02] at [hg r6633] [git :6634] Exporting tag [ffcall-1-10] at [hg r6665] [git :6666] Exporting tag [clisp_2_33_84-2005-07-07] at [hg r9539] [git :9540] Exporting tag [clisp_2_34-2005-07-20] at [hg r9572] [git :9573] Exporting tag [clisp_2_35-2005-08-29] at [hg r9736] [git :9737] Exporting tag [clisp_2_36-2005-12-04] at [hg r10332] [git :10333] Exporting tag [clisp_2_37-2006-01-02] at [hg r10470] [git :10471] Exporting tag [clisp_2_38-2006-01-24] at [hg r10547] [git :10548] Exporting tag [clisp_2_39-2006-07-16] at [hg r10867] [git :10868] Exporting tag [clisp_2_40-2006-09-23] at [hg r10987] [git :10988] Exporting tag [clisp_2_41-2006-10-13] at [hg r11030] [git :11031] Exporting tag [clisp_2_42-2007-10-16] at [hg r11478] [git :11479] Exporting tag [clisp_2_43-2007-11-18] at [hg r11744] [git :11745] Exporting tag [clisp_2_44-2008-02-02] at [hg r12080] [git :12081] Exporting tag [pre-threads-merge] at [hg r12104] [git :12105] Exporting tag [clisp_2_45-2008-05-15] at [hg r12520] [git :12521] Exporting tag [clisp_2_46-2008-07-02] at [hg r12843] [git :12844] Exporting tag [clisp_2_47-2008-10-23] at [hg r13363] [git :13364] Exporting tag [post-threads-merge] at [hg r13402] [git :13403] Exporting tag [post-threads1-merge] at [hg r13479] [git :13480] Exporting tag [clisp_2_48-2009-07-28] at [hg r14179] [git :14180] Exporting tag [clisp_2_49-2010-07-07] at [hg r14974] [git :14975] Exporting tag [clisp_2_49_50-2017-03-19] at [hg r15857] [git :15858] Exporting tag [clisp_2_49_51-2017-04-17] at [hg r15891] [git :15892] Exporting tag [clisp_2_49_60-2017-06-25] at [hg r15941] [git :15942] Exporting tag [clisp_2_49_90-2018-02-11] at [hg r16633] [git :16634] Exporting tag [clisp_2_49_92-2018-02-18] at [hg r16656] [git :16657] Therefore rename the tags by hand: git_tag_rename() { git tag "$2" "$1" && git tag -d "$1" } git_tag_rename "release-1999-01-08" "clisp-2.22-1999-01-08" git_tag_rename "release-1999-07-22" "clisp-2.23-1999-07-22" git_tag_rename "release-2000-03-06" "clisp-2.24-2000-03-06" git_tag_rename "clisp_2_25-2001-03-15" "clisp-2.25-2001-03-15" git_tag_rename "clisp_2_25_1-2001-04-06" "clisp-2.25.1-2001-04-06" git_tag_rename "clisp_2_26-2001-05-23" "clisp-2.26-2001-05-23" git_tag_rename "clisp_2_27-2001-07-17" "clisp-2.27-2001-07-17" git_tag_rename "clisp_2_28-2002-03-03" "clisp-2.28-2002-03-03" git_tag_rename "clisp_2_30-2002-09-15" "clisp-2.30-2002-09-15" git_tag_rename "clisp_2_31-2003-09-01" "clisp-2.31-2003-09-01" git_tag_rename "clisp_2_32-2003-12-29" "clisp-2.32-2003-12-29" git_tag_rename "clisp_2_33-2004-03-17" "clisp-2.33-2004-03-17" git_tag_rename "clisp_2_33_1-2004-05-22" "clisp-2.33.1-2004-05-22" git_tag_rename "clisp_2_33_2-2004-06-02" "clisp-2.33.2-2004-06-02" git_tag_rename "clisp_2_33_84-2005-07-07" "clisp-2.33.84-2005-07-07" git_tag_rename "clisp_2_34-2005-07-20" "clisp-2.34-2005-07-20" git_tag_rename "clisp_2_35-2005-08-29" "clisp-2.35-2005-08-29" git_tag_rename "clisp_2_36-2005-12-04" "clisp-2.36-2005-12-04" git_tag_rename "clisp_2_37-2006-01-02" "clisp-2.37-2006-01-02" git_tag_rename "clisp_2_38-2006-01-24" "clisp-2.38-2006-01-24" git_tag_rename "clisp_2_39-2006-07-16" "clisp-2.39-2006-07-16" git_tag_rename "clisp_2_40-2006-09-23" "clisp-2.40-2006-09-23" git_tag_rename "clisp_2_41-2006-10-13" "clisp-2.41-2006-10-13" git_tag_rename "clisp_2_42-2007-10-16" "clisp-2.42-2007-10-16" git_tag_rename "clisp_2_43-2007-11-18" "clisp-2.43-2007-11-18" git_tag_rename "clisp_2_44-2008-02-02" "clisp-2.44-2008-02-02" git_tag_rename "clisp_2_45-2008-05-15" "clisp-2.45-2008-05-15" git_tag_rename "clisp_2_46-2008-07-02" "clisp-2.46-2008-07-02" git_tag_rename "clisp_2_47-2008-10-23" "clisp-2.47-2008-10-23" git_tag_rename "clisp_2_48-2009-07-28" "clisp-2.48-2009-07-28" git_tag_rename "clisp_2_49-2010-07-07" "clisp-2.49-2010-07-07" git_tag_rename "clisp_2_49_50-2017-03-19" "clisp-2.49.50-2017-03-19" git_tag_rename "clisp_2_49_51-2017-04-17" "clisp-2.49.51-2017-04-17" git_tag_rename "clisp_2_49_60-2017-06-25" "clisp-2.49.60-2017-06-25" git_tag_rename "clisp_2_49_90-2018-02-11" "clisp-2.49.90-2018-02-11" git_tag_rename "clisp_2_49_92-2018-02-18" "clisp-2.49.92-2018-02-18" git_tag_rename "devel" "checkin-1998-07-22" git_tag_rename "ffcall-1-10" "ffcall-1.10" git_tag_rename "ffcall-1-8" "ffcall-1.8" git_tag_rename "ffcall-1-9" "ffcall-1.9" git_tag_rename "libcharset_1_0" "libcharset-1.0" git_tag_rename "libcharset_1_1" "libcharset-1.1" git_tag_rename "libiconv_1_6_1" "libiconv-1.6.1" git_tag_rename "libiconv_1_7" "libiconv-1.7" 10) Delete obsolete tags: $ git tag -d snap-1998-07-30 snap-1998-08-05 snap-1998-08-12 snap-1998-08-29 11) Place specific tags: git checkout master git checkout bfff60158a0fee74fe91e03b8f6ca8714889bdb4 ('2001-01-29 16:08:48') git tag -f ffcall-1.8 git checkout b60ce6bed39109a2b37355288d47e31bbf2cd0d4 ('2001-04-06 19:12:37') git tag clisp-2.25.1-2001-04-06-approx git tag -d clisp-2.25.1-2001-04-06 git checkout 1d02292fa7604b5389c21a7eff30273ef3b9db03 ('2001-05-21 20:09:01') git tag -f clisp-2.26-2001-05-23 git checkout 52a98271c6eeee89bafd69ea7d1cbef9b472ae5b ('2001-07-17 00:45:51') git tag -f clisp-2.27-2001-07-17 git checkout 98544bbd05f3b41516cf9a8074a887e639de8ef3 ('2002-03-06 06:05:19') git tag clisp-2.28-2002-03-03-approx git tag -d clisp-2.28-2002-03-03 12) Unify unlabeled-* branches in a clisp-2.29 branch. br=unlabeled-1.1367.2 br=unlabeled-1.1367.2.1.2 br=unlabeled-1.1367.2.1.2.3.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~8 # remove 3 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-ChangeLog git branch -D unlabeled-1.1367.2.1.2 git branch -D unlabeled-1.1367.2 br=unlabeled-1.10.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-spvw_genera1.d br=unlabeled-1.16.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-describe.lisp br=unlabeled-1.17.2 br=unlabeled-1.17.2.1.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~4 # remove 2 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-clisp.spec git branch -D unlabeled-1.17.2 br=unlabeled-1.174.2 br=unlabeled-1.174.2.1.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~4 # remove 2 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-CHANGES.LOG git branch -D unlabeled-1.174.2 br=unlabeled-1.177.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-spvw.d br=unlabeled-1.18.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-ffcall-avcall+vacall-configure br=unlabeled-1.20.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-ffcall-callback-vacall_r-configure br=unlabeled-1.22.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-version.h br=unlabeled-1.241.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-lispbibl.d br=unlabeled-1.25.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-ffcall-aclocal.m4 br=unlabeled-1.259.2 br=unlabeled-1.259.2.1.2 br=unlabeled-1.259.2.1.2.1.4 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~6 # remove 3 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-stream.d git branch -D unlabeled-1.259.2.1.2 git branch -D unlabeled-1.259.2 br=unlabeled-1.33.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-spvw_memfile.d br=unlabeled-1.54.2 br=unlabeled-1.54.2.1.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~4 # remove 2 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-init.lisp git branch -D unlabeled-1.54.2 br=unlabeled-1.55.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-encoding.d br=unlabeled-1.70.2 br=unlabeled-1.70.2.2.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~5 # remove 2 cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-compiler.lisp git branch -D unlabeled-1.70.2 br=unlabeled-1.71.2 git checkout ${br} git reset --hard HEAD^ git rebase -i HEAD~2 # remove cvs2hg generated "fixup commit" git branch -m ${br} clisp-2.29-aclocal.m4 git checkout clisp-2.29-2002-07-25 git reset --hard HEAD^ git rebase -i HEAD~11 # remove 4 cvs2hg generated "fixup commit" and 'regenerated' # merge conflict in src/compiler.lisp git merge 4aa15022fcb7bf0856a9e22b32a9591d1ee0d115 # = clisp-2.29-stream.d minus the last 2 commits git merge clisp-2.29-compiler.lisp # merge conflict in src/compiler.lisp by choosing the 'progn' variant. git merge clisp-2.29-CHANGES.LOG \ clisp-2.29-ChangeLog \ clisp-2.29-aclocal.m4 \ clisp-2.29-ffcall-avcall+vacall-configure \ clisp-2.29-ffcall-callback-vacall_r-configure \ clisp-2.29-describe.lisp \ clisp-2.29-encoding.d \ clisp-2.29-ffcall-aclocal.m4 \ clisp-2.29-init.lisp \ clisp-2.29-lispbibl.d \ clisp-2.29-spvw.d \ clisp-2.29-spvw_genera1.d \ clisp-2.29-spvw_memfile.d \ clisp-2.29-version.h # The result is not exactly clisp-2.29, because there are differences: modules/clx/new-clx/clx.f: ADD 'NOTREACHED and thus NOTIMPLEMENTED and UNDEFINED require a semicolon' 2002-03-06 src/_clisp.html: MINUS last 1 commit src/defs1.lisp: ADD some other (lost) patch src/encoding.lisp: ADD some other (lost) patch src/inspect.lisp: ADD some other (lost) patch src/makemake.in: ADD some other (lost) patch src/type.lisp: ADD some other (lost) patch src/compiler.lisp: ADD some other (lost) patch git tag clisp-2.29-2002-07-24-approx git merge clisp-2.29-clisp.spec clisp-2.29-stream.d git branch -m clisp-2.29-2002-07-25 clisp-2.29 git branch -d clisp-2.29-CHANGES.LOG git branch -d clisp-2.29-ChangeLog git branch -d clisp-2.29-aclocal.m4 git branch -d clisp-2.29-ffcall-avcall+vacall-configure git branch -d clisp-2.29-ffcall-callback-vacall_r-configure git branch -d clisp-2.29-clisp.spec git branch -d clisp-2.29-compiler.lisp git branch -d clisp-2.29-describe.lisp git branch -d clisp-2.29-encoding.d git branch -d clisp-2.29-ffcall-aclocal.m4 git branch -d clisp-2.29-init.lisp git branch -d clisp-2.29-lispbibl.d git branch -d clisp-2.29-spvw.d git branch -d clisp-2.29-spvw_genera1.d git branch -d clisp-2.29-spvw_memfile.d git branch -d clisp-2.29-stream.d git branch -d clisp-2.29-version.h 13) Now for real: Prepare for 10 GB of swap. git gc --prune=now --aggressive |
