Git Conversion Lazarus

From Lazarus wiki
Revision as of 15:59, 1 July 2021 by Martin (talk | contribs) (Translating references to svn rev (r12345) to #hash)

This page has the detailed steps used to convert the Lazarus SVN repro to git.


SubGIT

The initial conversion will be done using TMate Subgit.

Install

Unpack subgit to

   ~/subgit/subgit-3.3.11

Required

An author.txt file

 svn-name - git-name <email>

Run / main repro

   cd ~/subgit/subgit-3.3.11
   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz.git
   
   gedit laz.git/subgit/config 

And add the following (the first line, replaces an existing setting)

   [svn]
     trunk = trunk:refs/heads/main
     gitCommitMessage = %message\\n\\n SVN %revision@%branch\\n
     
   [translate]  
     eols = false
     otherProperties = false
     ignores = false

The translate options prevent .gitignore and .gitattributes. Those can be translated, but will later be replaced...

   cp authors.txt laz.git/subgit/authors.txt
   ./bin/subgit install laz.git

Without skipping the gitignore/gitattributes, there will be an error at revision 47330. The process will hang. nterrupt it and editI

   lazarus.git/svn/.metadata

and set branches-maxRev and tags-maxRev to 47332 (it should be at 47330) https://stackoverflow.com/questions/18918215/skip-revisions-when-installing-subgit-repository Continue with

   ./bin/subgit install laz-bin.git

Stop the daemon

   ./bin/subgit shutdown laz-bin.git/

Run / binaries repro

   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz-bin.git
   gedit laz.git/subgit/config 
     [svn]
     gitCommitMessage = %message\\n\\n SVN %revision@%branch\\n
     trunk = binaries:refs/heads/main
     cp authors.txt laz-bin.git/subgit/authors.txt

follow the steps above

Part 2

  • Updating gitignore
  • Prefixing all issue references with #
  • translating svn revisions to git hashes

Set a git user

   git config --local user.email "lazarus@lazarus-ide.org"
   git config --local user.name "lazarus"

Clean out left overs from subgit

   git for-each-ref --format="%(refname)" refs/svn/ | xargs -n 1 git update-ref -d

Clean left overs from binaries

There are a few tags, that lead to binaries (gdb.exe) ending up in the data

  git tag -d binaries_0_9_26
  git tag -d binaries_0_9_26_2
  git tag -d binaries_0_9_28
  git tag -d binaries_0_9_30
  git tag -d binaries_0_9_30_2
  git tag -d binaries_0_9_30_2RC1
  git tag -d binaries_1_0

Updating gitignore/gitattributes / Prefix issues with #

The below combines the 2 steps. It can be run in 2 separate calls.

It will run for about one hour.

   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --index-filter '  cp ~/subgit/subgit-3.3.11/.gitignore  ~/subgit/subgit-3.3.11/.gitattributes . ; git add .gitignore .gitattributes '  --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f  -- --all


Cleanup

   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all ;git gc --prune=now


Translating references to svn rev (r12345) to #hash

This must be the very last filter-branch. As each filter-branch creates new hashes for the commits, and that would invalidate the hash refs created here.

  • The first batch will update only commits it branch main.
This will also only be references to other commits in main.
Merges from branches to main can not be updated.
  • The 2nd batch will update only commits in branches. This means the hashes for the main branch will be left intact.
This will update merges from main to fixes


   rm -rf /tmp/rmap
   mkdir /tmp/rmap
   git rev-list  main | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "SVN [0-9][0-9]*@"  | sed "s/.*SVN \([0-9]*\)@.*/\1/" ) ; echo $0 > /tmp/rmap/$a'
   git rev-list  main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict; 
   unless ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print;  next; }
   my %f;  foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
     my \$rvl=\$1;    
     foreach my \$a (split / *[\/,] */, \$rvl) {
     unless (\$f{\$a}){
   	\$f{\$a}=q/#/;
       my(\$a1, \$a2) = split / *- */, \$a;
   	my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>;   chomp \$c; }
   	my \$h; if(\$c && (open \$h, q/</, qq#../map/\$c#)) { 	  
   	   \$c=<\$h>;   chomp \$c;  \$c = q/#/.substr \$c, 0,10; 
         if (\$a2) {
   	    my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>;   chomp \$c2; }
   	    my \$h; if(\$c2 && (open \$h, q/</, qq#../map/\$c2#)) { 	  \$c2=<\$h>;   chomp \$c2;   	   
           \$c = \$c.q/-#/.substr( \$c2, 0,10); 
   	    }
   	    else { \$c = \$c.q/..# / unless \$c eq q/#/;  warn q/MAP2 FAIL /.\$c2 }
   	  }
   	  \$f{\$a}=\$c;
   	}
   	else {warn q/MAP1 fail/.\$a;}
     }} 
   }  
   foreach my \$k (keys %f) {   my \$v = \$f{\$k};
     unless (\$v eq q/#/) {     s/([ r,\/]\${k}\b)/\$1 \$v/g;   }
   }   
   warn \$_ if keys %f;
   print; "
   ' -f -- --all
   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d


   rm -rf /tmp/rmap
   mkdir /tmp/rmap
   git rev-list  --all | xargs -L1 /bin/sh -c 'a=$( git log -n1 --pretty='%B' $0 | grep "SVN [0-9][0-9]*@"  | sed "s/.*SVN \([0-9]*\)@.*/\1/" ) ; echo $0 > /tmp/rmap/$a'
   git rev-list  main | xargs -L1 /bin/sh -c 'echo 1 > /tmp/rmap/m_$0'
   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --msg-filter 'perl -wne "use strict; 
   if ( -e qq#/tmp/rmap/m_\$ENV{GIT_COMMIT}# ) { print;  next; }
   my %f;  foreach(/(?:(?:commit|rev|revisions?|revision\(s\)|revert|reverted) +| r)([1-9][0-9]{3,4}\b(?:[\/,-] *[1-9][0-9]{3,4}\b)*)\b/gi) {
     my \$rvl=\$1;    
     foreach my \$a (split / *[\/,] */, \$rvl) {
     unless (\$f{\$a}){
   	\$f{\$a}=q/#/;
       my(\$a1, \$a2) = split / *- */, \$a;
   	my \$c; my \$rh; if( open \$rh, q/</, qq#/tmp/rmap/\$a1#) { \$c=<\$rh>;   chomp \$c; }
       if (\$c) {	unless ( -e qq#/tmp/rmap/m_\$c# ) { 
          my \$h; if(open \$h, q/</, qq#../map/\$c#) {   \$c=<\$h>;   chomp \$c; }
   	}}
   	if(\$c ) {
   	   \$c = q/#/.substr \$c, 0,10; 
         if (\$a2) {
   	    my \$c2; my \$rh2; if( open \$rh2, q/</, qq#/tmp/rmap/\$a1#) { \$c2=<\$rh2>;   chomp \$c2; }
           if (\$c2) {	unless ( -e qq#/tmp/rmap/m_\$c2# ) { 
              my \$h; if(open \$h, q/</, qq#../map/\$c2#) {   \$c2=<\$h>;   chomp \$c2; }
   	    }}
   	    if(\$c2) {   \$c = \$c.q/-#/.substr( \$c2, 0,10);     }
   	    else { \$c = \$c.q/..# / unless \$c eq q/#/;  warn q/MAP2 FAIL /.\$c2 }
   	  }
   	  \$f{\$a}=\$c;
   	}
   	else {warn q/MAP1 fail/.\$a;}
     }} 
   }  
   foreach my \$k (keys %f) {   my \$v = \$f{\$k};
     unless (\$v eq q/#/) {     s/([ r,\/]\${k}\b)/\$1 \$v/g;   }
   }   
   warn \$_ if keys %f;
   print; "
   ' -f -- --all
   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all ;git gc --prune=now
   rm -rf /tmp/rmap

Updating tags

This must be done after all filter-branches

  • Annotate all tags.
  • Create some more tags

This needs the git user being set

   git for-each-ref --format="%(refname:short)" refs/tags/ | xargs -L1 /bin/sh -c 'git tag -f -a $0 -m $0 $0 '


   git rev-list --in-commit-order  --reverse   fixes_2_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_3" -m "main to version 2.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_2_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_1" -m "main to version 2.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_8..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_9" -m "main to version 1.9" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_6..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_7" -m "main to version 1.7" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_4..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_5" -m "main to version 1.5" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_3" -m "main to version 1.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_1" -m "main to version 1.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_30..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_31" -m "main to version 0.9.31" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_28..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_29" -m "main to version 0.9.29" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_26..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_27" -m "main to version 0.9.27" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_24..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_25" -m "main to version 0.9.25" $0'
   git rev-list --in-commit-order  --reverse   main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_0" -m "main init" $0'
   
   git rev-list --in-commit-order  --reverse   main..fixes_2_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-2_2" -m "fixes 2.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_2_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-2_0" -m "fixes 2.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_8 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_8" -m "fixes 1.8 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_6 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_6" -m "fixes 1.6 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_4 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_4" -m "fixes 1.4 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_2" -m "fixes 1.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_0" -m "fixes 1.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_30 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_30" -m "fixes 0.9.30 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_28 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_28" -m "fixes 0.9.28 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_26 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_26" -m "fixes 0.9.26 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_24 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_24" -m "fixes 0.9.24 branch" $0'
   
   git rev-list --in-commit-order  --reverse   main..free-sparta | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "free-sparta" -m "free-sparta branch" $0'
   git rev-list --in-commit-order  --reverse   main..gtk-splitup | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "gtk-splitup" -m "gtk-splitup branch" $0'
   git rev-list --in-commit-order  --reverse   main..lcl-smartlink | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "lcl-smartlink" -m "lcl-smartlink branch" $0'
   git rev-list --in-commit-order  --reverse   main..listviewsortindicator | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "listviewsortindicator" -m "listviewsortindicator branch" $0'
   git rev-list --in-commit-order  --reverse   main..macosfullkeyboardaccess | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "macosfullkeyboardaccess" -m "macosfullkeyboardaccess branch" $0'

Comparing data before/after filter-branch

With subject line / Will show changes in subject

   git log --pretty='format:%ae %an %cd  %s %d' --graph --all > ../log.txt

Without subject line / Should have no diff

  • This compares the graph and committers => Thus it will show if all commits are present, and if all branches are correct
   git log --pretty='format:%ae %an %cd %d' --graph --all > ../log.txt
  • compare the changes to files (gitignore / gitattributes will diff for the top commit(s)) / --graph can be added too
(slow)
   git log --all --pretty='%ae %an %cd %d' --stat > ../statlog.txt
  • Compare the overall amount of lines in the log.
   git log --all | wc -l


Make sure you do that after "clean up" in each step. Otherwise the internal subgit or filter-branch revs will go into the compare.

Also "removing the binaries" introduces a diff.

Data

Please mail me for any changes. Do not make changes here. Otherwise the changes will not make it into the conversion.

.gitignore

   **/*.o
   **/*.ppu
   **/*.exe
   **/*.bak
   **/*.diff
   **/*.patch
   **/units/
   **/lib/
   **/*;[1-9]
   **/*.~*
   **/*.*~
   **/*.fpcunit.ini
   **/*.lps
   **/*.compiled
   /docs/chm/**/*.chm
   /docs/chm/**/*.xct
   /docs/chm/**/*.txt
   /libeay32.dll
   /ssleay32.dll
   /lazarus.cfg
   ide/revision.inc
   
   !**/patches/*.patch
   !**/patches/*.diff

.gitattributes

   * text=auto !eol
   **/*.pp text
   **/*.pas text
   **/*.inc text
   **/*.lfm text
   **/*.lrs text
   **/*.lps text
   **/*.lpr text
   **/*.lpi text
   **/*.lpk text
   **/*.lpl text
   **/*.lrj text
   **/*.dpr text
   **/*.dfm text
   **/*.pot text
   **/*.po text
   **/*.rc text
   **/*.txt text
   **/*.txt.sample text
   **/*.ini text
   **/*.cfg text
   **/*.iss text
   **/*.isl text
   **/*.xml text
   **/*.xsl text
   **/*.html text
   **/*.css text
   **/*.patch text
   **/*.md text
   **/*.bat text
   **/*.sh text
   **/*.compiled text
   **/*.bdsproj text
   **/*.plist text
   **/*.h text
   **/*.c text
   **/*.cpp text
   **/*.in text
   **/Makefile text
   **/Makefile.compiled text
   **/Makefile.fpc text
   **/*.exe -text
   **/*.dll -text
   **/*.so -text
   **/*.gif -text
   **/*.jpg -text
   **/*.ico -text
   **/*.svg -text
   **/*.bmp -text
   **/*.res -text
   **/*.ttf -text
   **/*.xpm -text
   **/*.cur -text
   **/*.db3 -text
   **/*.dbf -text
   **/*.mbf -text
   **/*.mdx -text
   **/*.fbk -text
   **/*.odt -text
   **/*.ods -text
   **/*.odg -text
   **/*.odp -text
   **/*.xls -text
   **/*.pdf -text
   **/*.zip -text
   **/*.rar -text
   **/*.tar -text
   **/*.tgz -text
   **/*.gz -text
   **/*.icns -text
   **/*.dcr -text
   components/aggpas/**/*.ppm -text
   **/*.app/Contents/MacOS/* -text