Git Conversion Lazarus

From Lazarus wiki
Revision as of 15:21, 1 July 2021 by Martin (talk | contribs) (Comparing data before/after filter-branch)

This page has the detailed steps used to convert the Lazarus SVN repro to git.


SubGIT

The initial conversion will be done using TMate Subgit.

Install

Unpack subgit to

   ~/subgit/subgit-3.3.11

Required

An author.txt file

 svn-name - git-name <email>

Run / main repro

   cd ~/subgit/subgit-3.3.11
   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz.git
   
   gedit laz.git/subgit/config 

And add the following (the first line, replaces an existing setting)

   [svn]
     trunk = trunk:refs/heads/main
     gitCommitMessage = %message\\n\\n SVN %revision@%branch\\n
     
   [translate]  
     eols = false
     otherProperties = false
     ignores = false

The translate options prevent .gitignore and .gitattributes. Those can be translated, but will later be replaced...

   cp authors.txt laz.git/subgit/authors.txt
   ./bin/subgit install laz.git

Without skipping the gitignore/gitattributes, there will be an error at revision 47330. The process will hang. nterrupt it and editI

   lazarus.git/svn/.metadata

and set branches-maxRev and tags-maxRev to 47332 (it should be at 47330) https://stackoverflow.com/questions/18918215/skip-revisions-when-installing-subgit-repository Continue with

   ./bin/subgit install laz-bin.git

Stop the daemon

   ./bin/subgit shutdown laz-bin.git/

Run / binaries repro

   ./bin/subgit configure https://svn.freepascal.org/svn/lazarus laz-bin.git
   gedit laz.git/subgit/config 
     [svn]
     gitCommitMessage = %message\\n\\n SVN %revision@%branch\\n
     trunk = binaries:refs/heads/main
     cp authors.txt laz-bin.git/subgit/authors.txt

follow the steps above

Part 2

  • Updating gitignore
  • Prefixing all issue references with #
  • translating svn revisions to git hashes

Set a git user

   git config --local user.email "lazarus@lazarus-ide.org"
   git config --local user.name "lazarus"

Clean out left overs from subgit

   git for-each-ref --format="%(refname)" refs/svn/ | xargs -n 1 git update-ref -d

Clean left overs from binaries

There are a few tags, that lead to binaries (gdb.exe) ending up in the data

  git tag -d binaries_0_9_26
  git tag -d binaries_0_9_26_2
  git tag -d binaries_0_9_28
  git tag -d binaries_0_9_30
  git tag -d binaries_0_9_30_2
  git tag -d binaries_0_9_30_2RC1
  git tag -d binaries_1_0

Updating gitignore/gitattributes / Prefix issues with #

The below combines the 2 steps. It can be run in 2 separate calls.

It will run for about one hour.

   FILTER_BRANCH_SQUELCH_WARNING=1 git filter-branch --index-filter '  cp ~/subgit/subgit-3.3.11/.gitignore  ~/subgit/subgit-3.3.11/.gitattributes . ; git add .gitignore .gitattributes '  --msg-filter ' perl -ne "while (s/((?:iss?ues?|[ (](?:bugs?|mantis|fix|fixes|patch)\b)(?: +id)?[ :]+(?:#?0*[1-9][0-9]{2,4}(?: *(?:[ ,]| and ) *))*)(0*[1-9][0-9]{2,4})\b/\1#\2/i) {}; print" ' -f  -- --all


Cleanup

   git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
   git reflog expire --expire=now --all ;git gc --prune=now


Translating references to svn rev (r12345) to #hash

This must be the very last filter-branch. As each filter-branch creates new hashes for the commits, and that would invalidate the hash refs created here.


Updating tags

This must be done after all filter-branches

  • Annotate all tags.
  • Create some more tags

This needs the git user being set

   git for-each-ref --format="%(refname:short)" refs/tags/ | xargs -L1 /bin/sh -c 'git tag -f -a $0 -m $0 $0 '


   git rev-list --in-commit-order  --reverse   fixes_2_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_3" -m "main to version 2.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_2_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-2_1" -m "main to version 2.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_8..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_9" -m "main to version 1.9" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_6..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_7" -m "main to version 1.7" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_4..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_5" -m "main to version 1.5" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_2..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_3" -m "main to version 1.3" $0'
   git rev-list --in-commit-order  --reverse   fixes_1_0..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-1_1" -m "main to version 1.1" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_30..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_31" -m "main to version 0.9.31" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_28..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_29" -m "main to version 0.9.29" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_26..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_27" -m "main to version 0.9.27" $0'
   git rev-list --in-commit-order  --reverse   fixes_0_9_24..main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_9_25" -m "main to version 0.9.25" $0'
   git rev-list --in-commit-order  --reverse   main | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "main-0_0" -m "main init" $0'
   
   git rev-list --in-commit-order  --reverse   main..fixes_2_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-2_2" -m "fixes 2.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_2_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-2_0" -m "fixes 2.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_8 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_8" -m "fixes 1.8 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_6 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_6" -m "fixes 1.6 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_4 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_4" -m "fixes 1.4 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_2 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_2" -m "fixes 1.2 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_1_0 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-1_0" -m "fixes 1.0 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_30 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_30" -m "fixes 0.9.30 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_28 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_28" -m "fixes 0.9.28 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_26 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_26" -m "fixes 0.9.26 branch" $0'
   git rev-list --in-commit-order  --reverse   main..fixes_0_9_24 | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "fixes-0_9_24" -m "fixes 0.9.24 branch" $0'
   
   git rev-list --in-commit-order  --reverse   main..free-sparta | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "free-sparta" -m "free-sparta branch" $0'
   git rev-list --in-commit-order  --reverse   main..gtk-splitup | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "gtk-splitup" -m "gtk-splitup branch" $0'
   git rev-list --in-commit-order  --reverse   main..lcl-smartlink | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "lcl-smartlink" -m "lcl-smartlink branch" $0'
   git rev-list --in-commit-order  --reverse   main..listviewsortindicator | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "listviewsortindicator" -m "listviewsortindicator branch" $0'
   git rev-list --in-commit-order  --reverse   main..macosfullkeyboardaccess | head -n 1 | xargs -L1 /bin/sh -c 'git tag -a "macosfullkeyboardaccess" -m "macosfullkeyboardaccess branch" $0'

Comparing data before/after filter-branch

With subject line / Will show changes in subject

   git log --pretty='format:%ae %an %cd  %s %d' --graph --all > ../log.txt

Without subject line / Should have no diff

  • This compares the graph and committers => Thus it will show if all commits are present, and if all branches are correct
   git log --pretty='format:%ae %an %cd %d' --graph --all > ../log.txt
  • compare the changes to files (gitignore / gitattributes will diff for the top commit(s)) / --graph can be added too
   git log --all --pretty='%ae %an %cd %d' --stat > ../statlog.txt
  • Compare the overall amount of lines in the log.
   git log --all | wc -l


Make sure you do that after "clean up" in each step. Otherwise the internal subgit or filter-branch revs will go into the compare.

Also "removing the binaries" introduces a diff.

Data

Please mail me for any changes. Do not make changes here. Otherwise the changes will not make it into the conversion.

.gitignore

   **/*.o
   **/*.ppu
   **/*.exe
   **/*.bak
   **/*.diff
   **/*.patch
   **/units/
   **/lib/
   **/*;[1-9]
   **/*.~*
   **/*.*~
   **/*.fpcunit.ini
   **/*.lps
   **/*.compiled
   /docs/chm/**/*.chm
   /docs/chm/**/*.xct
   /docs/chm/**/*.txt
   /libeay32.dll
   /ssleay32.dll
   /lazarus.cfg
   ide/revision.inc
   
   !**/patches/*.patch
   !**/patches/*.diff

.gitattributes

   * text=auto !eol
   **/*.pp text
   **/*.pas text
   **/*.inc text
   **/*.lfm text
   **/*.lrs text
   **/*.lps text
   **/*.lpr text
   **/*.lpi text
   **/*.lpk text
   **/*.lpl text
   **/*.lrj text
   **/*.dpr text
   **/*.dfm text
   **/*.pot text
   **/*.po text
   **/*.rc text
   **/*.txt text
   **/*.txt.sample text
   **/*.ini text
   **/*.cfg text
   **/*.iss text
   **/*.isl text
   **/*.xml text
   **/*.xsl text
   **/*.html text
   **/*.css text
   **/*.patch text
   **/*.md text
   **/*.bat text
   **/*.sh text
   **/*.compiled text
   **/*.bdsproj text
   **/*.plist text
   **/*.h text
   **/*.c text
   **/*.cpp text
   **/*.in text
   **/Makefile text
   **/Makefile.compiled text
   **/Makefile.fpc text
   **/*.exe -text
   **/*.dll -text
   **/*.so -text
   **/*.gif -text
   **/*.jpg -text
   **/*.ico -text
   **/*.svg -text
   **/*.bmp -text
   **/*.res -text
   **/*.ttf -text
   **/*.xpm -text
   **/*.cur -text
   **/*.db3 -text
   **/*.dbf -text
   **/*.mbf -text
   **/*.mdx -text
   **/*.fbk -text
   **/*.odt -text
   **/*.ods -text
   **/*.odg -text
   **/*.odp -text
   **/*.xls -text
   **/*.pdf -text
   **/*.zip -text
   **/*.rar -text
   **/*.tar -text
   **/*.tgz -text
   **/*.gz -text
   **/*.icns -text
   **/*.dcr -text
   components/aggpas/**/*.ppm -text
   **/*.app/Contents/MacOS/* -text