Hacking:repo-shrink of 2014-06 - ParabolaWiki

# Hacking:repo-shrink of 2014-06

abslibre.git has gotten large (>150MB) because some files have been committed that shouldn't have been. It is slow to clone, and is taking too much disk space on the server.

To correct this, lukeshu wrote a filter-branch script to shrink the repo. If you have an existing checkout of abslibre.git, you will get a warning about the upstream changing. You have two options:

1. Delete (or backup) your checkout, and clone the new version.
2. Run the script yourself on your copy. On lukeshu's box, it took about 13 minutes. On the git server, it took about 8 minutes.

If you have unpushed commits in your copy, and are not comfortable with git rebase, then running the script on you copy may be the better option. If you are comfortable with git rebase, then you don't need me to explain what to do.

## Running the script

The script is:

    cleanup.sh

    #!/bin/bash

    files=(
    	# sources
    	libre-testing/hplip-libre/hplip-3.12.4.tar.gz
    	libre-testing/hplip-libre/hplip-3.12.4.tar.gz.asc
    	pcr/ryzom-hg/.ryzom-hg-20131213

    	# sources
    	# I've verified that all of these PKGBUILD behave correctly without these files
    	libre/blackbox-libre/blackbox-0.70.1.tar.gz
    	libre/dvdrip-libre/dvdrip-0.98.11.tar.gz
    	pcr/python2-sfml2/1.4.zip
    	pcr/python2-sfml2/master.zip
    	pcr/qtkeychain/qtkeychain-0.1.zip

    	# compressed files
    	# I would rather they were uncompressed, but I guess they can stay
    	#libre/linux-libre/patch-3.14-gnu-3.14.1-gnu.xz
    	#pcr/debootstrap-libre/debootstrap.8.gz

    	# binaries
    	pcr/wuala # non-free, .jar file checked into git

    	# logs
    	libre/p7zip-libre/p7zip-libre-9.13-2-i686-build.log

    	# vim/kate swap files
    	kernels/aufs3-libre/.PKGBUILD.kate-swp
    	libre/gnu-ghostscript/.PKGBUILD.swp
    	libre/grub2/.archlinux_grub2_mkconfig_fixes.patch.swp
    	libre/grub2/.archlinux_grub_mkconfig_fixes.patch.swp
    	libre/hplip-libre/.hplip.install.swp
    	libre/iceweasel-libre/.libre.patch.swp
    	libre/kdebase-konqueror-libre/.PKGBUILD.swp
    	libre/linux-libre-tools/.PKGBUILD.swp
    	social/hunspell-pt-br/.PKGBUILD.kate-swp
    	'~emulatorman'/hunspell-pt-br/.PKGBUILD.kate-swp
    )

    git filter-branch --prune-empty --index-filter "git rm -r --cached --ignore-unmatch $(printf '%q ' "${files[@]}")" master

Before doing any of this, make sure you have no uncommitted files.

My process of running the script was:

    $ emacs cleanup.sh
    ( enter the above script )
    $ cd abslibre
    $ time bash ../cleanup.sh
    ( hundreds (thousands?) of lines of output ommitted )
    Rewrite 390443060f96a9599bdea9e18811d5a56e4c5b64 (6953/6954)rm 'pcr/qtkeychain/qtkeychain-0.1.zip'
    Rewrite d07c850a109062459c30ac81a4097d13603872ee (6954/6954)rm 'pcr/qtkeychain/qtkeychain-0.1.zip'

    Ref 'refs/heads/master' was rewritten

    real    13m26.555s
    user    3m31.543s
    sys     1m44.310s
    $ cd ..
    $ mv abslibre abslibre.bak
    $ git clone file:///${PWD}/abslibre.bak abslibre
    Cloning into 'abslibre'...
    remote: Counting objects: 39965, done.
    remote: Compressing objects: 100% (28925/28925), done.
    remote: Total 39965 (delta 14456), reused 27420 (delta 8807)
    Receiving objects: 100% (39965/39965), 27.82 MiB | 13.71 MiB/s, done.
    Resolving deltas: 100% (14456/14456), done.
    Checking connectivity... done.
    $ cp abslibre.bak/.git/config abslibre/.git/config
    $ du -h --max-depth 0 abslibre{,.bak}
    50M     abslibre
    223M    abslibre.bak

The reason for performing the clone is that git tries hard to not delete your data, and that the "large" version is still sitting there. It could be purged in the existing repository, but it is safer to clone it, especially if you aren't extremely familiar with git. You probably want to keep your abslibre.bak for a while, just in case.

Category:

- Hacking