Hacking:Big Repo Break - ParabolaWiki

# Hacking:Big Repo Break

On August 17th, 2011 the repo broke really bad. This are the steps we took to recover it.

## First problem

User:Fauno discovers there were many repo-update processes hanging around and kills them. A manual update shows ~3000 packages needed sync. When that threshold is reached, User:Fauno discovers almost all packages were removed from repos and moved to old/packages, apparently by overzealous ftpdir-cleanup script.

## Recovery attempts

Fauno: modified dbscripts.git/repo-sanity-check into repo-restore-to-normal. The scripts takes packages from abs and looks for them on old/packages and places them back into staging/$repo. Then db-update is ran, but it fails because many "duplicate" packages where staged at the same time... [to be continued]

[some time after]

Jyaworski: On the server, there are files in ~ called reposcript.sh and broken-symlinks.txt. The reposcript goes through the repo and checks for broken symlinks. Currently, these symlinks are only echoed, not removed. The output of the command is in broken-symlinks.txt.

[some time after on IRC]

    <fauno> i don't get it
    <fauno> i asked jyaworski to find repos' broken symlinks
    <fauno> there're 23000 broken symlinks
    <fauno> that means everything on repos is broken ¬¬
    <fauno> i have ~15000 packages on pool
    <fauno> i'm going to do this:
    <fauno> * rename pool/packages to pool/restore
    <fauno> * run repo-restore-to-normal from pool/restore to staging/
    <fauno> only for [core]
    <fauno> and see how it goes
    <fauno> i also merged pool/community to pool/restore
    <fauno> restoring [core]...
    <fauno> it was working but then i realized some packages weren't supposed to be there

[some time later] Fauno realized repo-restore-to-normal had several bugs and after fixing them and emptying pool/packages, [core] could be restored. [to be continued]

Category:

- Security