For awhile now we’ve been seeing issues where a git-mirror directory has become corrupted. I haven’t been able to narrow down under what conditions exactly it happens, but when it has happened, I see errors like (sorry about the redactions/obfuscations, we’re a bit security paranoid here :-) ):
2023-02-14 22:17:45 UTC Preparing working directory
2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/builds/mac-buildkite-10-xxx-xx-xx-1/org-name/repo-redacted
2023-02-14 22:17:45 UTC # Host "github.org-name.net" already in list of known hosts at "/Users/ci/.ssh/known_hosts"
2023-02-14 22:17:45 UTC # Using git-mirrors experiment 🧪
2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/repos
2023-02-14 22:17:45 UTC # Commit "0ea2865f5505f6abf85faf24ce50bb1aabebdf88" exists in mirror
2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/builds/mac-buildkite-10-xxx-xx-xx-1/org/repo-redacted
2023-02-14 22:17:45 UTC $ git remote set-url origin git@github.org-name.net:project/project-mobile.git
2023-02-14 22:17:45 UTC $ git submodule foreach --recursive "git clean -ffxdq"
2023-02-14 22:17:45 UTC $ git clean -ffxdq
2023-02-14 22:17:45 UTC # Fetch and checkout pull request head from GitHub
2023-02-14 22:17:45 UTC $ git fetch -v --prune -- origin refs/pull/11379/head
2023-02-14 22:17:45 UTC error: refs/remotes/origin/branch-redaccted-phase-1.2-part2 does not point to a valid object!
2023-02-14 22:17:46 UTC error: refs/remotes/origin/branch-redaccted-phase-1.2-part2 does not point to a valid object!
2023-02-14 22:17:46 UTC fatal: bad object HEAD
2023-02-14 22:17:46 UTC error: github.org-name.net:project/project-mobile.git did not send all necessary objects
2023-02-14 22:17:46 UTC
2023-02-14 22:17:46 UTC Auto packing the repository in background for optimum performance.
2023-02-14 22:17:46 UTC See "git help gc" for manual housekeeping.
2023-02-14 22:17:46 UTC warning: The last gc run reported the following. Please correct the root cause
2023-02-14 22:17:46 UTC and remove .git/gc.log.
2023-02-14 22:17:46 UTC Automatic cleanup will not be performed until the file is removed.
2023-02-14 22:17:46 UTC
2023-02-14 22:17:46 UTC fatal: bad object refs/remotes/origin/branch-redaccted-phase-1.2-part2
2023-02-14 22:17:46 UTC fatal: failed to run repack
Then, getting on the box and trying some git commands myself, I see that the repo seems corrupt beyond saving:
ci@buildkite-10-xxx-xx-xx dirname % git status
git status
fatal: bad object HEAD
ci@buildkite-10-xxx-xx-xx dirname % git fsck
git fsck
Checking object directories: 100% (256/256), done.
Checking object directories: 100% (256/256), done.
Checking objects: 100% (2830123/2830123), done.
error: refs/remotes/origin/branch-redacted-phase-1.2-part2: invalid sha1 pointer 36e4292bb7e145c21bafede7e770744fecc1cc7d
error: HEAD: invalid sha1 pointer ce9065576e659bdce9bfc430268f323f2fa4cdec
error: b28a9e7189249d4dd8ef266498bce370334eb53c: invalid sha1 pointer in cache-tree
dangling commit 2200e0e8e4bfd38c21f9420e4d1c10d5a9dca024
dangling commit c0002093f93f5971b1bc8be9ace9bc6ebcc35db3
... [Many many lines of dangling commit or dangling tag...]
dangling tag 53c5df929794b62a3bf8a79f2443ae8bda61dd24
... [Many many lines of dangling commit or dangling tag...]
Not sure what the best approach is, other than just turning off the git-mirrors experiment. For now I’ve just been deleting the mirror directory, but it seems to be happening a fair amount so that’s not really sustainable.