For awhile now we’ve been seeing issues where a git-mirror directory has become corrupted. I haven’t been able to narrow down under what conditions exactly it happens, but when it has happened, I see errors like (sorry about the redactions/obfuscations, we’re a bit security paranoid here :-) ):
2023-02-14 22:17:45 UTC Preparing working directory 2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/builds/mac-buildkite-10-xxx-xx-xx-1/org-name/repo-redacted 2023-02-14 22:17:45 UTC # Host "github.org-name.net" already in list of known hosts at "/Users/ci/.ssh/known_hosts" 2023-02-14 22:17:45 UTC # Using git-mirrors experiment 🧪 2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/repos 2023-02-14 22:17:45 UTC # Commit "0ea2865f5505f6abf85faf24ce50bb1aabebdf88" exists in mirror 2023-02-14 22:17:45 UTC $ cd /Users/ci/buildkite/builds/mac-buildkite-10-xxx-xx-xx-1/org/repo-redacted 2023-02-14 22:17:45 UTC $ git remote set-url origin email@example.com:project/project-mobile.git 2023-02-14 22:17:45 UTC $ git submodule foreach --recursive "git clean -ffxdq" 2023-02-14 22:17:45 UTC $ git clean -ffxdq 2023-02-14 22:17:45 UTC # Fetch and checkout pull request head from GitHub 2023-02-14 22:17:45 UTC $ git fetch -v --prune -- origin refs/pull/11379/head 2023-02-14 22:17:45 UTC error: refs/remotes/origin/branch-redaccted-phase-1.2-part2 does not point to a valid object! 2023-02-14 22:17:46 UTC error: refs/remotes/origin/branch-redaccted-phase-1.2-part2 does not point to a valid object! 2023-02-14 22:17:46 UTC fatal: bad object HEAD 2023-02-14 22:17:46 UTC error: github.org-name.net:project/project-mobile.git did not send all necessary objects 2023-02-14 22:17:46 UTC 2023-02-14 22:17:46 UTC Auto packing the repository in background for optimum performance. 2023-02-14 22:17:46 UTC See "git help gc" for manual housekeeping. 2023-02-14 22:17:46 UTC warning: The last gc run reported the following. Please correct the root cause 2023-02-14 22:17:46 UTC and remove .git/gc.log. 2023-02-14 22:17:46 UTC Automatic cleanup will not be performed until the file is removed. 2023-02-14 22:17:46 UTC 2023-02-14 22:17:46 UTC fatal: bad object refs/remotes/origin/branch-redaccted-phase-1.2-part2 2023-02-14 22:17:46 UTC fatal: failed to run repack
Then, getting on the box and trying some git commands myself, I see that the repo seems corrupt beyond saving:
ci@buildkite-10-xxx-xx-xx dirname % git status git status fatal: bad object HEAD ci@buildkite-10-xxx-xx-xx dirname % git fsck git fsck Checking object directories: 100% (256/256), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (2830123/2830123), done. error: refs/remotes/origin/branch-redacted-phase-1.2-part2: invalid sha1 pointer 36e4292bb7e145c21bafede7e770744fecc1cc7d error: HEAD: invalid sha1 pointer ce9065576e659bdce9bfc430268f323f2fa4cdec error: b28a9e7189249d4dd8ef266498bce370334eb53c: invalid sha1 pointer in cache-tree dangling commit 2200e0e8e4bfd38c21f9420e4d1c10d5a9dca024 dangling commit c0002093f93f5971b1bc8be9ace9bc6ebcc35db3 ... [Many many lines of dangling commit or dangling tag...] dangling tag 53c5df929794b62a3bf8a79f2443ae8bda61dd24 ... [Many many lines of dangling commit or dangling tag...]
Not sure what the best approach is, other than just turning off the git-mirrors experiment. For now I’ve just been deleting the mirror directory, but it seems to be happening a fair amount so that’s not really sustainable.