<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>Luke Manning - Blog — GitHub</title>
        <link>https://lukemanning.ie/</link>
        <description>Breaking things. Building things. Writing about it. (tag: GitHub)</description>
        <lastBuildDate>Wed, 30 Sep 2026 12:46:24 GMT</lastBuildDate>
        <docs>https://validator.w3.org/feed/docs/rss2.html</docs>
        <generator>https://github.com/jpmonette/feed</generator>
        <language>en</language>
        <copyright>All rights reserved 2026, Luke Manning</copyright>
        <atom:link href="https://lukemanning.ie/feeds/github.xml" rel="self" type="application/rss+xml"/>
        <item>
            <title><![CDATA[My .gitignore Has Been Ignoring My Lockfile This Whole Time]]></title>
            <link>https://lukemanning.ie/blog/repo-audit-gitignore-lockfile</link>
            <guid isPermaLink="true">https://lukemanning.ie/blog/repo-audit-gitignore-lockfile</guid>
            <pubDate>Tue, 08 Sep 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[<p><strong>TL;DR:</strong> I asked opencode to audit this repo's config against best practices for a private solo repo. Secrets hygiene came back clean. The genuine surprise was line 7 of my own <code>.gitignore</code>, which said <code>/package-lock.json</code>, so my lockfile was never tracked, and Dependabot vulnerability alerts were switched off. Also ~40 stale branches, a loud <code>git branch -d</code> refusal, and the last command in the <code>&#x26;&#x26;</code> chain silently never running because of it. Fixed in commit <code>42cf1f0</code>, plus two git aliases that encode the right cleanup order, <code>git pull &#x26;&#x26; git sweep</code>.</p>
<hr>
<p>I asked my agent (<a href="/blog/anthropic-open-source-walled-garden-clawdbot-opencode">opencode</a>) to review this repo's config. The whole ask was to check ManningWorks/lukemanning-site against best practices for a private solo-dev repo and tell me if anything needs changing.</p>
<p>My honest expectation was a list of green ticks and maybe a nitpick about something I'd decided on purpose. The site builds, Vercel deploys it, nothing's on fire. What could be wrong?</p>
<p>Most of it was fine. Some of it really wasn't.</p>
<h2>The parts that were fine</h2>
<p>Secrets hygiene, the part I actually cared about, came back clean. <code>.env*</code> is gitignored and <code>.env.local</code> holds a real GITHUB_TOKEN and a DEV_TO_API_KEY, untracked. Better still, the audit ran this check</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">git</span><span style="color:#9ECBFF"> log</span><span style="color:#79B8FF"> --all</span><span style="color:#79B8FF"> --oneline</span><span style="color:#79B8FF"> --</span><span style="color:#9ECBFF"> '.env*'</span></span></code></pre>
<p>Empty output. No env file was ever committed, on any branch, ever. That's the check I'd never have thought to run.</p>
<p>Ignores for <code>node_modules</code>, <code>.next</code>, <code>.velite</code>, tsbuildinfo were all correct. No LICENSE file, also correct for a private repo. No license means all rights reserved by default.</p>
<h2>My own .gitignore was the problem</h2>
<p>Then it got to line 7 of my <code>.gitignore</code>.</p>
<pre><code>/package-lock.json
</code></pre>
<p>My own lockfile. At some point I told git to ignore it, deliberately, and then apparently never thought about it again.</p>
<p>I pushed back on this one, because ignoring a lockfile is a real opinion. Plenty of library maintainers do exactly that, and <a href="/blog/i-shipped-a-library-now-what">I've shipped a library myself</a>, so the instinct has history.</p>
<p>But this isn't a library, it's a deployed app. Vercel builds this site from the repo, and with no tracked lockfile every build resolves dependencies fresh. <a href="/blog/npm-package-json-vs-package-lock-json">What package-lock.json actually does</a> is pin those resolutions to exact versions, which is what makes deploys reproducible. Without it, a transitive dependency can publish a breaking version and break a deploy with zero code changes on my side. Dependabot also needs the lockfile to open version update PRs at all.</p>
<p>Extra mess in the same area was pnpm archaeology. A <code>.pnpmfile.cjs</code> in the repo root and a <code>pnpm</code> block in <code>package.json</code>, leftovers from a pnpm phase. Meanwhile <code>.npmrc</code> and <code>package-lock.json</code> say npm is the actual current tool, the lockfile sitting on disk, untracked. Two package managers' worth of config, one of them dead.</p>
<h2>Dependabot was off. One API call fixed it</h2>
<p><a href="https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts">Dependabot alerts</a> are GitHub watching my dependencies for known vulnerabilities and telling me when it finds one. They were off, so nothing would have told me when a dependency picked up a CVE. The audit caught it via the API.</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/vulnerability-alerts</span></span></code></pre>
<p>404. On this endpoint that's just how GitHub says off, which threw me because 404 usually means the repo name is wrong. The docs say 204 means enabled, and this endpoint never returns 200.</p>
<p>The fix was a PUT.</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#79B8FF"> -X</span><span style="color:#9ECBFF"> PUT</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/vulnerability-alerts</span></span></code></pre>
<p>Enabling alerts is free even on private repos. One API call.</p>
<h2>Branch protection needs GitHub Pro on private repos</h2>
<p>Branch protection on private repos needs GitHub Pro. The API says so directly.</p>
<pre><code>Upgrade to GitHub Pro or make this repository public
</code></pre>
<p>Secret scanning and push protection aren't available on free private repos either. My first thought was that both are moot for a solo dev, but that's only half true. Secret scanning would have real value here, because I'm the most likely person to leak my own secrets. That one's a genuine gap, and closing it costs GitHub Pro money. Push protection, fair enough, there's nobody else's push to protect against. At least now I know it's a plan limit and not a config miss.</p>
<h2>Forty stale branches and one silent failure</h2>
<p>The pile was ~40 stale branches, local and remote. Every PR merged on GitHub left its head branch behind, and I'd clearly never gone back to clean up. Mostly merged, long forgotten.</p>
<p>I set the bar before deleting anything. Only branches proven merged get deleted. Two checks. <code>git branch --merged master</code> and its <code>-r</code> twin for ancestry, plus a cross-check against <code>gh pr list --state merged</code>.</p>
<p>One branch failed the ancestry check and was still safe to delete, and that's the interesting case.</p>
<pre><code>8ca8fd1 feat: auto-inject referral boxes... (#26)
</code></pre>
<p>That's the output of <code>git log master..feat/referral-link-injection --oneline</code>. One commit, and it wasn't an ancestor of master. But the <code>(#26)</code> suffix is a <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges">squash-merge</a> fingerprint. My PRs land as squash merges, and squash doesn't carry a branch's commits into master. It forges one fresh commit and stamps "title (#PR)" on it.</p>
<p>Which leaves a question. That stamp belongs on the commit squash forges on master, not on anything sitting at a branch tip. How this one got there, I never fully reconstructed. My best guess is the branch picked up a squash-style commit somewhere along the way, a cherry-pick of one or a reset onto one.</p>
<p>The cross-check showed the branch actually went out through PR #53, same name, merged, feature live on master. It got reused across PRs at some point and the stale #26 fingerprint never washed off. An orphaned copy, in other words. Tip not in master's history anywhere, changes already live.</p>
<p>The branch was a husk.</p>
<p><code>git branch -d</code> would refuse it forever, so it needed <code>-D</code> (force-delete, the shortcut for <code>--delete --force</code>), with the receipt above as justification.</p>
<p>Then the batch deletion failed halfway through.</p>
<p>The chain was a long run of <code>git branch -d</code> calls for the proven-merged branches, then <code>git branch -D feat/referral-link-injection</code> at the very end. One of the <code>-d</code> deletions refused for <code>merge/review-publisher-published-posts-into-master</code> because it apparently wasn't merged into master, but it was. The reason is a rule I didn't know existed. From the <a href="https://git-scm.com/docs/git-branch">git man page on <code>-d</code></a>.</p>
<blockquote>
<p>The branch must be fully merged in its upstream branch, or in HEAD if no upstream was set.</p>
</blockquote>
<p>This branch tracked a differently named remote upstream, <code>origin/review/publisher-published-posts</code>. The refusal explained it clearly enough. Not yet merged to 'refs/remotes/origin/review/publisher-published-posts', even though it is merged to HEAD.</p>
<p>AI figured it out and helped me clean it up.</p>
<h2>Sweep before pull doesn't work</h2>
<p>First instinct for the cleanup was <code>git sweep &#x26;&#x26; git pull</code>. Wrong order, and for a different reason than the <code>-d</code> refusal earlier. The sweep's filter, <code>git branch --merged &#x3C;base></code>, reads whatever base branch it points at. That's a separate check from the <code>-d</code> safety rule above. The filter decides what makes the delete list, and the <code>-d</code> check gives each branch its final veto. If I swept before pulling, my local master didn't yet contain the merge I'd just done on GitHub, so the freshly merged branch never even made the delete list. No error, just a sweep that did nothing useful, so I needed a second sweep anyway. Pull first, then sweep.</p>
<p>The aliases that landed in <code>~/.gitconfig</code></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">[alias]</span></span>
<span class="line"><span style="color:#F97583">	sync</span><span style="color:#E1E4E8"> = </span><span style="color:#9ECBFF">"!f() { git pull &#x26;&#x26; git sweep; }; f"</span></span>
<span class="line"><span style="color:#F97583">	sweep</span><span style="color:#E1E4E8"> = </span><span style="color:#9ECBFF">"!f() { git fetch --prune; b=$(git symbolic-ref --short refs/remotes/origin/HEAD 2>/dev/null) || b=$(git rev-parse --verify -q main >/dev/null 2>&#x26;1 &#x26;&#x26; echo main || echo master); git branch --merged \"</span><span style="color:#E1E4E8">$b\</span><span style="color:#9ECBFF">" | grep -vE \"</span><span style="color:#E1E4E8">^\\*\</span><span style="color:#9ECBFF">" | grep -Fvx \"</span><span style="color:#E1E4E8">  $b\</span><span style="color:#9ECBFF">" | xargs -r git branch -d; }; f"</span></span></code></pre>
<p>(<code>-r</code> is a GNU xargs thing, it makes xargs do nothing on empty input. macOS's BSD xargs doesn't have it.)</p>
<p>The real protection for the drafts is the filter, not the <code>-d</code>. <code>git branch --merged &#x3C;base></code> only lists branches that are ancestors of the base branch, so the unmerged <code>feat/draft-*</code> branches never make the delete list at all. The <code>-d</code> is the backstop. And as that <code>merge/review-publisher...</code> refusal showed, the backstop has its own opinions about upstreams and can refuse mid-sweep. It just refuses safely, which is why it's still <code>-d</code> and not <code>-D</code>. Every sweep run so far: drafts untouched.</p>
<p>The first cut of the sweep alias had a bug I noticed after: the grep skipped any branch name containing "master", so <code>merge/review-publisher-published-posts-into-master</code> would never make the sweep list. It would just sit there forever. That bugged me enough to rewrite it, and the alias above is the rewrite. It resolves the base branch now, <code>origin/HEAD</code> if set, then a <code>main</code>/<code>master</code> fallback, and the greps only skip the current branch and the base itself. The <code>into-master</code> branch gets swept like everything else.</p>
<p>Then one more API call to set <code>delete_branch_on_merge=true</code>, so merged PR head branches auto-delete on the remote from now on, and the pile can't quietly rebuild itself.</p>]]></description>
            <content:encoded><![CDATA[<p><strong>TL;DR:</strong> I asked opencode to audit this repo's config against best practices for a private solo repo. Secrets hygiene came back clean. The genuine surprise was line 7 of my own <code>.gitignore</code>, which said <code>/package-lock.json</code>, so my lockfile was never tracked, and Dependabot vulnerability alerts were switched off. Also ~40 stale branches, a loud <code>git branch -d</code> refusal, and the last command in the <code>&#x26;&#x26;</code> chain silently never running because of it. Fixed in commit <code>42cf1f0</code>, plus two git aliases that encode the right cleanup order, <code>git pull &#x26;&#x26; git sweep</code>.</p>
<hr>
<p>I asked my agent (<a href="/blog/anthropic-open-source-walled-garden-clawdbot-opencode">opencode</a>) to review this repo's config. The whole ask was to check ManningWorks/lukemanning-site against best practices for a private solo-dev repo and tell me if anything needs changing.</p>
<p>My honest expectation was a list of green ticks and maybe a nitpick about something I'd decided on purpose. The site builds, Vercel deploys it, nothing's on fire. What could be wrong?</p>
<p>Most of it was fine. Some of it really wasn't.</p>
<h2>The parts that were fine</h2>
<p>Secrets hygiene, the part I actually cared about, came back clean. <code>.env*</code> is gitignored and <code>.env.local</code> holds a real GITHUB_TOKEN and a DEV_TO_API_KEY, untracked. Better still, the audit ran this check</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">git</span><span style="color:#9ECBFF"> log</span><span style="color:#79B8FF"> --all</span><span style="color:#79B8FF"> --oneline</span><span style="color:#79B8FF"> --</span><span style="color:#9ECBFF"> '.env*'</span></span></code></pre>
<p>Empty output. No env file was ever committed, on any branch, ever. That's the check I'd never have thought to run.</p>
<p>Ignores for <code>node_modules</code>, <code>.next</code>, <code>.velite</code>, tsbuildinfo were all correct. No LICENSE file, also correct for a private repo. No license means all rights reserved by default.</p>
<h2>My own .gitignore was the problem</h2>
<p>Then it got to line 7 of my <code>.gitignore</code>.</p>
<pre><code>/package-lock.json
</code></pre>
<p>My own lockfile. At some point I told git to ignore it, deliberately, and then apparently never thought about it again.</p>
<p>I pushed back on this one, because ignoring a lockfile is a real opinion. Plenty of library maintainers do exactly that, and <a href="/blog/i-shipped-a-library-now-what">I've shipped a library myself</a>, so the instinct has history.</p>
<p>But this isn't a library, it's a deployed app. Vercel builds this site from the repo, and with no tracked lockfile every build resolves dependencies fresh. <a href="/blog/npm-package-json-vs-package-lock-json">What package-lock.json actually does</a> is pin those resolutions to exact versions, which is what makes deploys reproducible. Without it, a transitive dependency can publish a breaking version and break a deploy with zero code changes on my side. Dependabot also needs the lockfile to open version update PRs at all.</p>
<p>Extra mess in the same area was pnpm archaeology. A <code>.pnpmfile.cjs</code> in the repo root and a <code>pnpm</code> block in <code>package.json</code>, leftovers from a pnpm phase. Meanwhile <code>.npmrc</code> and <code>package-lock.json</code> say npm is the actual current tool, the lockfile sitting on disk, untracked. Two package managers' worth of config, one of them dead.</p>
<h2>Dependabot was off. One API call fixed it</h2>
<p><a href="https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts">Dependabot alerts</a> are GitHub watching my dependencies for known vulnerabilities and telling me when it finds one. They were off, so nothing would have told me when a dependency picked up a CVE. The audit caught it via the API.</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/vulnerability-alerts</span></span></code></pre>
<p>404. On this endpoint that's just how GitHub says off, which threw me because 404 usually means the repo name is wrong. The docs say 204 means enabled, and this endpoint never returns 200.</p>
<p>The fix was a PUT.</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#79B8FF"> -X</span><span style="color:#9ECBFF"> PUT</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/vulnerability-alerts</span></span></code></pre>
<p>Enabling alerts is free even on private repos. One API call.</p>
<h2>Branch protection needs GitHub Pro on private repos</h2>
<p>Branch protection on private repos needs GitHub Pro. The API says so directly.</p>
<pre><code>Upgrade to GitHub Pro or make this repository public
</code></pre>
<p>Secret scanning and push protection aren't available on free private repos either. My first thought was that both are moot for a solo dev, but that's only half true. Secret scanning would have real value here, because I'm the most likely person to leak my own secrets. That one's a genuine gap, and closing it costs GitHub Pro money. Push protection, fair enough, there's nobody else's push to protect against. At least now I know it's a plan limit and not a config miss.</p>
<h2>Forty stale branches and one silent failure</h2>
<p>The pile was ~40 stale branches, local and remote. Every PR merged on GitHub left its head branch behind, and I'd clearly never gone back to clean up. Mostly merged, long forgotten.</p>
<p>I set the bar before deleting anything. Only branches proven merged get deleted. Two checks. <code>git branch --merged master</code> and its <code>-r</code> twin for ancestry, plus a cross-check against <code>gh pr list --state merged</code>.</p>
<p>One branch failed the ancestry check and was still safe to delete, and that's the interesting case.</p>
<pre><code>8ca8fd1 feat: auto-inject referral boxes... (#26)
</code></pre>
<p>That's the output of <code>git log master..feat/referral-link-injection --oneline</code>. One commit, and it wasn't an ancestor of master. But the <code>(#26)</code> suffix is a <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges">squash-merge</a> fingerprint. My PRs land as squash merges, and squash doesn't carry a branch's commits into master. It forges one fresh commit and stamps "title (#PR)" on it.</p>
<p>Which leaves a question. That stamp belongs on the commit squash forges on master, not on anything sitting at a branch tip. How this one got there, I never fully reconstructed. My best guess is the branch picked up a squash-style commit somewhere along the way, a cherry-pick of one or a reset onto one.</p>
<p>The cross-check showed the branch actually went out through PR #53, same name, merged, feature live on master. It got reused across PRs at some point and the stale #26 fingerprint never washed off. An orphaned copy, in other words. Tip not in master's history anywhere, changes already live.</p>
<p>The branch was a husk.</p>
<p><code>git branch -d</code> would refuse it forever, so it needed <code>-D</code> (force-delete, the shortcut for <code>--delete --force</code>), with the receipt above as justification.</p>
<p>Then the batch deletion failed halfway through.</p>
<p>The chain was a long run of <code>git branch -d</code> calls for the proven-merged branches, then <code>git branch -D feat/referral-link-injection</code> at the very end. One of the <code>-d</code> deletions refused for <code>merge/review-publisher-published-posts-into-master</code> because it apparently wasn't merged into master, but it was. The reason is a rule I didn't know existed. From the <a href="https://git-scm.com/docs/git-branch">git man page on <code>-d</code></a>.</p>
<blockquote>
<p>The branch must be fully merged in its upstream branch, or in HEAD if no upstream was set.</p>
</blockquote>
<p>This branch tracked a differently named remote upstream, <code>origin/review/publisher-published-posts</code>. The refusal explained it clearly enough. Not yet merged to 'refs/remotes/origin/review/publisher-published-posts', even though it is merged to HEAD.</p>
<p>AI figured it out and helped me clean it up.</p>
<h2>Sweep before pull doesn't work</h2>
<p>First instinct for the cleanup was <code>git sweep &#x26;&#x26; git pull</code>. Wrong order, and for a different reason than the <code>-d</code> refusal earlier. The sweep's filter, <code>git branch --merged &#x3C;base></code>, reads whatever base branch it points at. That's a separate check from the <code>-d</code> safety rule above. The filter decides what makes the delete list, and the <code>-d</code> check gives each branch its final veto. If I swept before pulling, my local master didn't yet contain the merge I'd just done on GitHub, so the freshly merged branch never even made the delete list. No error, just a sweep that did nothing useful, so I needed a second sweep anyway. Pull first, then sweep.</p>
<p>The aliases that landed in <code>~/.gitconfig</code></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">[alias]</span></span>
<span class="line"><span style="color:#F97583">	sync</span><span style="color:#E1E4E8"> = </span><span style="color:#9ECBFF">"!f() { git pull &#x26;&#x26; git sweep; }; f"</span></span>
<span class="line"><span style="color:#F97583">	sweep</span><span style="color:#E1E4E8"> = </span><span style="color:#9ECBFF">"!f() { git fetch --prune; b=$(git symbolic-ref --short refs/remotes/origin/HEAD 2>/dev/null) || b=$(git rev-parse --verify -q main >/dev/null 2>&#x26;1 &#x26;&#x26; echo main || echo master); git branch --merged \"</span><span style="color:#E1E4E8">$b\</span><span style="color:#9ECBFF">" | grep -vE \"</span><span style="color:#E1E4E8">^\\*\</span><span style="color:#9ECBFF">" | grep -Fvx \"</span><span style="color:#E1E4E8">  $b\</span><span style="color:#9ECBFF">" | xargs -r git branch -d; }; f"</span></span></code></pre>
<p>(<code>-r</code> is a GNU xargs thing, it makes xargs do nothing on empty input. macOS's BSD xargs doesn't have it.)</p>
<p>The real protection for the drafts is the filter, not the <code>-d</code>. <code>git branch --merged &#x3C;base></code> only lists branches that are ancestors of the base branch, so the unmerged <code>feat/draft-*</code> branches never make the delete list at all. The <code>-d</code> is the backstop. And as that <code>merge/review-publisher...</code> refusal showed, the backstop has its own opinions about upstreams and can refuse mid-sweep. It just refuses safely, which is why it's still <code>-d</code> and not <code>-D</code>. Every sweep run so far: drafts untouched.</p>
<p>The first cut of the sweep alias had a bug I noticed after: the grep skipped any branch name containing "master", so <code>merge/review-publisher-published-posts-into-master</code> would never make the sweep list. It would just sit there forever. That bugged me enough to rewrite it, and the alias above is the rewrite. It resolves the base branch now, <code>origin/HEAD</code> if set, then a <code>main</code>/<code>master</code> fallback, and the greps only skip the current branch and the base itself. The <code>into-master</code> branch gets swept like everything else.</p>
<p>Then one more API call to set <code>delete_branch_on_merge=true</code>, so merged PR head branches auto-delete on the remote from now on, and the pile can't quietly rebuild itself.</p>]]></content:encoded>
            <category>github</category>
        </item>
        <item>
            <title><![CDATA[My Agent Closed Most of the Open Projex Issues in One Round Without Me Reading the Code]]></title>
            <link>https://lukemanning.ie/blog/my-agent-closed-most-projex-issues-in-one-round</link>
            <guid isPermaLink="true">https://lukemanning.ie/blog/my-agent-closed-most-projex-issues-in-one-round</guid>
            <pubDate>Wed, 19 Aug 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[<p>On a Saturday afternoon in August, I told my agent to fix every open issue in the Projex repo and prepare a release. There were ten open. Some were HIGH priority. Some were LOW.</p>
<p>I went for a walk.</p>
<p>When I came back, the work had landed across three branches. Each subagent had worked in its own worktree on its own branch. The build was green on each one. The tests were green. The lint was green. The typecheck was green. The release manager subagent had drafted the changelog. Nine of the ten issues were closed. The tenth stayed open — it needed a real union restructure, not a mechanical fix.</p>
<p>I had not read a single line of the diff yet.</p>
<h2>What I actually asked for</h2>
<p>I had a backlog of small things in <a href="/blog/building-projex-retrospective">the Projex repo</a>. Tagged-union cleanups. Type tightening. Documentation gaps. A bug where one of the smart-grid props was a documented prop but a no-op at runtime. A redundancy where two functions with slightly different spellings did the same thing.</p>
<p>I described this to my main agent. The main agent looked at the issue tracker, saw the labels (<code>bug</code>, <code>enhancement</code>, <code>documentation</code>), grouped them by file area, and <a href="/blog/opencode-subagent-permissions-ordering-trap">dispatched three subagents</a> in parallel.</p>
<p>One subagent got the package.json + bundling issues. One got the type-system + tagged-union issues. One got the documentation + test-coverage issues. Each one worked in a separate worktree on a separate branch. Each one committed locally and reported back. A fourth subagent, the release manager, handled release prep alongside them: the version bump and the changelog.</p>
<p>I watched the transcript. Mostly I stayed out of the way. I made tea.</p>
<h2>What the subagents actually did</h2>
<p>I saw the dispatch messages. I saw the report-back messages. I did not read the intermediate diffs. The subagents were set up to commit per-issue. Each commit was meant to be independently reviewable.</p>
<p>A few things I noticed in the report-backs:</p>
<ul>
<li>One subagent caught a redundancy I hadn't seen. Two exported functions, <code>normalizeStats</code> and <code>normaliseStats</code>, with the American and British spellings. Both did the same thing. The codebase had drifted to the British spelling in the actual logic. The American spelling was the alias. Both were exported. The subagent deprecated the American one with a JSDoc tag and updated the docs to point at the British spelling.</li>
<li>One subagent found a related issue while fixing another one. While narrowing the <code>ProjectStats</code> union, it noticed <code>FetchProjectDataResult.commits</code> was using <code>undefined</code> while sibling fields used <code>null</code>. The subagent opened a new issue and included the fix in the same branch.</li>
<li>One subagent flagged a peer-dependency problem I had been ignoring for two months. The CLI packages (<code>ts-morph</code>, <code>chalk</code>, <code>@inquirer/prompts</code>, <code>commander</code>) were installed by every consumer, even ones who only imported the components. The subagent moved them to optional <code>peerDependencies</code> so consumers importing only components stopped dragging in the CLI bundle.</li>
</ul>
<p>None of these were in my original brief. The subagents went past the edges of what I asked for, in the direction of "things that were obviously wrong in the same file area."</p>
<h2>Where I read the code</h2>
<p>The first time I read any of the code was after all three subagents finished and opened their PRs. I skimmed the diffs before merging. Not a line-by-line review. A sanity check.</p>
<p>I was looking for decisions the AI made without asking me. Function names I wouldn't have picked. Behaviour that wasn't in the brief. Edits that touched code outside the file area I asked about. The kind of things a real code review catches, except I was reviewing the decisions, not the code.</p>
<p>Some of the diffs were four lines. Some were thirty. None of them were complex enough to need a real review. They were tagged-union narrowings, JSDoc additions, dependency relocations. The kind of work where you skim it once, you understand it, you move on.</p>
<p>If I'd skimmed each PR as it landed, I'd have read the rename with no idea the docs were about to change under it. Reading the batch, I could see the <code>normaliseStats</code> deprecation and the docs update pointing at it in the same sitting. The batch skim was faster than piecemeal would have been.</p>
<h2>Where I did intervene</h2>
<p>I didn't push back on any of the code. The three branches each shipped clean. I steered the architecture around the loop, not the code inside it.</p>
<p>The dispatch went out as three parallel <code>opencode run</code> invocations, not three subagents. I asked for that change when the opencode TUI failed on the first attempt and the right path was to skip the interactive layer. I also argued for splitting the release prep out from the fix work, because trying to do both in the same dispatch kept blocking on the release-manager hitting its timeout before the fix branches landed.</p>
<h2>What this loop replaced</h2>
<p>My previous loop was one PR at a time. I'd describe an issue to the agent. The agent would open a PR. I'd skim it, sanity-check the decisions, merge or push back. One issue, one PR, one round of skimming. Repeat.</p>
<p>For this kind of small mechanical work, that's fine. It works. But the context switching adds up. Each PR is its own session — its own dispatch, its own transcript, its own review pass. The overhead is small per PR and large per backlog.</p>
<p>The new loop:</p>
<ol>
<li>Describe the backlog.</li>
<li>Wait.</li>
<li>Skim the batch of PRs.</li>
<li>Push the release prep.</li>
</ol>
<p>The release prep runs alongside the fix work instead of after it. One description covers all of them.</p>
<p>I want to be careful about what I'm claiming here. I'm not claiming the subagents did better work than the agent would have done one PR at a time. Most of these issues were mechanical. The interesting decisions — which redundancy to deprecate, which naming to standardize — the agent would have surfaced them either way, given the brief. What I'm claiming is that the per-PR overhead moved out of my hands and the interesting decisions stayed in my hands.</p>
<h2>Reading at the end, not in the middle</h2>
<p>I did not read the code while it was being written.</p>
<p>In the old loop, I skimmed each PR after the agent opened it. One PR at a time.</p>
<p>In the new loop, the skim happened after the writing finished across all three PRs. The subagents were the feedback loop during the work. I was the feedback loop at the end.</p>
<p>There's a different cost structure. A wrong fix in the old loop was caught in the per-PR skim, or it shipped. A wrong fix in the new loop is caught in the batch skim, or it ships.</p>
<p>I shipped none of the wrong fixes in this batch. The issues were mechanical and the code area was small. If I'd asked the subagents to redesign the <code>normalise</code> function, I would have read every line, pushed back, rewritten pieces.</p>
<p>The loop works for the kind of work that has a clear right answer. The loop does not work for the kind of work that needs taste. I haven't found the line yet.</p>
<p>For this batch, the line was "moves stuff around, adds JSDoc, narrows types." Below the line, I delegated. Above the line, I didn't. The line is in a different place than I would have guessed.</p>
<h2>Running it again</h2>
<p>I'm going to run this loop again. On a different repo. On a different kind of work.</p>
<p>I want to see what happens when the issues aren't mechanical. I want to see where the line moves. I want to see what kinds of work I delegate that I later wish I hadn't, and what kinds I keep that the loop could have handled.</p>
<p>I'm not going to delegate design decisions. I'm not going to delegate <a href="/blog/i-shipped-a-library-now-what">"what should this library do"</a>. I'm going to delegate "make this library do what it already says it does, correctly."</p>
<p>The batch skim at the end is non-negotiable. That's the part I own.</p>]]></description>
            <content:encoded><![CDATA[<p>On a Saturday afternoon in August, I told my agent to fix every open issue in the Projex repo and prepare a release. There were ten open. Some were HIGH priority. Some were LOW.</p>
<p>I went for a walk.</p>
<p>When I came back, the work had landed across three branches. Each subagent had worked in its own worktree on its own branch. The build was green on each one. The tests were green. The lint was green. The typecheck was green. The release manager subagent had drafted the changelog. Nine of the ten issues were closed. The tenth stayed open — it needed a real union restructure, not a mechanical fix.</p>
<p>I had not read a single line of the diff yet.</p>
<h2>What I actually asked for</h2>
<p>I had a backlog of small things in <a href="/blog/building-projex-retrospective">the Projex repo</a>. Tagged-union cleanups. Type tightening. Documentation gaps. A bug where one of the smart-grid props was a documented prop but a no-op at runtime. A redundancy where two functions with slightly different spellings did the same thing.</p>
<p>I described this to my main agent. The main agent looked at the issue tracker, saw the labels (<code>bug</code>, <code>enhancement</code>, <code>documentation</code>), grouped them by file area, and <a href="/blog/opencode-subagent-permissions-ordering-trap">dispatched three subagents</a> in parallel.</p>
<p>One subagent got the package.json + bundling issues. One got the type-system + tagged-union issues. One got the documentation + test-coverage issues. Each one worked in a separate worktree on a separate branch. Each one committed locally and reported back. A fourth subagent, the release manager, handled release prep alongside them: the version bump and the changelog.</p>
<p>I watched the transcript. Mostly I stayed out of the way. I made tea.</p>
<h2>What the subagents actually did</h2>
<p>I saw the dispatch messages. I saw the report-back messages. I did not read the intermediate diffs. The subagents were set up to commit per-issue. Each commit was meant to be independently reviewable.</p>
<p>A few things I noticed in the report-backs:</p>
<ul>
<li>One subagent caught a redundancy I hadn't seen. Two exported functions, <code>normalizeStats</code> and <code>normaliseStats</code>, with the American and British spellings. Both did the same thing. The codebase had drifted to the British spelling in the actual logic. The American spelling was the alias. Both were exported. The subagent deprecated the American one with a JSDoc tag and updated the docs to point at the British spelling.</li>
<li>One subagent found a related issue while fixing another one. While narrowing the <code>ProjectStats</code> union, it noticed <code>FetchProjectDataResult.commits</code> was using <code>undefined</code> while sibling fields used <code>null</code>. The subagent opened a new issue and included the fix in the same branch.</li>
<li>One subagent flagged a peer-dependency problem I had been ignoring for two months. The CLI packages (<code>ts-morph</code>, <code>chalk</code>, <code>@inquirer/prompts</code>, <code>commander</code>) were installed by every consumer, even ones who only imported the components. The subagent moved them to optional <code>peerDependencies</code> so consumers importing only components stopped dragging in the CLI bundle.</li>
</ul>
<p>None of these were in my original brief. The subagents went past the edges of what I asked for, in the direction of "things that were obviously wrong in the same file area."</p>
<h2>Where I read the code</h2>
<p>The first time I read any of the code was after all three subagents finished and opened their PRs. I skimmed the diffs before merging. Not a line-by-line review. A sanity check.</p>
<p>I was looking for decisions the AI made without asking me. Function names I wouldn't have picked. Behaviour that wasn't in the brief. Edits that touched code outside the file area I asked about. The kind of things a real code review catches, except I was reviewing the decisions, not the code.</p>
<p>Some of the diffs were four lines. Some were thirty. None of them were complex enough to need a real review. They were tagged-union narrowings, JSDoc additions, dependency relocations. The kind of work where you skim it once, you understand it, you move on.</p>
<p>If I'd skimmed each PR as it landed, I'd have read the rename with no idea the docs were about to change under it. Reading the batch, I could see the <code>normaliseStats</code> deprecation and the docs update pointing at it in the same sitting. The batch skim was faster than piecemeal would have been.</p>
<h2>Where I did intervene</h2>
<p>I didn't push back on any of the code. The three branches each shipped clean. I steered the architecture around the loop, not the code inside it.</p>
<p>The dispatch went out as three parallel <code>opencode run</code> invocations, not three subagents. I asked for that change when the opencode TUI failed on the first attempt and the right path was to skip the interactive layer. I also argued for splitting the release prep out from the fix work, because trying to do both in the same dispatch kept blocking on the release-manager hitting its timeout before the fix branches landed.</p>
<h2>What this loop replaced</h2>
<p>My previous loop was one PR at a time. I'd describe an issue to the agent. The agent would open a PR. I'd skim it, sanity-check the decisions, merge or push back. One issue, one PR, one round of skimming. Repeat.</p>
<p>For this kind of small mechanical work, that's fine. It works. But the context switching adds up. Each PR is its own session — its own dispatch, its own transcript, its own review pass. The overhead is small per PR and large per backlog.</p>
<p>The new loop:</p>
<ol>
<li>Describe the backlog.</li>
<li>Wait.</li>
<li>Skim the batch of PRs.</li>
<li>Push the release prep.</li>
</ol>
<p>The release prep runs alongside the fix work instead of after it. One description covers all of them.</p>
<p>I want to be careful about what I'm claiming here. I'm not claiming the subagents did better work than the agent would have done one PR at a time. Most of these issues were mechanical. The interesting decisions — which redundancy to deprecate, which naming to standardize — the agent would have surfaced them either way, given the brief. What I'm claiming is that the per-PR overhead moved out of my hands and the interesting decisions stayed in my hands.</p>
<h2>Reading at the end, not in the middle</h2>
<p>I did not read the code while it was being written.</p>
<p>In the old loop, I skimmed each PR after the agent opened it. One PR at a time.</p>
<p>In the new loop, the skim happened after the writing finished across all three PRs. The subagents were the feedback loop during the work. I was the feedback loop at the end.</p>
<p>There's a different cost structure. A wrong fix in the old loop was caught in the per-PR skim, or it shipped. A wrong fix in the new loop is caught in the batch skim, or it ships.</p>
<p>I shipped none of the wrong fixes in this batch. The issues were mechanical and the code area was small. If I'd asked the subagents to redesign the <code>normalise</code> function, I would have read every line, pushed back, rewritten pieces.</p>
<p>The loop works for the kind of work that has a clear right answer. The loop does not work for the kind of work that needs taste. I haven't found the line yet.</p>
<p>For this batch, the line was "moves stuff around, adds JSDoc, narrows types." Below the line, I delegated. Above the line, I didn't. The line is in a different place than I would have guessed.</p>
<h2>Running it again</h2>
<p>I'm going to run this loop again. On a different repo. On a different kind of work.</p>
<p>I want to see what happens when the issues aren't mechanical. I want to see where the line moves. I want to see what kinds of work I delegate that I later wish I hadn't, and what kinds I keep that the loop could have handled.</p>
<p>I'm not going to delegate design decisions. I'm not going to delegate <a href="/blog/i-shipped-a-library-now-what">"what should this library do"</a>. I'm going to delegate "make this library do what it already says it does, correctly."</p>
<p>The batch skim at the end is non-negotiable. That's the part I own.</p>]]></content:encoded>
            <category>ai</category>
            <category>opencode</category>
            <category>projex</category>
            <category>github</category>
        </item>
        <item>
            <title><![CDATA[gh issue view Kept Failing Because Ubuntu's ESM Outranks GitHub's Repo]]></title>
            <link>https://lukemanning.ie/blog/gh-issue-view-failing-ubuntu-deprecation-apt-pin</link>
            <guid isPermaLink="true">https://lukemanning.ie/blog/gh-issue-view-failing-ubuntu-deprecation-apt-pin</guid>
            <pubDate>Fri, 14 Aug 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[<p><strong>TL;DR:</strong> Ubuntu 24.04's packaged gh (2.45.0) is broken. GitHub sunset Projects (classic), and 2.45.0 still requests that field in its GraphQL query, so what reads like a deprecation warning is actually a hard error that kills every <code>gh issue view</code> and <code>gh pr view</code>. Three problems stacked: the warning was the error, GitHub's apt repo lives at <code>/packages</code> not <code>/apt</code>, and Ubuntu ESM out-pins GitHub at priority 510 vs 500. Fixed on 2.97.0.</p>
<hr>
<p>I was trying to read an issue on this site's repo:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> issue</span><span style="color:#9ECBFF"> view</span><span style="color:#79B8FF"> 28</span><span style="color:#79B8FF"> --repo</span><span style="color:#9ECBFF"> ManningWorks/lukemanning-site</span><span style="color:#79B8FF"> --comments</span></span></code></pre>
<p>What came back was this:</p>
<pre><code>GraphQL: Projects (classic) is being deprecated in favor of the new Projects
experience, see: https://github.blog/changelog/2024-05-23-sunset-notice-projects-classic/.
(repository.issue.projectCards)
</code></pre>
<p>I asked my AI assistant (<a href="/blog/anthropic-open-source-walled-garden-clawdbot-opencode">opencode</a>) what it meant. It told me it was just a deprecation warning, not an error, and the command works fine.</p>
<p>Except it kept telling me the command had failed. So I pushed back: "You keep telling me that command fails though." It ran the command, and a different error surfaced. This one was mine. In my message I'd written <code>gh issue view 28 --repo --comments</code> with no repo value, assuming it would fill in the blanks. It ran exactly what I'd written. <code>--repo</code> with no value swallowed <code>--comments</code>:</p>
<pre><code>expected the "[HOST/]OWNER/REPO" format, got "--comments"
</code></pre>
<p>Fine. My shorthand, my missing value. With the repo filled in, the original failure came straight back:</p>
<p>Exit code 1. Zero bytes on stdout. The deprecation message, alone, on stderr.</p>
<p>That's when it clicked. The "warning" was the failure. The command produced nothing except a note formatted like a heads-up about the future.</p>
<p>The assistant then decided the deprecation message was a red herring that would only appear once the repo resolved. Also wrong. Twice in one conversation.</p>
<h2>Problem One: The Warning Was the Error</h2>
<p>Ubuntu ships gh 2.45.0. That version requests <code>repository.issue.projectCards</code> as part of its GraphQL query for issue views. GitHub sunset Projects (classic) in May 2024 and now returns that field as a hard GraphQL error. A hard error kills the whole query, so every <code>gh issue view</code> and <code>gh pr view</code> on 2.45.x fails with output that reads like a deprecation notice.</p>
<p>This is a known thing: <a href="https://github.com/cli/cli/issues/11992">cli/cli#11992</a>. The maintainers' answer is to stop using distro packages and install from GitHub's official repo. GitHub's <a href="https://github.com/cli/cli/blob/trunk/docs/install_linux.md">Linux install docs</a> now say it outright: as of November 2025 they strongly recommend the official Debian packages, since the community-distributed 2.45.x and 2.46.x are broken against current GitHub APIs.</p>
<p>So Ubuntu's gh is broken in a way GitHub officially acknowledges. Weirdly comforting.</p>
<h2>The Workaround</h2>
<p><code>gh api</code> goes through REST, not GraphQL, so it still worked:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/issues/28</span><span style="color:#79B8FF"> --jq</span><span style="color:#9ECBFF"> '.title, .body'</span></span>
<span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/issues/28/comments</span><span style="color:#79B8FF"> --jq</span><span style="color:#9ECBFF"> '.[] | .author.login, .body'</span></span></code></pre>
<p>Clunky, but it let me read the issue while I sorted out the upgrade.</p>
<h2>Problem Two: The Wrong Repo URL</h2>
<p>The assistant set up GitHub's apt repo from memory: <code>https://cli.github.com/apt</code>.</p>
<pre><code>The repository 'https://cli.github.com/apt stable Release' does not have a Release file.
</code></pre>
<p>That's a 404. The repo isn't there. I enjoyed this one. The assistant guessed a URL instead of checking the docs, which is the exact thing it tells me not to do constantly.</p>
<p>GitHub's own <a href="https://github.com/cli/cli/blob/trunk/docs/install_linux.md">install docs</a> say it's <code>https://cli.github.com/packages</code>. The assistant followed them this time, keyring staged through /tmp since wget can't write to <code>/etc/apt/keyrings</code> as me:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">wget</span><span style="color:#79B8FF"> -nv</span><span style="color:#79B8FF"> -O</span><span style="color:#9ECBFF"> /tmp/githubcli-archive-keyring.gpg</span><span style="color:#9ECBFF"> https://cli.github.com/packages/githubcli-archive-keyring.gpg</span></span>
<span class="line"><span style="color:#B392F0">cat</span><span style="color:#9ECBFF"> /tmp/githubcli-archive-keyring.gpg</span><span style="color:#F97583"> |</span><span style="color:#B392F0"> sudo</span><span style="color:#9ECBFF"> tee</span><span style="color:#9ECBFF"> /etc/apt/keyrings/githubcli-archive-keyring.gpg</span><span style="color:#F97583"> ></span><span style="color:#9ECBFF"> /dev/null</span></span>
<span class="line"><span style="color:#B392F0">sudo</span><span style="color:#9ECBFF"> chmod</span><span style="color:#9ECBFF"> go+r</span><span style="color:#9ECBFF"> /etc/apt/keyrings/githubcli-archive-keyring.gpg</span></span>
<span class="line"><span style="color:#79B8FF">echo</span><span style="color:#9ECBFF"> "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main"</span><span style="color:#F97583"> |</span><span style="color:#B392F0"> sudo</span><span style="color:#9ECBFF"> tee</span><span style="color:#9ECBFF"> /etc/apt/sources.list.d/github-cli.list</span></span></code></pre>
<p>Repo added, <code>apt update</code> clean, <code>apt install gh</code>... nothing. Still 2.45.0. Three failure layers, for anyone counting.</p>
<h2>Problem Three: ESM Refuses to Lose</h2>
<p><code>apt-cache policy gh</code> explained it:</p>
<pre><code>gh:
  Installed: 2.45.0-1ubuntu0.3+esm3
  Candidate: 2.45.0-1ubuntu0.3+esm3
  Version table:
     2.97.0 500
        500 https://cli.github.com/packages stable/main amd64 Packages
 *** 2.45.0-1ubuntu0.3+esm3 510
        510 https://esm.ubuntu.com/apps/ubuntu noble-apps-security/main amd64 Packages
</code></pre>
<p>GitHub's repo offers 2.97.0 at priority 500, the apt default. Ubuntu ESM pins its version at 510. ESM is Expanded Security Maintenance, the security-update stream that comes with Ubuntu Pro, and this machine is Ubuntu 24.04 (noble), amd64, with Pro enabled. Higher priority wins the candidate slot, even though ESM's 2.45.0 is ancient.</p>
<p>An explicit version bypasses the priority contest:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">sudo</span><span style="color:#9ECBFF"> apt</span><span style="color:#9ECBFF"> install</span><span style="color:#9ECBFF"> gh=</span><span style="color:#79B8FF">2.97.0</span></span></code></pre>
<p>That worked. <code>gh --version</code> reported 2.97.0, and <code>gh issue view 28 --comments</code> (from the repo root, so no <code>--repo</code> needed) finally exited 0 with full output.</p>
<h2>The Pin, Validated</h2>
<p>The explicit install worked, but the candidate was still ESM's 2.45.0. The assistant said I needed a pin file so GitHub's repo wins going forward. Before touching anything in <code>/etc/apt/preferences.d/</code>, I asked it to validate that. Do Ubuntu or GitHub actually document this, or is it forum folklore?</p>
<p>Three primary sources:</p>
<ol>
<li>
<p><strong>My own machine.</strong> <code>/etc/apt/preferences.d/ubuntu-pro-esm-apps</code>, shipped by ubuntu-pro-client, pins <code>release o=UbuntuESMApps</code> at 510. The file's own comment says it's deliberate: when ESM is enabled, ESM packages are preferred over non-ESM ones.</p>
</li>
<li>
<p><strong><code>man apt_preferences</code>.</strong> Priorities from 500 to 990 install a version unless a newer one is already installed. 1000 or higher is required to downgrade. So ESM can't drag me back to 2.45.0, but a plain <code>apt upgrade</code> would never pick up a future GitHub release either. I'd be frozen on 2.97.0.</p>
</li>
<li>
<p><strong>Canonical's tracker and docs.</strong> <a href="https://github.com/canonical/ubuntu-pro-client/issues/3330">ubuntu-pro-client issue #3330</a> is someone hitting exactly this: a newer fish from a PPA, ignored in favour of ESM's. A Canonical maintainer closed it as intended, because cloud customers didn't want ESM security fixes silently reverted. The stated remedy:</p>
<blockquote>
<p>manually configure apt... by creating a preferences file pinning the PPA to a higher version than ESM.</p>
</blockquote>
<p>Canonical's Pro Client docs say close to the same: give the third-party repo at least 510.</p>
</li>
</ol>
<p>So the pin is real, and documented. Good.</p>
<h2>Why 600 and Not 510</h2>
<p>Those two sources don't quite agree, it turns out. The docs allow equal: at least 510. The issue thread says higher than ESM. And what apt actually does when two repos sit at the same priority, I couldn't find in <code>man apt_preferences</code>. Maybe there's a tie-break rule somewhere. I didn't find it.</p>
<p>I let the assistant pick the number in the end, within the bounds the man page had already given me. It went with 600: unconditionally "GitHub's repo is authoritative for gh", above both suggested floors, and below 1000 so nothing can ever downgrade me.</p>
<p>Works for me. 510 would satisfy the docs, but a pin that only ties the thing it's fighting isn't a config I want to reason about again in six months.</p>
<p><code>/etc/apt/preferences.d/gh</code>:</p>
<pre><code># gh: prefer GitHub official apt repo over Ubuntu ESM pin (510).
# ESM gh 2.45.x is broken vs current GitHub APIs - see cli/cli#11992.
# Canonical documents this remedy: https://documentation.ubuntu.com/pro-client/en/latest/explanations/about_esm/
Package: gh
Pin: origin cli.github.com
Pin-Priority: 600
</code></pre>
<p>Then I verified with <code>apt-cache policy gh</code>, which now shows <code>*** 2.97.0 600</code>. I checked because the assistant had flagged the failure mode to watch for: if the origin in the pin doesn't match the host in the sources list, there's no error. The pin just doesn't apply.</p>
<h2>What Stuck</h2>
<p>Any one of these three alone would have been a <a href="/blog/git-corrupt-object-recovery">proper rabbit hole</a>.</p>
<p>The assistant was confidently wrong twice, and that's the part I keep relearning. First that the message was harmless, then that it was a red herring. Both times, actually running the command with stdout and stderr separated settled it in seconds. Asking an AI what a command does is <a href="/blog/premature-optimization-multi-agent-prompts">asking it to hallucinate from training data</a>. The apt URL guess showed exactly how that goes.</p>
<p>gh works now. Which is nice, because I originally just wanted to read one issue.</p>]]></description>
            <content:encoded><![CDATA[<p><strong>TL;DR:</strong> Ubuntu 24.04's packaged gh (2.45.0) is broken. GitHub sunset Projects (classic), and 2.45.0 still requests that field in its GraphQL query, so what reads like a deprecation warning is actually a hard error that kills every <code>gh issue view</code> and <code>gh pr view</code>. Three problems stacked: the warning was the error, GitHub's apt repo lives at <code>/packages</code> not <code>/apt</code>, and Ubuntu ESM out-pins GitHub at priority 510 vs 500. Fixed on 2.97.0.</p>
<hr>
<p>I was trying to read an issue on this site's repo:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> issue</span><span style="color:#9ECBFF"> view</span><span style="color:#79B8FF"> 28</span><span style="color:#79B8FF"> --repo</span><span style="color:#9ECBFF"> ManningWorks/lukemanning-site</span><span style="color:#79B8FF"> --comments</span></span></code></pre>
<p>What came back was this:</p>
<pre><code>GraphQL: Projects (classic) is being deprecated in favor of the new Projects
experience, see: https://github.blog/changelog/2024-05-23-sunset-notice-projects-classic/.
(repository.issue.projectCards)
</code></pre>
<p>I asked my AI assistant (<a href="/blog/anthropic-open-source-walled-garden-clawdbot-opencode">opencode</a>) what it meant. It told me it was just a deprecation warning, not an error, and the command works fine.</p>
<p>Except it kept telling me the command had failed. So I pushed back: "You keep telling me that command fails though." It ran the command, and a different error surfaced. This one was mine. In my message I'd written <code>gh issue view 28 --repo --comments</code> with no repo value, assuming it would fill in the blanks. It ran exactly what I'd written. <code>--repo</code> with no value swallowed <code>--comments</code>:</p>
<pre><code>expected the "[HOST/]OWNER/REPO" format, got "--comments"
</code></pre>
<p>Fine. My shorthand, my missing value. With the repo filled in, the original failure came straight back:</p>
<p>Exit code 1. Zero bytes on stdout. The deprecation message, alone, on stderr.</p>
<p>That's when it clicked. The "warning" was the failure. The command produced nothing except a note formatted like a heads-up about the future.</p>
<p>The assistant then decided the deprecation message was a red herring that would only appear once the repo resolved. Also wrong. Twice in one conversation.</p>
<h2>Problem One: The Warning Was the Error</h2>
<p>Ubuntu ships gh 2.45.0. That version requests <code>repository.issue.projectCards</code> as part of its GraphQL query for issue views. GitHub sunset Projects (classic) in May 2024 and now returns that field as a hard GraphQL error. A hard error kills the whole query, so every <code>gh issue view</code> and <code>gh pr view</code> on 2.45.x fails with output that reads like a deprecation notice.</p>
<p>This is a known thing: <a href="https://github.com/cli/cli/issues/11992">cli/cli#11992</a>. The maintainers' answer is to stop using distro packages and install from GitHub's official repo. GitHub's <a href="https://github.com/cli/cli/blob/trunk/docs/install_linux.md">Linux install docs</a> now say it outright: as of November 2025 they strongly recommend the official Debian packages, since the community-distributed 2.45.x and 2.46.x are broken against current GitHub APIs.</p>
<p>So Ubuntu's gh is broken in a way GitHub officially acknowledges. Weirdly comforting.</p>
<h2>The Workaround</h2>
<p><code>gh api</code> goes through REST, not GraphQL, so it still worked:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/issues/28</span><span style="color:#79B8FF"> --jq</span><span style="color:#9ECBFF"> '.title, .body'</span></span>
<span class="line"><span style="color:#B392F0">gh</span><span style="color:#9ECBFF"> api</span><span style="color:#9ECBFF"> repos/ManningWorks/lukemanning-site/issues/28/comments</span><span style="color:#79B8FF"> --jq</span><span style="color:#9ECBFF"> '.[] | .author.login, .body'</span></span></code></pre>
<p>Clunky, but it let me read the issue while I sorted out the upgrade.</p>
<h2>Problem Two: The Wrong Repo URL</h2>
<p>The assistant set up GitHub's apt repo from memory: <code>https://cli.github.com/apt</code>.</p>
<pre><code>The repository 'https://cli.github.com/apt stable Release' does not have a Release file.
</code></pre>
<p>That's a 404. The repo isn't there. I enjoyed this one. The assistant guessed a URL instead of checking the docs, which is the exact thing it tells me not to do constantly.</p>
<p>GitHub's own <a href="https://github.com/cli/cli/blob/trunk/docs/install_linux.md">install docs</a> say it's <code>https://cli.github.com/packages</code>. The assistant followed them this time, keyring staged through /tmp since wget can't write to <code>/etc/apt/keyrings</code> as me:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">wget</span><span style="color:#79B8FF"> -nv</span><span style="color:#79B8FF"> -O</span><span style="color:#9ECBFF"> /tmp/githubcli-archive-keyring.gpg</span><span style="color:#9ECBFF"> https://cli.github.com/packages/githubcli-archive-keyring.gpg</span></span>
<span class="line"><span style="color:#B392F0">cat</span><span style="color:#9ECBFF"> /tmp/githubcli-archive-keyring.gpg</span><span style="color:#F97583"> |</span><span style="color:#B392F0"> sudo</span><span style="color:#9ECBFF"> tee</span><span style="color:#9ECBFF"> /etc/apt/keyrings/githubcli-archive-keyring.gpg</span><span style="color:#F97583"> ></span><span style="color:#9ECBFF"> /dev/null</span></span>
<span class="line"><span style="color:#B392F0">sudo</span><span style="color:#9ECBFF"> chmod</span><span style="color:#9ECBFF"> go+r</span><span style="color:#9ECBFF"> /etc/apt/keyrings/githubcli-archive-keyring.gpg</span></span>
<span class="line"><span style="color:#79B8FF">echo</span><span style="color:#9ECBFF"> "deb [arch=amd64 signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main"</span><span style="color:#F97583"> |</span><span style="color:#B392F0"> sudo</span><span style="color:#9ECBFF"> tee</span><span style="color:#9ECBFF"> /etc/apt/sources.list.d/github-cli.list</span></span></code></pre>
<p>Repo added, <code>apt update</code> clean, <code>apt install gh</code>... nothing. Still 2.45.0. Three failure layers, for anyone counting.</p>
<h2>Problem Three: ESM Refuses to Lose</h2>
<p><code>apt-cache policy gh</code> explained it:</p>
<pre><code>gh:
  Installed: 2.45.0-1ubuntu0.3+esm3
  Candidate: 2.45.0-1ubuntu0.3+esm3
  Version table:
     2.97.0 500
        500 https://cli.github.com/packages stable/main amd64 Packages
 *** 2.45.0-1ubuntu0.3+esm3 510
        510 https://esm.ubuntu.com/apps/ubuntu noble-apps-security/main amd64 Packages
</code></pre>
<p>GitHub's repo offers 2.97.0 at priority 500, the apt default. Ubuntu ESM pins its version at 510. ESM is Expanded Security Maintenance, the security-update stream that comes with Ubuntu Pro, and this machine is Ubuntu 24.04 (noble), amd64, with Pro enabled. Higher priority wins the candidate slot, even though ESM's 2.45.0 is ancient.</p>
<p>An explicit version bypasses the priority contest:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">sudo</span><span style="color:#9ECBFF"> apt</span><span style="color:#9ECBFF"> install</span><span style="color:#9ECBFF"> gh=</span><span style="color:#79B8FF">2.97.0</span></span></code></pre>
<p>That worked. <code>gh --version</code> reported 2.97.0, and <code>gh issue view 28 --comments</code> (from the repo root, so no <code>--repo</code> needed) finally exited 0 with full output.</p>
<h2>The Pin, Validated</h2>
<p>The explicit install worked, but the candidate was still ESM's 2.45.0. The assistant said I needed a pin file so GitHub's repo wins going forward. Before touching anything in <code>/etc/apt/preferences.d/</code>, I asked it to validate that. Do Ubuntu or GitHub actually document this, or is it forum folklore?</p>
<p>Three primary sources:</p>
<ol>
<li>
<p><strong>My own machine.</strong> <code>/etc/apt/preferences.d/ubuntu-pro-esm-apps</code>, shipped by ubuntu-pro-client, pins <code>release o=UbuntuESMApps</code> at 510. The file's own comment says it's deliberate: when ESM is enabled, ESM packages are preferred over non-ESM ones.</p>
</li>
<li>
<p><strong><code>man apt_preferences</code>.</strong> Priorities from 500 to 990 install a version unless a newer one is already installed. 1000 or higher is required to downgrade. So ESM can't drag me back to 2.45.0, but a plain <code>apt upgrade</code> would never pick up a future GitHub release either. I'd be frozen on 2.97.0.</p>
</li>
<li>
<p><strong>Canonical's tracker and docs.</strong> <a href="https://github.com/canonical/ubuntu-pro-client/issues/3330">ubuntu-pro-client issue #3330</a> is someone hitting exactly this: a newer fish from a PPA, ignored in favour of ESM's. A Canonical maintainer closed it as intended, because cloud customers didn't want ESM security fixes silently reverted. The stated remedy:</p>
<blockquote>
<p>manually configure apt... by creating a preferences file pinning the PPA to a higher version than ESM.</p>
</blockquote>
<p>Canonical's Pro Client docs say close to the same: give the third-party repo at least 510.</p>
</li>
</ol>
<p>So the pin is real, and documented. Good.</p>
<h2>Why 600 and Not 510</h2>
<p>Those two sources don't quite agree, it turns out. The docs allow equal: at least 510. The issue thread says higher than ESM. And what apt actually does when two repos sit at the same priority, I couldn't find in <code>man apt_preferences</code>. Maybe there's a tie-break rule somewhere. I didn't find it.</p>
<p>I let the assistant pick the number in the end, within the bounds the man page had already given me. It went with 600: unconditionally "GitHub's repo is authoritative for gh", above both suggested floors, and below 1000 so nothing can ever downgrade me.</p>
<p>Works for me. 510 would satisfy the docs, but a pin that only ties the thing it's fighting isn't a config I want to reason about again in six months.</p>
<p><code>/etc/apt/preferences.d/gh</code>:</p>
<pre><code># gh: prefer GitHub official apt repo over Ubuntu ESM pin (510).
# ESM gh 2.45.x is broken vs current GitHub APIs - see cli/cli#11992.
# Canonical documents this remedy: https://documentation.ubuntu.com/pro-client/en/latest/explanations/about_esm/
Package: gh
Pin: origin cli.github.com
Pin-Priority: 600
</code></pre>
<p>Then I verified with <code>apt-cache policy gh</code>, which now shows <code>*** 2.97.0 600</code>. I checked because the assistant had flagged the failure mode to watch for: if the origin in the pin doesn't match the host in the sources list, there's no error. The pin just doesn't apply.</p>
<h2>What Stuck</h2>
<p>Any one of these three alone would have been a <a href="/blog/git-corrupt-object-recovery">proper rabbit hole</a>.</p>
<p>The assistant was confidently wrong twice, and that's the part I keep relearning. First that the message was harmless, then that it was a red herring. Both times, actually running the command with stdout and stderr separated settled it in seconds. Asking an AI what a command does is <a href="/blog/premature-optimization-multi-agent-prompts">asking it to hallucinate from training data</a>. The apt URL guess showed exactly how that goes.</p>
<p>gh works now. Which is nice, because I originally just wanted to read one issue.</p>]]></content:encoded>
            <category>github</category>
            <category>ubuntu</category>
        </item>
        <item>
            <title><![CDATA[Git Said My Objects Were Corrupt - How I Recovered Without Losing Anything]]></title>
            <link>https://lukemanning.ie/blog/git-corrupt-object-recovery</link>
            <guid isPermaLink="true">https://lukemanning.ie/blog/git-corrupt-object-recovery</guid>
            <pubDate>Thu, 08 Jan 2026 00:00:00 GMT</pubDate>
            <description><![CDATA[<p>Running <code>git log</code> broke my repository.</p>
<p>I was on WSL (Windows Subsystem for Linux) working on my blog project like any other day. Ran <code>git log</code> to check my recent commits and got this:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> b92de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8">) is corrupt</span></span></code></pre>
<p>Three corrupt objects. Empty files. Admittedly I have somewhat limited Git experience, but I had never seen this before.</p>
<p>Was I about to lose commits?</p>
<hr>
<h2>The Diagnosis</h2>
<p>First, I needed to figure out how bad this was. I navigated to my repository root (where the <code>.git</code> folder is) and ran:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> fsck</span><span style="color:#79B8FF"> --full</span></span></code></pre>
<p>This command scans every object in your repository for corruption. The <code>--full</code> flag makes it check everything, not just what's reachable from current branches.</p>
<p>The output was worse than I thought:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> 287186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#E1E4E8">) is corrupt</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> b92de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8">) is corrupt</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> e2ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#E1E4E8">) is corrupt</span></span></code></pre>
<p>Three corrupt object files. All empty (0 bytes).</p>
<p>I didn't know what these objects were exactly - could have been commits, could have been file contents, could have been something else. That was the scary part. Were these unreferenced objects I wouldn't miss, or was I about to lose actual work?</p>
<p>Git splits object hashes into two-character directories (<code>b9/</code>) to avoid having millions of files in a single folder. That's normal Git internals.</p>
<p>If you're managing a home server and run into data recovery issues, I also wrote about <a href="/blog/unraid-docker-label-fix">fixing Unraid Docker containers after an upgrade</a> — less Git-related, but same problem-solving mindset.</p>
<h2>The Safety-First Question</h2>
<p>The initial solution I got was:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">rm</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span></span></code></pre>
<p>But I stopped. Because here's the thing: <strong>what if this doesn't work?</strong></p>
<p>I asked: "Do we have to <code>rm</code>? Can we <code>mv</code> in case this doesn't work? Or am I screwed either way?"</p>
<p>This turned out to be the right question. Even though these files were corrupt and empty, having a backup before doing anything destructive felt smarter. Even if it was arguably pointless. Sometimes you need a safety net, you know?</p>
<h2>The Actual Fix</h2>
<p>Here's what I did instead:</p>
<p><strong>Step 1: Create a backup location</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">mkdir</span><span style="color:#79B8FF"> -p</span><span style="color:#9ECBFF"> .git/corrupt-backup</span></span></code></pre>
<p>Will this work? I wasn't sure, but at least I'd have a backup if things went sideways.</p>
<p><strong>Step 2: Move the corrupt files (don't delete)</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">mv</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/corrupt-backup/</span></span></code></pre>
<p>Now the corrupt objects are gone from Git's perspective, but I still have them if I need to investigate later.</p>
<p>I did try running <code>git fetch origin</code> before moving the files, but Git still threw the same corruption error. The empty files were blocking everything - they had to go.</p>
<p><strong>Step 3: Recover from the remote</strong></p>
<p>This was the scary part. From the same repository root, I ran:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">git</span><span style="color:#9ECBFF"> fetch</span><span style="color:#9ECBFF"> origin</span></span></code></pre>
<p>Git compared my local repo to GitHub, saw the missing objects, and re-downloaded them to <code>.git/objects/</code>. Because these commits existed on GitHub, I got back the exact object files I'd lost.</p>
<p><code>origin</code> is the default name Git gives to your main remote. You can check your remotes with <code>git remote -v</code> if you're curious.</p>
<p><strong>Step 4: Verify everything is fixed</strong></p>
<p>That output looked good. But I needed to be sure. So I ran <code>fsck</code> again:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> fsck</span><span style="color:#79B8FF"> --full</span></span>
<span class="line"><span style="color:#B392F0">Checking</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> directories:</span><span style="color:#9ECBFF"> 100%</span><span style="color:#E1E4E8"> (256/256), done.</span></span>
<span class="line"><span style="color:#B392F0">Checking</span><span style="color:#9ECBFF"> objects:</span><span style="color:#9ECBFF"> 100%</span><span style="color:#E1E4E8"> (257/257), done.</span></span></code></pre>
<p><strong>Step 5: Confirm Git commands work</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> log</span><span style="color:#79B8FF"> --oneline</span><span style="color:#79B8FF"> -10</span></span>
<span class="line"><span style="color:#B392F0">b92de20</span><span style="color:#9ECBFF"> Adding</span><span style="color:#9ECBFF"> +2</span><span style="color:#9ECBFF"> reviews</span><span style="color:#9ECBFF"> for</span><span style="color:#9ECBFF"> setup</span><span style="color:#9ECBFF"> blog</span><span style="color:#9ECBFF"> post</span></span>
<span class="line"><span style="color:#B392F0">626e596</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">content</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> apply</span><span style="color:#9ECBFF"> multi-agent</span><span style="color:#9ECBFF"> review</span><span style="color:#9ECBFF"> fixes</span><span style="color:#9ECBFF"> to</span><span style="color:#9ECBFF"> setting-up-velite-nextjs-revised</span></span>
<span class="line"><span style="color:#B392F0">42f41d7</span><span style="color:#9ECBFF"> Token</span><span style="color:#9ECBFF"> optimization</span><span style="color:#9ECBFF"> plan</span><span style="color:#9ECBFF"> improvements</span></span>
<span class="line"><span style="color:#B392F0">db5c595</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">content</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> apply</span><span style="color:#9ECBFF"> multi-agent</span><span style="color:#9ECBFF"> review</span><span style="color:#9ECBFF"> fixes</span><span style="color:#9ECBFF"> to</span><span style="color:#9ECBFF"> setting-up-velite-nextjs-revised</span></span>
<span class="line"><span style="color:#B392F0">4bf342a</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">orchestrators</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> use</span><span style="color:#9ECBFF"> VOICE_GUIDE.md</span><span style="color:#9ECBFF"> instead</span><span style="color:#9ECBFF"> of</span><span style="color:#9ECBFF"> reading</span><span style="color:#9ECBFF"> recent</span><span style="color:#9ECBFF"> posts</span></span>
<span class="line"><span style="color:#79B8FF">...</span></span></code></pre>
<p>Everything works. Repository fully recovered.</p>
<h2>What If You Don't Have a Remote?</h2>
<p>Honestly, I got lucky. I had pushed everything to GitHub, so <code>git fetch origin</code> worked immediately.</p>
<p>If I hadn't had a remote, I would have been in trouble. I could have tried <code>git reflog</code> to see if Git still had references locally, but if the commits weren't pushed and weren't in reflog, they'd be gone.</p>
<p>This is why regular pushes matter - they're not just for sharing, they're for disaster recovery.</p>
<hr>
<h2>What I Learned</h2>
<p>Object corruption happens. System crashes, disk issues, power loss, process killed at the wrong moment - any of these can leave you with empty 0-byte files in <code>.git/objects/</code>. I don't know which one hit me.</p>
<p>Remote repositories are your safety net. Because I'd pushed these commits to GitHub, <code>git fetch</code> could restore them. The objects weren't actually lost, they just needed to be re-downloaded. If I hadn't pushed recently, this would've been scarier.</p>
<p>Always have a rollback plan. Moving files instead of deleting them was the right call. Even though they were corrupt and useless, having them in <code>.git/corrupt-backup/</code> meant I could investigate or restore if something went wrong. I asked myself: "before any destructive operation, do I have an escape hatch?" That question saved me stress, even though I didn't end up needing the backup.</p>
<p><code>git fsck</code> saved me time. When I first ran <code>git fsck --full</code>, I was overwhelmed by how verbose the output was. But once I realized it was showing me EXACTLY which three files were corrupt (with full paths and hashes), I knew what I needed to fix. Before this, I was guessing. After running fsck, I had precision.</p>
<hr>
<p>Git object corruption sounded terrifying when it first happened. But because I had GitHub and had pushed my commits, recovery was straightforward. The remote wasn't just for sharing code, it was my disaster recovery system.</p>
<p>And that <code>mv</code> instead of <code>rm</code>? Saved me from unnecessary stress. Even though the files were genuinely corrupt and I never needed them again, having that backup gave me confidence to proceed.</p>]]></description>
            <content:encoded><![CDATA[<p>Running <code>git log</code> broke my repository.</p>
<p>I was on WSL (Windows Subsystem for Linux) working on my blog project like any other day. Ran <code>git log</code> to check my recent commits and got this:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> b92de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8">) is corrupt</span></span></code></pre>
<p>Three corrupt objects. Empty files. Admittedly I have somewhat limited Git experience, but I had never seen this before.</p>
<p>Was I about to lose commits?</p>
<hr>
<h2>The Diagnosis</h2>
<p>First, I needed to figure out how bad this was. I navigated to my repository root (where the <code>.git</code> folder is) and ran:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> fsck</span><span style="color:#79B8FF"> --full</span></span></code></pre>
<p>This command scans every object in your repository for corruption. The <code>--full</code> flag makes it check everything, not just what's reachable from current branches.</p>
<p>The output was worse than I thought:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> 287186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#E1E4E8">) is corrupt</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> b92de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#E1E4E8">) is corrupt</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">error:</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> file</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#9ECBFF"> is</span><span style="color:#9ECBFF"> empty</span></span>
<span class="line"><span style="color:#B392F0">fatal:</span><span style="color:#9ECBFF"> loose</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> e2ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#E1E4E8"> (stored </span><span style="color:#9ECBFF">in</span><span style="color:#9ECBFF"> .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#E1E4E8">) is corrupt</span></span></code></pre>
<p>Three corrupt object files. All empty (0 bytes).</p>
<p>I didn't know what these objects were exactly - could have been commits, could have been file contents, could have been something else. That was the scary part. Were these unreferenced objects I wouldn't miss, or was I about to lose actual work?</p>
<p>Git splits object hashes into two-character directories (<code>b9/</code>) to avoid having millions of files in a single folder. That's normal Git internals.</p>
<p>If you're managing a home server and run into data recovery issues, I also wrote about <a href="/blog/unraid-docker-label-fix">fixing Unraid Docker containers after an upgrade</a> — less Git-related, but same problem-solving mindset.</p>
<h2>The Safety-First Question</h2>
<p>The initial solution I got was:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">rm</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span></span></code></pre>
<p>But I stopped. Because here's the thing: <strong>what if this doesn't work?</strong></p>
<p>I asked: "Do we have to <code>rm</code>? Can we <code>mv</code> in case this doesn't work? Or am I screwed either way?"</p>
<p>This turned out to be the right question. Even though these files were corrupt and empty, having a backup before doing anything destructive felt smarter. Even if it was arguably pointless. Sometimes you need a safety net, you know?</p>
<h2>The Actual Fix</h2>
<p>Here's what I did instead:</p>
<p><strong>Step 1: Create a backup location</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">mkdir</span><span style="color:#79B8FF"> -p</span><span style="color:#9ECBFF"> .git/corrupt-backup</span></span></code></pre>
<p>Will this work? I wasn't sure, but at least I'd have a backup if things went sideways.</p>
<p><strong>Step 2: Move the corrupt files (don't delete)</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">mv</span><span style="color:#9ECBFF"> .git/objects/28/7186e66df4cf7057d2897bc1da0027affbfbe5</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/b9/2de209f45b133daeeb97a46ea9ed354a2b9664</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/objects/e2/ec99bf1bb260a461c1b7670ad2e891ab0d7f88</span><span style="color:#79B8FF"> \</span></span>
<span class="line"><span style="color:#9ECBFF">   .git/corrupt-backup/</span></span></code></pre>
<p>Now the corrupt objects are gone from Git's perspective, but I still have them if I need to investigate later.</p>
<p>I did try running <code>git fetch origin</code> before moving the files, but Git still threw the same corruption error. The empty files were blocking everything - they had to go.</p>
<p><strong>Step 3: Recover from the remote</strong></p>
<p>This was the scary part. From the same repository root, I ran:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">git</span><span style="color:#9ECBFF"> fetch</span><span style="color:#9ECBFF"> origin</span></span></code></pre>
<p>Git compared my local repo to GitHub, saw the missing objects, and re-downloaded them to <code>.git/objects/</code>. Because these commits existed on GitHub, I got back the exact object files I'd lost.</p>
<p><code>origin</code> is the default name Git gives to your main remote. You can check your remotes with <code>git remote -v</code> if you're curious.</p>
<p><strong>Step 4: Verify everything is fixed</strong></p>
<p>That output looked good. But I needed to be sure. So I ran <code>fsck</code> again:</p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> fsck</span><span style="color:#79B8FF"> --full</span></span>
<span class="line"><span style="color:#B392F0">Checking</span><span style="color:#9ECBFF"> object</span><span style="color:#9ECBFF"> directories:</span><span style="color:#9ECBFF"> 100%</span><span style="color:#E1E4E8"> (256/256), done.</span></span>
<span class="line"><span style="color:#B392F0">Checking</span><span style="color:#9ECBFF"> objects:</span><span style="color:#9ECBFF"> 100%</span><span style="color:#E1E4E8"> (257/257), done.</span></span></code></pre>
<p><strong>Step 5: Confirm Git commands work</strong></p>
<pre class="shiki github-dark" style="background-color:#24292e;color:#e1e4e8" tabindex="0"><code><span class="line"><span style="color:#B392F0">$</span><span style="color:#9ECBFF"> git</span><span style="color:#9ECBFF"> log</span><span style="color:#79B8FF"> --oneline</span><span style="color:#79B8FF"> -10</span></span>
<span class="line"><span style="color:#B392F0">b92de20</span><span style="color:#9ECBFF"> Adding</span><span style="color:#9ECBFF"> +2</span><span style="color:#9ECBFF"> reviews</span><span style="color:#9ECBFF"> for</span><span style="color:#9ECBFF"> setup</span><span style="color:#9ECBFF"> blog</span><span style="color:#9ECBFF"> post</span></span>
<span class="line"><span style="color:#B392F0">626e596</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">content</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> apply</span><span style="color:#9ECBFF"> multi-agent</span><span style="color:#9ECBFF"> review</span><span style="color:#9ECBFF"> fixes</span><span style="color:#9ECBFF"> to</span><span style="color:#9ECBFF"> setting-up-velite-nextjs-revised</span></span>
<span class="line"><span style="color:#B392F0">42f41d7</span><span style="color:#9ECBFF"> Token</span><span style="color:#9ECBFF"> optimization</span><span style="color:#9ECBFF"> plan</span><span style="color:#9ECBFF"> improvements</span></span>
<span class="line"><span style="color:#B392F0">db5c595</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">content</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> apply</span><span style="color:#9ECBFF"> multi-agent</span><span style="color:#9ECBFF"> review</span><span style="color:#9ECBFF"> fixes</span><span style="color:#9ECBFF"> to</span><span style="color:#9ECBFF"> setting-up-velite-nextjs-revised</span></span>
<span class="line"><span style="color:#B392F0">4bf342a</span><span style="color:#9ECBFF"> refactor</span><span style="color:#E1E4E8">(</span><span style="color:#B392F0">orchestrators</span><span style="color:#E1E4E8">)</span><span style="color:#9ECBFF">:</span><span style="color:#9ECBFF"> use</span><span style="color:#9ECBFF"> VOICE_GUIDE.md</span><span style="color:#9ECBFF"> instead</span><span style="color:#9ECBFF"> of</span><span style="color:#9ECBFF"> reading</span><span style="color:#9ECBFF"> recent</span><span style="color:#9ECBFF"> posts</span></span>
<span class="line"><span style="color:#79B8FF">...</span></span></code></pre>
<p>Everything works. Repository fully recovered.</p>
<h2>What If You Don't Have a Remote?</h2>
<p>Honestly, I got lucky. I had pushed everything to GitHub, so <code>git fetch origin</code> worked immediately.</p>
<p>If I hadn't had a remote, I would have been in trouble. I could have tried <code>git reflog</code> to see if Git still had references locally, but if the commits weren't pushed and weren't in reflog, they'd be gone.</p>
<p>This is why regular pushes matter - they're not just for sharing, they're for disaster recovery.</p>
<hr>
<h2>What I Learned</h2>
<p>Object corruption happens. System crashes, disk issues, power loss, process killed at the wrong moment - any of these can leave you with empty 0-byte files in <code>.git/objects/</code>. I don't know which one hit me.</p>
<p>Remote repositories are your safety net. Because I'd pushed these commits to GitHub, <code>git fetch</code> could restore them. The objects weren't actually lost, they just needed to be re-downloaded. If I hadn't pushed recently, this would've been scarier.</p>
<p>Always have a rollback plan. Moving files instead of deleting them was the right call. Even though they were corrupt and useless, having them in <code>.git/corrupt-backup/</code> meant I could investigate or restore if something went wrong. I asked myself: "before any destructive operation, do I have an escape hatch?" That question saved me stress, even though I didn't end up needing the backup.</p>
<p><code>git fsck</code> saved me time. When I first ran <code>git fsck --full</code>, I was overwhelmed by how verbose the output was. But once I realized it was showing me EXACTLY which three files were corrupt (with full paths and hashes), I knew what I needed to fix. Before this, I was guessing. After running fsck, I had precision.</p>
<hr>
<p>Git object corruption sounded terrifying when it first happened. But because I had GitHub and had pushed my commits, recovery was straightforward. The remote wasn't just for sharing code, it was my disaster recovery system.</p>
<p>And that <code>mv</code> instead of <code>rm</code>? Saved me from unnecessary stress. Even though the files were genuinely corrupt and I never needed them again, having that backup gave me confidence to proceed.</p>]]></content:encoded>
            <category>github</category>
        </item>
    </channel>
</rss>