An empty directory that does nothing except exist

My phone's Syncthing folder had been stuck at 99% before, and every time I'd written it off as a quirk or blamed the phone. This time it went to 99% again and I actually wanted to know why. The folder mirrors the drafts directory of this blog — on the NucBox it's ~/workspace/lukemanning-site/content/posts, on the Pixel it's just where I read drafts. I did the obvious thing and blamed the phone. Killed the app, reopened it, checked the wifi. Still 99%.

The phone was fine. I got Hermes to poke the NucBox side and the picture over there was completely different. The Pixel was connected, streaming away happily. The blog posts folder on the NucBox was sitting in an error state, about 5KB short of done, with this:

Error on folder "Blog Posts" (blog-posts-jl): folder marker missing
(this indicates potential data loss, search docs/forum to get information
about how to proceed)

Folder marker missing. Very dramatic error message for what turns out to be an empty directory.

What the marker is

Every Syncthing folder has a .stfolder directory inside it. It's empty. It does nothing except exist. When Syncthing opens a folder it checks for that directory, and if it's gone it refuses to write anything to the folder at all. The thinking is sound: if the marker vanished, maybe the folder contents got replaced by something that isn't a sync folder anymore, and barging in with writes could destroy data. So Syncthing stops and waits for a human.

In my case the contents were fine. The marker was just gone.

The fix is embarrassingly small:

mkdir ~/workspace/lukemanning-site/content/posts/.stfolder

Then a rescan through the API:

curl -X POST -H "X-API-Key: $KEY" \
  "http://127.0.0.1:8384/rest/db/scan?folder=blog-posts-jl"

And the log says the happy thing:

INFO: Cleared error on folder "Blog Posts" (blog-posts-jl)

Folder went to 100%, phone caught up. Except for one detail I hadn't clocked yet: the last 5KB it was stuck on was .obsidian config files. From the phone. The phone had been waiting to download its own settings back.

Why it kept happening

This error wasn't new to me. I'd hit it plenty of times before, and every time I figured it was a Syncthing quirk, or something to do with changes I was making on the phone. I'd ask Hermes to review, it would fix the folder, we'd move on. This time I wanted to understand why. So: journalctl.

Aug 20 16:48  Error on folder "Blog Posts": folder marker missing
Sep 09 19:01  Error on folder "Blog Posts": folder marker missing
Sep 11 06:46  Error on folder "Blog Posts": folder marker missing
Sep 17 00:16  Error on folder "Blog Posts": folder marker missing
Sep 18 22:10  Error on folder "Blog Posts": folder marker missing
Sep 29 17:44  Error on folder "Blog Posts": folder marker missing

Six times since August. Every few days the marker gets deleted, Syncthing notices on its next scan and locks the folder, and the phone quietly sticks at 99% until the marker is recreated and a rescan runs.

The git connection

.stfolder is an empty directory, and git cannot see empty directories. Not "doesn't track them well": at all. It's not in git status as untracked, it's not in any listing, git clean doesn't protect it. From git's point of view the directory does not exist, so from git's point of view deleting it changes nothing.

Which means any git clean -fd in this repo kills the marker while sparing everything that's actually in .gitignore. .next, node_modules, .obsidian all survive. The one item that matters to Syncthing dies.

My first guess was branch switching, or a stash doing something weird. I went looking for evidence and cleared both: plain git checkout and git stash can't delete the directory, because git can't see it in order to delete it. But there's a plausible chain. When git stash pop fails to restore untracked files, the documented remedy is git clean -fd to clear the way. This repo currently has eight stashes sitting in it, three on master, all three from dependabot work. And the reflog shows constant branch churn, including a string of back-and-forth checkouts inside a single hour. The suspect list is short: me or one of the agents. Nothing else has access to this box. One of us may well have followed git's advice and run the clean.

Nobody's logs confess to it though. I checked bash history, the OpenCode session database, Hermes session logs, all the cron jobs. Nothing that touches content/posts, no git clean anywhere.

So: no confession. But the fix doesn't depend on solving the whodunit. git clean spares ignored files, so the protection is one line in .gitignore:

# syncthing folder marker - untracked empty dir, killed by git clean -fd
content/posts/.stfolder/

I honestly don't know if this is the fully correct thing to do but it seems to be working well since making this change, and that's good enough for me.