I was wrong.

Not wrong about the architecture. Wrong about whether it was actually working.

TL;DR: I declared the Docker setup working. Then cron jobs vanished into the wrong database, sage was running two Hermes instances, and the actual bug turned out to be a Python loop that had nothing to do with Docker.

The Problem I Didn't Know I Had (Again)

I ran docker exec hermes-sage ps aux to verify sage was healthy. It showed one process. Good. I fired a test cron on sage. It said it created successfully. I fired a test cron on rex. It said it created successfully. I waited.

Nothing arrived.

I assumed the Telegram bots weren't set up correctly. I went down a whole path of checking bot tokens, allowed user lists, TELEGRAM_HOME_CHANNEL settings. Then I checked the logs and found something stranger.

Sage's cron engine wasn't running the jobs. Neither was rex's. They were being created (the CLI returned success), but they were going into the host's state database, not the container's.

And underneath that, I discovered something worse: sage wasn't running one Hermes instance. It was running two.

Why 'ps aux' Lied to Me

The Docker entrypoint for the Hermes image is /init, which is s6-overlay. s6-overlay scans /run/service/ and starts everything it finds there. My compose file passed --profile sage to the gateway command, which should have meant "run sage profile only."

What actually happened: the compose command: was ignored. The ENTRYPOINT (/init) runs as PID 1, starts s6, and s6 starts all the services in /run/service/, gateway-default and gateway-sage both. Two Hermes instances, same process tree.

The ps aux I'd run to "verify" sage was healthy showed one gateway at the top of the output, so I stopped reading. The output ran longer than one screen. When I finally ran docker exec hermes-sage ps aux | grep gateway, two gateway processes showed up: the one I expected, and a second one nested under s6 that I'd scrolled straight past.

The compose command: instruction doesn't override an ENTRYPOINT. It gets handed to the entrypoint as arguments instead. That's in Docker's docs under how ENTRYPOINT and CMD interact. I'd never internalised it.

The Cron Database Shell Game

Once I grasped that sage was running two instances, the cron mystery made more sense. The CLI command hermes cron create was running inside the container, but it reads its config from environment variables, HOME especially, since that's where Hermes looks for its config directory.

I checked what HOME actually was inside the container:

$ docker exec hermes-sage env | grep HOME
HOME=/home/luke

That's the hermes user's passwd entry, not /opt/data. So when hermes cron create ran without --profile sage, it looked for config at /home/luke/.hermes, which doesn't exist in the container, and fell back to writing jobs to /opt/data/cron/jobs.json instead of /opt/data/profiles/sage/cron/jobs.json.

The sage gateway was reading from the right place. The cron CLI was writing to the wrong place. Jobs created successfully. Jobs never fired.

The fix was passing --profile sage to every cron command inside the container. But I also had to fix HOME, which led to the next problem.

The su -m Trap

I wanted the hermes process to run with HOME=/opt/data so it read from the right config. The entrypoint script ran as root, then used su hermes to drop privileges. Easy enough.

su -m is meant to preserve the parent environment, so I set HOME=/opt/data before calling su and expected it to carry through. It didn't. The shell su spawned was resetting HOME back to the hermes user's passwd entry (/home/luke) on the way up, undoing what su -m had preserved. By the time hermes actually ran, HOME had been clobbered again.

The reliable fix was env -i, which wipes the environment entirely before setting only what I want. Nothing left for the shell to reset:

exec env -i HOME=/opt/data HERMES_HOME=/opt/data HERMES_PROFILE=sage \
  su -m -s /bin/sh hermes -c "exec hermes gateway run --profile sage"

env -i clears everything, the explicit vars set what I need, and su -m preserves that clean state. hermes starts with exactly the HOME I specify.

Custom Entrypoint: Replacing PID 1

To stop gateway-default from starting, I needed to keep s6 from scanning it. The cleanest approach was replacing PID 1 entirely with a custom script that:

1. Runs stage2-hook.sh for Hermes bootstrap (UID remapping, chown)
2. Waits for s6 to register its services
3. Kills gateway-default via s6-svc -d
4. Starts only the sage gateway, using that same env -i exec line from above

Then the compose file binds the script as the entrypoint:

entrypoint: ["/entrypoint.sh"]
volumes:
  - /home/luke/.hermes/docker/entrypoints/sage-only.sh:/entrypoint.sh:ro

Now sage starts exactly one Hermes instance. No gateway-default. No confusion.

The Telegram 'Chat Not Found' Problem

Once I had real isolation working, the crons fired. The sage cron engine ran its job. The rex cron engine ran its job. Both logged completed successfully.

No Telegram messages arrived.

The error: Telegram send failed: Chat not found.

Sage has its own Telegram bot. Rex has its own Telegram bot. Separate bots, separate tokens. A Telegram bot can only send messages to people who have messaged that specific bot first. I'd only ever messaged the default Hermes bot, which knew about my chat. Sage's bot had never seen me.

The second issue was TELEGRAM_HOME_CHANNEL=RealLukeManning in the .env. That's a username, not a chat ID. The Telegram Bot API accepts numeric chat_id for any chat, but only public channels and supergroups can be addressed via @username — for private chats (the home chat here), the API silently fails on a username. Sage was using the username and failing silently.

The fix: message each bot directly first, or use numeric chat IDs in the delivery target (telegram:491962736, not telegram:RealLukeManning).

The env_file Trap

I thought the Telegram issue was just setup. Then I noticed sage's logs showed its Telegram module attempting to connect, but the bot token wasn't in the container's config.

The .env with the bot token was on the host at ~/.hermes/profiles/sage/.env. The container mounts the profile directory to /opt/data/profiles/sage. The hermes process runs from /opt/data. No .env at /opt/data.

The compose file wasn't passing it through:

# Missing
env_file:
  - /home/luke/.hermes/profiles/sage/.env

Without the token, the Telegram connection failed gracefully, logged a warning, and didn't die. Messages never arrived.

The Final Test (Or So I Thought)

With all fixes in place, I fired simultaneous test crons on both containers. Both ran. Both Telegram bots delivered. Five seconds apart, completely independent.

Sage: one instance, sage profile, sage bot, sage cron. Rex: one instance, rex profile, rex bot, rex cron. Each container starting independently via systemd, each with its .env passed through, cron jobs created with --profile and numeric chat IDs.

I thought that was it. The containers were finally doing what I'd built them to do.

Then last Tuesday I got a message from Sage at 8am.

The daily system health check I'd set up months ago, before Docker and before any of this, was delivering through Sage's Telegram bot instead of the default gateway.

I swore up and down I'd fixed this. I had not fixed this.

What the Health Check Was Actually Doing

The health check runs every morning at 8am. Disk, swap, fail2ban, SSH failures. Then it delivers the result to Telegram. Crucially, it runs on the bare-metal default gateway, the one I never put in a container. Not sage, not rex.

The job itself was fine. The delivery was the problem. I opened the health-check cron job to see what it actually called, and found it was running a skill called hermes-telegram-send, one I didn't write, generated by an LLM using a template. That skill had one job: send a Telegram message from a cron context.

The skill worked by loading environment variables from profile .env files, in this order:

for profile in ["rex", "sage", "default"]:
    env_file = Path(f"/home/luke/.hermes/profiles/{profile}/.env")
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            # load the token

rex/.env exists, loads token. sage/.env exists, overwrites the token. default/.env doesn't exist, skipped.

Sage's Telegram bot token was the last one loaded. So when the health check cron ran, it used Sage's bot to send the message.

The message came from Sage. Because the skill hardcoded ["rex", "sage", "default"] and default didn't exist.

The Docker Containers Were Working Fine

This is the part that felt stupid to realise.

Sage was never misconfigured. She wasn't "replying when she shouldn't." She was doing exactly what her container was set up to do. The Docker isolation I'd spent a whole post documenting and debugging was completely correct. Separate containers, separate bots, separate processes. All working.

The bug wasn't in Docker. It wasn't in the architecture. It was in a Python loop in a skill that was generated without thinking about what happens when one of the hardcoded profile names doesn't exist.

The loop was supposed to try multiple profiles in case one didn't have a token. It would load rex, then sage, then default. The last one loaded would be the one used.

But there's no default profile directory. There's a bare-metal Hermes gateway running directly on the host, with its own .env at ~/.hermes/.env. That file has the actual Telegram token for the main bot.

The skill didn't know about ~/.hermes/.env. It only knew about the profile subdirectories.

The Real Fix: Baseline First, Never Overwrite

The skill now loads ~/.hermes/.env first as a baseline (the bare-metal gateway's token), then profile .env files only fill in empty slots. They never overwrite what's already set.

# ~/.hermes/.env loads FIRST as baseline
main_env = Path(f"{hermes_home}/.env")
if main_env.exists():
    for line in main_env.read_text().splitlines():
        if not line.strip() or line.lstrip().startswith("#") or "=" not in line:
            continue
        k, v = line.split("=", 1)
        os.environ[k] = v  # no guard, this is the baseline

# Profile files only fill empty slots, never overwrite
profile_order = ([current_profile] if current_profile else []) + ["rex", "sage"]
for profile in profile_order:
    env_file = Path(f"{hermes_home}/profiles/{profile}/.env")
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            if not line.strip() or line.lstrip().startswith("#") or "=" not in line:
                continue
            k, v = line.split("=", 1)
            if k not in os.environ:  # only fill empty slots
                os.environ[k] = v

The old code had two problems. First, when HERMES_PROFILE is empty (bare-metal default), the current profile list is empty and ["rex", "sage"] loads in that order, so sage overwrites rex. Second, the profile loop had no guard. os.environ[k] = v always overwrites, so even if the main .env loaded last, it was already too late. The fix: main .env first (no guard), profile files second (with if k not in os.environ guard).

The health check at 8am tomorrow should come from the right bot. For real this time.

What I Actually Learned

The implementation had three layers of subtle breakage. The compose command: doesn't override an ENTRYPOINT; if the image has one, the command gets passed to it as arguments. HOME matters more than I expected when su-ing to another user; env -i is the reliable way to lock it. And separate containers don't mean isolated if the entrypoint starts everything inside them regardless.

I fixed all of that. It was real work. And then the bug that actually mattered turned out to be none of it.

Three things I keep having to re-learn:

Infrastructure work doesn't protect you from code bugs. I spent weeks on Docker containers, systemd services, separate Telegram bots. All correct, all irrelevant to the actual symptom. A hardcoded list of profile names in a Python loop. The containers couldn't catch that, because containers don't catch Python logic errors.

LLM-generated skills inherit LLM failure modes. The skill came from a template with ["rex", "sage", "default"] baked in. Nobody stopped to ask what happens when default doesn't exist. That kind of assumption sits in code for months before anyone notices.

A fix I didn't verify was never really a fix. I thought I'd solved the Sage problem once isolation was working. The real fix had two subtle bugs hiding in it: an empty HERMES_PROFILE that silently disabled the "current profile first" logic, and a profile loop that overwrote tokens without checking if one was already set. I should have verified instead of trusting it.

The Docker setup is better now than before I started. The isolation is real. But the real fix required iterating on a Python loop that had nothing to do with Docker.

They were gateway profiles. In Hermes, the gateway is the long-running process that polls Telegram, runs scheduled jobs, and executes skills. "Profiles" means one gateway binary, one process manager, just different configs passed in via --profile.

One thing worth saying up front. This post is the "why I tried containers" half of the story. When I went to verify the setup, it wasn't actually working yet, and that's covered in part two of the series. I'm keeping this one focused on the reasoning and the container plumbing, because that part still stands even though my diagnosis turned out to be off.

The Problem I Didn't Know I Had

The symptom was cron replies coming from the wrong profile, and it was inconsistent, which is what made it hard to pin down. Sometimes a scheduled job would fire and default and sage would both reply. Sometimes only rex would send the update, when it wasn't his job to. It never happened with messages I sent directly — only cron.

My theory at the time was that the profiles shared too much. One scheduler, one process tree, no hard boundary between them, so jobs bled across. Three profiles all live in the same process, so when a cron job fired, I figured whichever gateway instance was free picked it up.

That theory turned out to be mostly wrong, which is a whole separate story. I'm laying it out anyway, because it's what drove me to containers, and the architecture reasoning is sound even if the diagnosis wasn't.

What I didn't realise at the time: gateway profiles aren't separate services. They're separate config directories, separate skill directories, separate .env files, and a --profile flag handed to the same gateway binary. Same process tree. Same supervisor. No hard boundary when one of them misbehaves.

I'd been treating them like independent services. They're not.

What the Docs Recommended (and Why I Went Further)

The Hermes docs recommend one container hosting all profiles, with s6-overlay (a process supervisor designed for containers) managing each profile as a first-class service. Mount the whole ~/.hermes directory to /opt/data, create profiles with hermes profile create, let s6 start and stop them.

I didn't think that would solve what I was seeing. With one container and s6 running every profile, sage and rex are still co-located processes sharing a network namespace and a data directory. If the problem was jobs bleeding across profiles, bundling them into one container wouldn't stop any of it. They'd just be supervised versions of the same shared setup.

So I ran separate containers for sage and rex, each mounting only its own profile directory. Real process isolation: separate network namespaces, separate PID 1, separate Telegram polling loops. At the process level they genuinely cannot interfere with each other.

The architecture was right, it just wasn't the cause of what I was seeing. That part's in the verification post.

The Mount Path Mistake

The mount paths. I spent far too long on this.

The profile needs to live at /opt/data/profiles/<name> inside the container, not at /opt/data. I kept mounting the rex profile to /opt/data, and Hermes would log Error: Profile 'rex' does not exist. Create it with: hermes profile create rex. Same files on the host, same mount command, but Hermes couldn't find the profile because it was looking in the wrong place.

The fix was obvious once I saw it:

# Wrong — mounts to /opt/data, Hermes expects /opt/data/profiles/rex
- /home/luke/.hermes/profiles/rex:/opt/data

# Right
- /home/luke/.hermes/profiles/rex:/opt/data/profiles/rex

The containers weren't the wrong architecture. The paths were wrong. A simple mistake that cost more time than it should have.

The Boot Problem

Containers don't start themselves. Bare-metal services auto-start through systemd. I needed the Docker containers to come up on boot too.

docker compose has no native systemd integration. The cleanest approach was a systemd user service that runs docker compose up -d for each profile's compose file. It feels slightly off, systemd managing containers instead of services, but it's straightforward and it works.

What I'd Tell Myself

When I was running three gateway profiles on bare metal, I was already past what profiles are designed for. They're great for development. Spinning up a new persona with different skills takes seconds. For anything that needs to actually stay separated, the isolation story falls apart.

Containers were the right architecture for that. The operational complexity is real (two compose files, a custom systemd service, separate log streams), but it's the right kind of complexity. Explicit, manageable, debuggable. When rex goes wrong, I look at the rex container, not a shared process tree.

I still run default bare-metal. Default is the daily driver; it has no scheduled cron that fires critical replies. Sage and rex are the workers. They get containers.

What I got wrong was assuming that building the right architecture meant the problem was solved. It didn't. When I went to verify the containers were actually doing their job, nothing worked. Sage was running two Hermes instances, cron jobs were vanishing into the wrong database, and the symptom that started all of this was still happening.

That's the debugging follow-up.

The Problem

There's a similar thread documented here in the Unraid Forums which I found helpful.

Turns out, Dockerman in Unraid 7.0+ (Unraid's Docker management plugin) uses a container label net.unraid.docker.managed=dockerman to determine which containers it actually manages. My containers were created way back on an older version of Unraid, so they didn't have this label. Without it, Dockerman basically said "not my problem" and refused to touch them.

The nuclear option would be to recreate every single container from scratch, but that's tedious and error-prone when you have dozens of containers with specific configurations. I know I could reuse an existing template as well.. but it still felt like a tedious task. So I did what I normally do and implemented an overly engineered solution to a problem that I could fixed pretty quickly doing it manually.

The Solution

The good news is you can add the missing label using Docker's CLI without manually reconfiguring everything.

I started by testing this on one container first - Jackett, which is one of my torrent index containers. I wanted to make sure the whole process worked before batch-processing everything.

First, I generated the docker run command using runlike (it inspects a running container and outputs the equivalent docker run command):

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
    assaflavie/runlike Jackett > /tmp/jackett_run.sh

(Note: you'll need root privileges to run docker commands in the Unraid terminal - otherwise you'll get "permission denied" errors.)

Then I reviewed what runlike generated:

cat /tmp/jackett_run.sh

This showed me the full docker run command with all the volumes, ports, environment variables - exact configuration for my Jackett container.

Next, I needed to add the missing label. I opened the file with nano:

nano /tmp/jackett_run.sh

And added --label net.unraid.docker.managed=dockerman and --detach=true right after docker run:

docker run --label net.unraid.docker.managed=dockerman --detach=true --name=Jackett ...

Then I stopped the old container, removed it, and recreated it with the new configuration:

docker stop Jackett
docker rm Jackett
bash /tmp/jackett_run.sh

Finally, I needed to make Unraid fully recognize the container. In the Docker tab, I clicked on Jackett to open its dropdown menu and selected "Force Update." This tells Unraid to add its other management labels.

After doing this, Jackett showed up properly in Unraid instead of as "3rd party." Success!

The Automated Script

Once I verified the manual process worked, I created a script to batch-process all my containers. The script loops through all running containers, generates the run command, injects the label, recreates the container, and queues a Force Update:

https://gist.github.com/ManningWorks/0f57db9c450d7b7dea741e585b31d23e

To use this, I added it to Unraid's User Scripts plugin (you can install it from Community Applications). This lets you run one-off scripts safely instead of running them directly in the Bash Shell.

If you're comfortable with scripts, this will process all your containers at once. Otherwise, the manual steps above work fine for one-off fixes.

Important Notes

By the way, if you run into different data recovery issues — like Git object corruption — the same instinct applies: move don't delete, have a rollback plan.

• Your data is safe - This process only removes and recreates the container definitions, not your actual data in appdata or volumes
• Test on one container first - Make sure the manual steps work for your setup before running the bash script
• The Force Update step matters - Don't skip it, as it adds the additional Unraid management labels
• Back up your flash drive - Before any major changes, it's always good practice to back up your Unraid configuration

Why This Happens

Unraid made this change to better track which containers it's managing versus containers you might have created manually or through other tools. It's actually a good change for container management, but it does mean old containers need this label added retroactively.

Hopefully this saves someone else the frustration of staring at dozens of "3rd party" containers after an upgrade! If this saved you an hour, the Gist is there to share.