Skip to main content
luke@terminal:/$ ls
homeblogprojectsaboutnow
luke@terminal:/blog$ cat hermes-profiles-to-docker-part-two.md

My Docker Containers Were Working. The Bug Was in a Python Loop.

› 2026-07-20↻ 2026-08-07⏱ 8 min read

This is a follow-up to my post on running Hermes profiles in Docker containers. In that post I described mounting each profile to its own container, setting up systemd to start them on boot, and declaring victory.

I was wrong.

Not wrong about the architecture. Wrong about whether it was actually working.

TL;DR: I declared the Docker setup working. Then cron jobs vanished into the wrong database, sage was running two Hermes instances, and the actual bug turned out to be a Python loop that had nothing to do with Docker.

The Problem I Didn't Know I Had (Again)

I ran docker exec hermes-sage ps aux to verify sage was healthy. It showed one process. Good. I fired a test cron on sage. It said it created successfully. I fired a test cron on rex. It said it created successfully. I waited.

Nothing arrived.

I assumed the Telegram bots weren't set up correctly. I went down a whole path of checking bot tokens, allowed user lists, TELEGRAM_HOME_CHANNEL settings. Then I checked the logs and found something stranger.

Sage's cron engine wasn't running the jobs. Neither was rex's. They were being created (the CLI returned success), but they were going into the host's state database, not the container's.

And underneath that, I discovered something worse: sage wasn't running one Hermes instance. It was running two.

Why 'ps aux' Lied to Me

The Docker entrypoint for the Hermes image is /init, which is s6-overlay. s6-overlay scans /run/service/ and starts everything it finds there. My compose file passed --profile sage to the gateway command, which should have meant "run sage profile only."

What actually happened: the compose command: was ignored. The ENTRYPOINT (/init) runs as PID 1, starts s6, and s6 starts all the services in /run/service/, gateway-default and gateway-sage both. Two Hermes instances, same process tree.

The ps aux I'd run to "verify" sage was healthy showed one gateway at the top of the output, so I stopped reading. The output ran longer than one screen. When I finally ran docker exec hermes-sage ps aux | grep gateway, two gateway processes showed up: the one I expected, and a second one nested under s6 that I'd scrolled straight past.

The compose command: instruction doesn't override an ENTRYPOINT. It gets handed to the entrypoint as arguments instead. That's in Docker's docs under how ENTRYPOINT and CMD interact. I'd never internalised it.

The Cron Database Shell Game

Once I grasped that sage was running two instances, the cron mystery made more sense. The CLI command hermes cron create was running inside the container, but it reads its config from environment variables, HOME especially, since that's where Hermes looks for its config directory.

I checked what HOME actually was inside the container:

$ docker exec hermes-sage env | grep HOME
HOME=/home/luke

That's the hermes user's passwd entry, not /opt/data. So when hermes cron create ran without --profile sage, it looked for config at /home/luke/.hermes, which doesn't exist in the container, and fell back to writing jobs to /opt/data/cron/jobs.json instead of /opt/data/profiles/sage/cron/jobs.json.

The sage gateway was reading from the right place. The cron CLI was writing to the wrong place. Jobs created successfully. Jobs never fired.

The fix was passing --profile sage to every cron command inside the container. But I also had to fix HOME, which led to the next problem.

The su -m Trap

I wanted the hermes process to run with HOME=/opt/data so it read from the right config. The entrypoint script ran as root, then used su hermes to drop privileges. Easy enough.

su -m is meant to preserve the parent environment, so I set HOME=/opt/data before calling su and expected it to carry through. It didn't. The shell su spawned was resetting HOME back to the hermes user's passwd entry (/home/luke) on the way up, undoing what su -m had preserved. By the time hermes actually ran, HOME had been clobbered again.

The reliable fix was env -i, which wipes the environment entirely before setting only what I want. Nothing left for the shell to reset:

exec env -i HOME=/opt/data HERMES_HOME=/opt/data HERMES_PROFILE=sage \
  su -m -s /bin/sh hermes -c "exec hermes gateway run --profile sage"

env -i clears everything, the explicit vars set what I need, and su -m preserves that clean state. hermes starts with exactly the HOME I specify.

Custom Entrypoint: Replacing PID 1

To stop gateway-default from starting, I needed to keep s6 from scanning it. The cleanest approach was replacing PID 1 entirely with a custom script that:

  1. Runs stage2-hook.sh for Hermes bootstrap (UID remapping, chown)
  2. Waits for s6 to register its services
  3. Kills gateway-default via s6-svc -d
  4. Starts only the sage gateway, using that same env -i exec line from above

Then the compose file binds the script as the entrypoint:

entrypoint: ["/entrypoint.sh"]
volumes:
  - /home/luke/.hermes/docker/entrypoints/sage-only.sh:/entrypoint.sh:ro

Now sage starts exactly one Hermes instance. No gateway-default. No confusion.

The Telegram 'Chat Not Found' Problem

Once I had real isolation working, the crons fired. The sage cron engine ran its job. The rex cron engine ran its job. Both logged completed successfully.

No Telegram messages arrived.

The error: Telegram send failed: Chat not found.

Sage has its own Telegram bot. Rex has its own Telegram bot. Separate bots, separate tokens. A Telegram bot can only send messages to people who have messaged that specific bot first. I'd only ever messaged the default Hermes bot, which knew about my chat. Sage's bot had never seen me.

The second issue was TELEGRAM_HOME_CHANNEL=RealLukeManning in the .env. That's a username, not a chat ID. The Telegram API needs a numeric chat ID to send messages. Sage was using the username and failing silently.

The fix: message each bot directly first, or use numeric chat IDs in the delivery target (telegram:491962736, not telegram:RealLukeManning).

The env_file Trap

I thought the Telegram issue was just setup. Then I noticed sage's logs showed its Telegram module attempting to connect, but the bot token wasn't in the container's config.

The .env with the bot token was on the host at ~/.hermes/profiles/sage/.env. The container mounts the profile directory to /opt/data/profiles/sage. The hermes process runs from /opt/data. No .env at /opt/data.

The compose file wasn't passing it through:

# Missing
env_file:
  - /home/luke/.hermes/profiles/sage/.env

Without the token, the Telegram connection failed gracefully, logged a warning, and didn't die. Messages never arrived.

The Final Test (Or So I Thought)

With all fixes in place, I fired simultaneous test crons on both containers. Both ran. Both Telegram bots delivered. Five seconds apart, completely independent.

Sage: one instance, sage profile, sage bot, sage cron. Rex: one instance, rex profile, rex bot, rex cron. Each container starting independently via systemd, each with its .env passed through, cron jobs created with --profile and numeric chat IDs.

I thought that was it. The containers were finally doing what I'd built them to do.

Then last Tuesday I got a message from Sage at 8am.

The daily system health check I'd set up months ago, before Docker and before any of this, was delivering through Sage's Telegram bot instead of the default gateway.

I swore up and down I'd fixed this. I had not fixed this.

What the Health Check Was Actually Doing

The health check runs every morning at 8am. Disk, swap, fail2ban, SSH failures. Then it delivers the result to Telegram. Crucially, it runs on the bare-metal default gateway, the one I never put in a container. Not sage, not rex.

The job itself was fine. The delivery was the problem. I opened the health-check cron job to see what it actually called, and found it was running a skill called hermes-telegram-send, one I didn't write, generated by an LLM using a template. That skill had one job: send a Telegram message from a cron context.

The skill worked by loading environment variables from profile .env files, in this order:

for profile in ["rex", "sage", "default"]:
    env_file = Path(f"/home/luke/.hermes/profiles/{profile}/.env")
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            # load the token

rex/.env exists, loads token. sage/.env exists, overwrites the token. default/.env doesn't exist, skipped.

Sage's Telegram bot token was the last one loaded. So when the health check cron ran, it used Sage's bot to send the message.

The message came from Sage. Because the skill hardcoded ["rex", "sage", "default"] and default didn't exist.

The Docker Containers Were Working Fine

This is the part that felt stupid to realise.

Sage was never misconfigured. She wasn't "replying when she shouldn't." She was doing exactly what her container was set up to do. The Docker isolation I'd spent a whole post documenting and debugging was completely correct. Separate containers, separate bots, separate processes. All working.

The bug wasn't in Docker. It wasn't in the architecture. It was in a Python loop in a skill that was generated without thinking about what happens when one of the hardcoded profile names doesn't exist.

The loop was supposed to try multiple profiles in case one didn't have a token. It would load rex, then sage, then default. The last one loaded would be the one used.

But there's no default profile directory. There's a bare-metal Hermes gateway running directly on the host, with its own .env at ~/.hermes/.env. That file has the actual Telegram token for the main bot.

The skill didn't know about ~/.hermes/.env. It only knew about the profile subdirectories.

The Real Fix: Baseline First, Never Overwrite

The skill now loads ~/.hermes/.env first as a baseline (the bare-metal gateway's token), then profile .env files only fill in empty slots. They never overwrite what's already set.

# ~/.hermes/.env loads FIRST as baseline
main_env = Path(f"{hermes_home}/.env")
if main_env.exists():
    for line in main_env.read_text().splitlines():
        os.environ[k] = v  # no guard, this is the baseline

# Profile files only fill empty slots, never overwrite
profile_order = ([current_profile] if current_profile else []) + ["rex", "sage"]
for profile in profile_order:
    env_file = Path(f"{hermes_home}/profiles/{profile}/.env")
    if env_file.exists():
        for line in env_file.read_text().splitlines():
            k, v = line.split("=", 1)
            if k not in os.environ:  # only fill empty slots
                os.environ[k] = v

The old code had two problems. First, when HERMES_PROFILE is empty (bare-metal default), the current profile list is empty and ["rex", "sage"] loads in that order, so sage overwrites rex. Second, the profile loop had no guard. os.environ[k] = v always overwrites, so even if the main .env loaded last, it was already too late. The fix: main .env first (no guard), profile files second (with if k not in os.environ guard).

The health check at 8am tomorrow should come from the right bot. For real this time.

What I Actually Learned

The implementation had three layers of subtle breakage. The compose command: doesn't override an ENTRYPOINT; if the image has one, the command gets passed to it as arguments. HOME matters more than I expected when su-ing to another user; env -i is the reliable way to lock it. And separate containers don't mean isolated if the entrypoint starts everything inside them regardless.

I fixed all of that. It was real work. And then the bug that actually mattered turned out to be none of it.

Three things I keep having to re-learn:

Infrastructure work doesn't protect you from code bugs. I spent weeks on Docker containers, systemd services, separate Telegram bots. All correct, all irrelevant to the actual symptom. A hardcoded list of profile names in a Python loop. The containers couldn't catch that, because containers don't catch Python logic errors.

LLM-generated skills inherit LLM failure modes. The skill came from a template with ["rex", "sage", "default"] baked in. Nobody stopped to ask what happens when default doesn't exist. That kind of assumption sits in code for months before anyone notices.

A fix I didn't verify was never really a fix. I thought I'd solved the Sage problem once isolation was working. The real fix had two subtle bugs hiding in it: an empty HERMES_PROFILE that silently disabled the "current profile first" logic, and a profile loop that overwrote tokens without checking if one was already set. I should have verified instead of trusting it.

The Docker setup is better now than before I started. The isolation is real. But the real fix required iterating on a Python loop that had nothing to do with Docker.

luke@terminal:/blog$ ls previous_post.sh
← Why I Ditched Hermes Profiles for Docker Containers