Built with Claude Code. See how these projects were built for what that means in practice.

Going live means seven things have to be running at once: the broadcast software, the chat-command bot, a text-to-speech server, a stats-and-control proxy, the Discord mirror bot, a chat client and this website. Starting them by hand meant forgetting one roughly every other stream, usually the one you only notice when a viewer points it out.

So there is a launcher, a process supervisor, and a watchdog. This write-up is mostly about the ways all three fooled me.

The shape

One PowerShell script starts everything, in order, and is idempotent — running it again when everything is already up starts nothing. The four console services run inside a single terminal multiplexer pane-per-service, each with its own scrollback and its own log file, and any one can be restarted without touching the others. A flag file records "the stack is meant to be up", which is what arms the watchdog; a second flag file force-disables it. A scheduled task runs the watchdog at login and every five minutes.

The cost of grouping services into one window is real and worth stating: they become that window's children, so closing it kills all four.

Where the launcher was quietly lying to me

The "is it already running?" checks matched nothing at all. They filtered processes by command line — which works fine in PowerShell 7 and does not exist in PowerShell 5.1. The desktop shortcut runs 5.1. So every guard silently matched nothing, and services double-started. That is where two copies of the web server, two stats servers and two mirror bots came from. The fix is a different cmdlet that works on both. (The old Windows tool everyone reaches for as a fallback has been removed from Windows 11 entirely, so it is not an option.)

Two processes can bind the same port on Windows without an error. Python's HTTP server sets address reuse, so the second copy binds happily and requests go to whichever won the race. Duplicate servers are invisible if you check the port instead of the process list.

A virtual-environment Python can be a shim that re-executes the base interpreter — and it is the child process that holds the port. Any check for "is this running from the right environment?" has to accept the process or its parent, or it reports the wrong interpreter every time.

Never build a kill list from a command-line substring. A filter looking for a script name matched the automation session's own shell — the string was sitting in its arguments — and killed it. An earlier version of the same mistake had taken the website down. Filter on the executable name as well, always exclude your own process ID, and print the matches and read them before killing anything.

A check with no image filter reported success and did nothing. The mirror bot's guard matched any process whose command line merely mentioned the bot's filename — a shell, an editor, the watchdog's own invocation — so it announced "already running" and never started the bot. Same family of bug as above, found twice in one session, the second time sitting in already-committed code.

The process supervisor silently refuses some config forms. Shell-style entries never start on Windows: it creates the pane, creates the log file, and launches nothing, with no error anywhere. Only a direct executable works. The config had used the broken form for all four services, which meant the grouped-window mode had never once worked in its entire existence.

Never version-pin an interpreter path in a config file. One was hardcoded and would have died silently at the next Python upgrade — and a failed process in the supervisor shows no error, it just never appears.

The watchdog that repaired healthy audio 422 times

My favourite bug in this whole stack.

The watchdog gained a check for audio levels drifting. It worked by embedding a small Python probe as a here-string inside the PowerShell script. It ran every five minutes for two days, "detected" a problem every single time, "repaired" it, and pinged me about it — 422 times.

The chain:

1. The scheduled task runs PowerShell 5.1, not 7.
2. 5.1 handles that here-string form differently and mangled the embedded Python's string formatting into a syntax error.
3. The error output was redirected to null, so the syntax error vanished.
4. The probe therefore returned an empty string.
5. Casting an empty string to an integer yields zero.
6. Zero is less than the threshold, so: "volume is at zero, repairing".

Four separate decisions, each individually reasonable, composing into a system that was confidently wrong on a five-minute cycle. Three lessons came out of it:

• Never embed one language inside another language's string literal. The probe is now a real file that no shell can reinterpret on its way past.
• Silencing errors on a check turns a broken check into a false positive. An unreadable probe must never coerce into a valid-looking value. It now logs "unreadable" and changes nothing.
• Test automation with the exact shell that will actually run it. The block passed under PowerShell 7 and failed under 5.1.

The tell had been in the log the whole time: a genuine alert names a number, and every one of these was blank. I was reading the log for a reason and not noticing the shape.

Also decided as a result: self-healing, idempotent conditions are now log-only. Alerting me that something was already fine is not monitoring, it is noise. Alerts are reserved for break and almost-break.

The failure mode that actually hurts is silent success

The other lesson worth the whole page.

Sounds played through the control panel stopped reaching Discord. Everything returned normally, nothing logged an error, and the sound still played on the broadcast — only Discord went quiet.

The cause: the mirroring copy preserved the source file's modification time. So a clip added hours earlier produced a temp copy that was born "old", and the cleanup sweep — which deletes anything older than fifteen minutes — deleted it microseconds after it was written. Port checks, process checks and error-log greps would all have shown green.

Monitor invariants, not crashes. The watchdog now asserts facts the stack depends on, every five minutes: that every sound file referenced actually exists on disk, that volumes are in range, that config files still parse as JSON, that temp copies are neither piling up nor going stale (which would mean the sweep died), and that the disk has room. Each check was verified to actually fire by deliberately breaking the thing it watches — a check that cannot fail is not worth running, and until you have watched it go red you do not know which of those you have.

A related one: a service can be up while the thing it exists to do is dead. The voice changer opens the microphone, the voice detection and its monitor device in a single audio stream, so a monitor device it cannot open kills the voice while every port, every process and its own control API report perfect health. Detection there is based on recency rather than counts — an error that has accumulated thousands of times over a log's life says nothing; the same error in the last five minutes says everything. Deliberately no auto-fix: the repair is choosing a different device in a GUI, which cannot be scripted, so it alerts and stops.

Smaller things that cost real time

Piping Python through a tee block-buffers its output. Through a pipe, the interpreter switches to block buffering, so plain print output never reaches the screen — while logging output still appears, because that flushes per record. One service's window was readable only because it happened to use a logging framework; any bare print in its startup path was invisible until exit.

A hidden-window flag does not stop a scheduled task flashing a console. The window is allocated first and hidden a moment later, so it flashed every five minutes. The fix is a GUI-subsystem shim that never allocates a console at all.

Single-threaded HTTP servers block everything on one slow handler. A 30-second speech call froze every other request, including the on-screen stats panel. A one-line change to a threading server fixed it — verified with an 11-second request while three other routes answered in under 300 milliseconds.

Listing processes costs about 600 milliseconds on this machine. Called once per one-second poll, it made the control panel saturate itself and render empty panels. Cached with a short TTL, the same endpoint returns in about 20 milliseconds. Same shape as the fix for re-dialling a dead dependency on every poll: never do expensive work on a fast timer without caching it.

Services started as a background shell of some other tool die when that tool exits. Three services went down between sessions for exactly this reason. Long-lived services must be started detached — by the launcher, the scheduled task or the supervisor — never as a child of something transient.

Two automation sessions silently undid each other. A scheduled task was confirmed disabled, then six minutes later was enabled and firing, grabbing a port half a second before the launcher and knocking a service out of the grouped window. It was not diagnosable from inside the session; the evidence looked like a phantom bug. Only one session should own this directory at a time.

Status

Working, and stable enough that I mostly forget it exists — which is the goal for infrastructure. Everything above is in the repository's own notes as a numbered list of lessons, because writing them down is the only reason I stopped making some of them twice.