GLaDOS Vol 6 · Ready for the House
ch 54 / 99
Chapter 54

Configuration You Regenerate

She still waits for you to type something

Everything she does begins with a person at a keyboard running uv run python labs/wire_core.py. That was fine while the machine was your laptop and you were sitting in front of it. It stops being fine the moment she moves to a small computer on a shelf with no monitor, because now three ordinary events end her: the machine reboots after a power cut, the process dies at four in the morning, or you close the SSH session she was running inside. She does not come back until someone notices and logs in.

Starting a program at boot, restarting it when it exits badly, and keeping its output readable are jobs the operating system already does for every other service on the box. On Linux that job belongs to systemd, and you ask for it by writing a unit file: a small text file declaring who runs the program, from which directory, with which command line, and what should happen when it exits.

The obvious move is to open an editor and type glados.service by hand. It works once. Then you rename the project folder, or deploy under a second account, or copy the file to the Raspberry Pi where your username is pi and your home is somewhere else, and every absolute path in that file is quietly wrong. systemd will not guess what you meant; it refuses to start and tells you so in a numeric code. Those paths are not knowledge you have. They are knowledge the machine has, and the registry from chapter 31 already holds the rest. So the rule for this chapter: the unit file is an output of the project, generated from the running environment and the component registry, never typed by a human.

From a template to a unit that fits this machine

▣ Build · stage 1 — the fixed part and the four values
# labs/service_unit.py
SERVICE_TEMPLATE = """[Unit]
Description=GLaDOS voice assistant
After=network.target

[Service]
Type=simple
User={user}
WorkingDirectory={work_dir}
ExecStart={python} {script}

[Install]
WantedBy=multi-user.target
"""

if __name__ == "__main__":
    print(SERVICE_TEMPLATE.format(
        user="kryo",
        work_dir="/home/kryo/src/GladOS",
        python="/home/kryo/src/GladOS/.venv/bin/python",
        script="/home/kryo/src/GladOS/labs/wire_core.py",
    ))
$ uv run python labs/service_unit.py
[Unit]
Description=GLaDOS voice assistant
After=network.target

[Service]
Type=simple
User=kryo
WorkingDirectory=/home/kryo/src/GladOS
ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/wire_core.py

[Install]
WantedBy=multi-user.target

A unit file is an INI file: named sections, one Key=Value per line. Most of it is structure that is identical on every machine in the world, and exactly four values differ. Splitting those four out as {} placeholders means the structure gets written once, correctly, and every later stage in this chapter only improves where the four values come from. Right now they come from your fingers, which is why this version is correct on precisely one computer.

▣ Build · stage 2 — ask the interpreter where it is
import getpass
import sys
from pathlib import Path

PROJECT_ROOT = Path(__file__).resolve().parents[1]
ENTRY_SCRIPT = PROJECT_ROOT / "labs" / "wire_core.py"


def unit_values() -> dict[str, str]:
    return {
        "user": getpass.getuser(),
        "work_dir": str(PROJECT_ROOT),
        "python": sys.executable,
        "script": str(ENTRY_SCRIPT),
    }


if __name__ == "__main__":
    for key, value in unit_values().items():
        print(f"{key:9} {value}")
$ uv run python labs/service_unit.py
user      kryo
work_dir  /home/kryo/src/GladOS
python    /home/kryo/src/GladOS/.venv/bin/python
script    /home/kryo/src/GladOS/labs/wire_core.py

Four hand-typed strings became four questions with authoritative answers. sys.executable is the absolute path of the interpreter running this line, so under uv run it is the workspace venv and the generator cannot disagree with the environment it was launched in. Path(__file__).resolve().parents[1] walks up from labs/service_unit.py to the project root, so a clone in a different directory produces the right answer with no edit. getpass.getuser() checks the usual environment variables and falls back to the system password database, so it still answers on a Pi that booted with nobody logged in. Your four lines will read differently from mine; that is the point.

▣ Build · stage 3 — what she is made of decides what must start first
from labs.system_config import SystemConfig, load_config

CONFIG_PATH = PROJECT_ROOT / "configs" / "system_config.json"

# What each subsystem needs the operating system to have running first.
UNIT_REQUIREMENTS = {
    "llm": ["ollama.service"],
    "stt": ["sound.target"],
    "tts": ["sound.target"],
}


def ordering_lines(cfg: SystemConfig) -> list[str]:
    units = ["network.target"]
    for comp in cfg.components:
        if not comp.enabled:
            continue
        for unit in UNIT_REQUIREMENTS.get(comp.name, []):
            if unit not in units:
                units.append(unit)
    lines = [f"After={' '.join(units)}"]
    wants = [u for u in units if u.endswith(".service")]
    if wants:
        lines.append(f"Wants={' '.join(wants)}")
    return lines


if __name__ == "__main__":
    for line in ordering_lines(load_config(CONFIG_PATH)):
        print(line)
$ uv run python labs/service_unit.py
After=network.target sound.target ollama.service
Wants=ollama.service
$ uv run python labs/service_unit.py   # after setting "enabled": false on the llm component
After=network.target sound.target

This is the part a hand-written unit file can never keep true. The registry records which subsystems exist and which are switched on; the table above records what each one needs from the operating system. Compose them and the deployment follows the configuration by itself: switch the language model off for a listening-only build and the dependency on Ollama leaves the unit, with nothing to remember. After= and Wants= are different promises. After= is ordering only: if that unit is starting too, wait for it. Wants= actually pulls the other service up, so it belongs on ollama.service and not on sound.target, which is a milestone in the boot sequence and not something you can want.

▣ Build · stage 4 — the whole unit, written and checked
# labs/service_unit.py -- full file
import difflib
import getpass
import socket
import sys
from pathlib import Path

from labs.system_config import SystemConfig, load_config

PROJECT_ROOT = Path(__file__).resolve().parents[1]
CONFIG_PATH = PROJECT_ROOT / "configs" / "system_config.json"
ENTRY_SCRIPT = PROJECT_ROOT / "labs" / "wire_core.py"
HEALTH_CHECK = PROJECT_ROOT / "glados" / "health_check.py"
GENERATED = Path("/tmp/glados.service")
INSTALLED = Path("/etc/systemd/system/glados.service")

UNIT_REQUIREMENTS = {
    "llm": ["ollama.service"],
    "stt": ["sound.target"],
    "tts": ["sound.target"],
}

SERVICE_TEMPLATE = """# Generated by labs/service_unit.py on {host}. Do not edit by hand.
[Unit]
Description=GLaDOS voice assistant
{ordering}

[Service]
Type=simple
User={user}
WorkingDirectory={work_dir}
Environment=PYTHONUNBUFFERED=1
Environment=GLADOS_CONFIG={config_path}
ExecStartPre={python} {health_check}
ExecStart={python} {script}
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
"""


def ordering_lines(cfg: SystemConfig) -> list[str]:
    units = ["network.target"]
    for comp in cfg.components:
        if not comp.enabled:
            continue
        for unit in UNIT_REQUIREMENTS.get(comp.name, []):
            if unit not in units:
                units.append(unit)
    lines = [f"After={' '.join(units)}"]
    wants = [u for u in units if u.endswith(".service")]
    if wants:
        lines.append(f"Wants={' '.join(wants)}")
    return lines


def render_unit(cfg: SystemConfig) -> str:
    if sys.prefix == sys.base_prefix:
        raise RuntimeError("not in the project venv: run with `uv run python labs/service_unit.py`")
    return SERVICE_TEMPLATE.format(
        host=socket.gethostname(),
        ordering="\n".join(ordering_lines(cfg)),
        user=getpass.getuser(),
        work_dir=PROJECT_ROOT,
        config_path=CONFIG_PATH,
        python=sys.executable,
        health_check=HEALTH_CHECK,
        script=ENTRY_SCRIPT,
    )


def check_installed(unit: str) -> int:
    if not INSTALLED.exists():
        print(f"{INSTALLED}: not installed yet")
        return 1
    current = INSTALLED.read_text()
    if current == unit:
        print(f"{INSTALLED}: matches this machine")
        return 0
    print("\n".join(difflib.unified_diff(
        current.splitlines(), unit.splitlines(),
        fromfile=str(INSTALLED), tofile="generated", lineterm="")))
    return 1


def main() -> int:
    unit = render_unit(load_config(CONFIG_PATH))
    if "--check" in sys.argv:
        return check_installed(unit)
    GENERATED.write_text(unit)
    print(unit)
    print(f"Wrote {GENERATED}")
    print(f"Install: sudo cp {GENERATED} {INSTALLED}")
    print("         sudo systemctl daemon-reload")
    print("         sudo systemctl enable --now glados")
    return 0


if __name__ == "__main__":
    sys.exit(main())
$ uv run python labs/service_unit.py --check
--- /etc/systemd/system/glados.service
+++ generated
@@ -10,7 +10,7 @@
 Environment=GLADOS_CONFIG=/home/kryo/src/GladOS/configs/system_config.json
 ExecStartPre=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/glados/health_check.py
-ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/voice_loop.py
+ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/wire_core.py
 Restart=on-failure
 RestartSec=10

Three additions earn their place. ExecStartPre runs the startup probe from chapter 33 first: if a model file is missing or Ollama is not answering, the probe exits non-zero, systemd never reaches ExecStart, and the reason lands in the log instead of a half-started assistant sitting there deaf. The header comment makes an installed unit name its own origin machine. And --check compares the installed file against what this project would generate right now, printing a diff and returning 1 when they differ; the run above came from a box whose unit still pointed at the old entry script. That turns "is the deployed configuration current?" into a command a timer can run instead of a question you answer by squinting.

Why this works: two supervisors, one boundary

systemd does not execute a unit file. It parses it into declarations and acts on them at defined moments. Type=simple says the process it spawns is the service, so it tracks that PID directly, which is correct for a Python program that never forks. ExecStart is a command, not a shell line: no PATH search, no ~ expansion, no pipes or redirections. That constraint is why an absolute interpreter path is mandatory, and why this chapter computes one instead of trusting anybody to type it. Restart=on-failure with RestartSec=10 asks for another attempt ten seconds after a non-zero exit. WantedBy= is read only when you run systemctl enable, which symlinks her into the boot sequence. Nothing in the file is read again until daemon-reload, so an edited unit stays invisible until you say so.

Now the layering question, because two supervisors are watching her and they watch different things. The watchdog from chapter 41 lives inside the process and watches progress: a turn finishes, a timestamp moves, and a background thread notices when the timestamps stop moving. systemd lives outside the process and watches its boundary: did it exit, with what status, and should something start it again. Neither can do the other's job. A wedged interpreter, blocked forever on a serial read, is a perfectly healthy service as far as systemd can tell, because the PID is alive and nothing exited. A machine that lost power at 03:00 is beyond the reach of a thread that no longer exists.

They connect at one point: the exit status. When in-process recovery has been tried and the heartbeat is still silent, the watchdog callback stops improvising and calls os._exit(1), which leaves immediately with a status of your choosing even from a thread that cannot raise an exception into a blocked main thread. systemd sees a non-zero exit, waits ten seconds, runs the health probe, and starts a fresh interpreter with newly loaded models and newly opened devices. The watchdog decides that she is stuck. systemd decides what happens next.

◆ Note — systemd has its own watchdog, and we are not using it

Type=notify plus WatchdogSec= lets a service ping systemd over a socket and be killed if the pings stop, the same idea one level out. It needs a notification handshake in your code and a binding to speak that protocol, and in exchange it duplicates a detector you already have, with less knowledge of the pipeline than yours has. Keeping Type=simple and letting the in-process watchdog choose the exit status keeps the decision where the information is. Ubuntu Zero goes deeper into unit types, sandboxing keys and timers if you want the wider tour; nothing here depends on it.

⚠ Worked failure — status=203/EXEC on the Pi

The generator worked so well on the laptop that copying its output to the Pi felt like the same operation: scp /tmp/glados.service pi@glados-pi:, install, enable, done. Then:

pi@glados-pi:~$ systemctl status glados --no-pager
× glados.service - GLaDOS voice assistant
     Loaded: loaded (/etc/systemd/system/glados.service; enabled)
     Active: activating (auto-restart) (Result: exit-code)
    Process: 1188 ExecStartPre=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/glados/health_check.py (code=exited, status=203/EXEC)
pi@glados-pi:~$ journalctl -u glados -n 4 --no-pager
glados-pi systemd[1188]: glados.service: Failed to locate executable /home/kryo/src/GladOS/.venv/bin/python: No such file or directory
glados-pi systemd[1188]: glados.service: Failed at step EXEC spawning /home/kryo/src/GladOS/.venv/bin/python: No such file or directory
glados-pi systemd[1]: glados.service: Control process exited, code=exited, status=203/EXEC
glados-pi systemd[1]: glados.service: Scheduled restart job, restart counter is at 7.

Captured on the bench; your hostnames, PIDs and timestamps will read differently. Exit code 203 is systemd's own, produced before any of your code runs: the exec failed. The path in the message answers the rest, since the Pi has no user kryo and no such directory. Two details repay a second read. The restart counter climbing to 7 is a self-heal applied to something no restart can fix, and an identical log line repeating on a fixed interval is the signature of a configuration error, where a real crash tends to vary. And the episode came from treating the unit file as the portable artifact when the portable artifact is the generator. The correct deploy is uv sync and uv run python labs/service_unit.py on the Pi, whose sys.executable, hostname and username are the ones that matter there. Running --check after the copy would have printed the mismatched User=, path and host line before systemd ever tried.

Checkpoint, and a machine that gets up on its own

✓ Checkpoint — what you can now do
  • I can name the three facts the generator reads from its own environment, and say which registry field decides whether Wants=ollama.service appears.
  • I can state what systemd does with Type=simple, ExecStart, Restart=on-failure and WantedBy=, and which of them is read only at enable time.
  • I can explain why a frozen process looks healthy to systemd, and name the one value the in-process watchdog uses to hand the problem outward.
  • Shown status=203/EXEC, I know the failure happened before Python started, and I can find the offending path in one journalctl line.
  • I can tell whether the installed unit still matches this project without opening either file.
  • I know why ExecStart cannot contain a pipe, a wildcard or $PATH lookup, and where setup logic goes instead.
⚡ Exercises — try first, then reveal
Exercise 1 — a service with no sudo. Add render_user_unit() producing a unit for systemctl --user: no User= line, WantedBy=default.target. Install it to ~/.config/systemd/user/glados.service and start it without root.

Copy the template, delete the User= line, change the install target, and drop user from the format() call. Then systemctl --user daemon-reload and systemctl --user enable --now glados, with logs under journalctl --user -u glados -f. One catch will bite you on a headless machine: a user service normally stops when your last session ends, so on the Pi run sudo loginctl enable-linger $USER to keep her running after you log out. User services are the right choice when she needs your audio session and your home directory and nothing privileged.

Exercise 2 — a brace that is not yours. Add an ExecStartPre line that plays a boot chime: /bin/bash -c 'aplay ${HOME}/sounds/boot.wav'. Predict what render_unit() does before you run it.

It crashes with KeyError: 'HOME'. str.format() has no idea that ${HOME} was addressed to a shell; it sees {HOME}, looks for a keyword argument by that name, and finds none. Escape a literal brace by doubling it, so the template line reads 'aplay ${{HOME}}/sounds/boot.wav' and the rendered file contains a single ${HOME} for bash to expand at start time. The general lesson outlives this file: when you template a configuration language, two syntaxes now share one string, and every literal brace belongs to whichever one runs first.

Exercise 3 — let systemd grade your file. Misspell a key as Restrat=on-failure, regenerate, and run systemd-analyze verify /tmp/glados.service.

You get a line like /tmp/glados.service:12: Unknown key 'Restrat' in section 'Service', ignoring. Read the last word carefully. systemd does not reject a unit for an unknown key; it logs and moves on, so a typo means the service simply never restarts, with no error at the moment you install it and no clue when it eventually dies for good. systemd-analyze verify is how you see those complaints without installing anything, which makes it a natural companion to --check: one asks whether the file is sane, the other asks whether it is current.

Pull the power, wait, plug it back in, and she comes up on her own with Ollama warmed and the health probe green. The software half of the deployment is now describable in one command that any machine can run for itself. The half still living in your head is the physical one: which pin drives the servo, what feeds 5 V, and which two wires you must never swap. Next chapter turns that into data the same way, so a connection can be checked by a program instead of remembered.