Configuration You Regenerate
She still waits for you to type something
Everything she does begins with a person at a keyboard running
uv run python labs/wire_core.py. That was fine while the machine was your
laptop and you were sitting in front of it. It stops being fine the moment she moves to
a small computer on a shelf with no monitor, because now three ordinary events end her:
the machine reboots after a power cut, the process dies at four in the morning, or you
close the SSH session she was running inside. She does not come back until someone
notices and logs in.
Starting a program at boot, restarting it when it exits badly, and keeping its output readable are jobs the operating system already does for every other service on the box. On Linux that job belongs to systemd, and you ask for it by writing a unit file: a small text file declaring who runs the program, from which directory, with which command line, and what should happen when it exits.
The obvious move is to open an editor and type glados.service by hand. It
works once. Then you rename the project folder, or deploy under a second account, or
copy the file to the Raspberry Pi where your username is pi and your home
is somewhere else, and every absolute path in that file is quietly wrong. systemd will
not guess what you meant; it refuses to start and tells you so in a numeric code. Those
paths are not knowledge you have. They are knowledge the machine has, and the registry
from chapter 31 already holds the rest. So the rule for this chapter: the unit
file is an output of the project, generated from the running environment and the
component registry, never typed by a human.
From a template to a unit that fits this machine
# labs/service_unit.py
SERVICE_TEMPLATE = """[Unit]
Description=GLaDOS voice assistant
After=network.target
[Service]
Type=simple
User={user}
WorkingDirectory={work_dir}
ExecStart={python} {script}
[Install]
WantedBy=multi-user.target
"""
if __name__ == "__main__":
print(SERVICE_TEMPLATE.format(
user="kryo",
work_dir="/home/kryo/src/GladOS",
python="/home/kryo/src/GladOS/.venv/bin/python",
script="/home/kryo/src/GladOS/labs/wire_core.py",
))
$ uv run python labs/service_unit.py
[Unit]
Description=GLaDOS voice assistant
After=network.target
[Service]
Type=simple
User=kryo
WorkingDirectory=/home/kryo/src/GladOS
ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/wire_core.py
[Install]
WantedBy=multi-user.target
A unit file is an INI file: named sections, one Key=Value per line. Most
of it is structure that is identical on every machine in the world, and exactly four
values differ. Splitting those four out as {} placeholders means the
structure gets written once, correctly, and every later stage in this chapter only
improves where the four values come from. Right now they come from your fingers,
which is why this version is correct on precisely one computer.
import getpass
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[1]
ENTRY_SCRIPT = PROJECT_ROOT / "labs" / "wire_core.py"
def unit_values() -> dict[str, str]:
return {
"user": getpass.getuser(),
"work_dir": str(PROJECT_ROOT),
"python": sys.executable,
"script": str(ENTRY_SCRIPT),
}
if __name__ == "__main__":
for key, value in unit_values().items():
print(f"{key:9} {value}")
$ uv run python labs/service_unit.py
user kryo
work_dir /home/kryo/src/GladOS
python /home/kryo/src/GladOS/.venv/bin/python
script /home/kryo/src/GladOS/labs/wire_core.py
Four hand-typed strings became four questions with authoritative answers.
sys.executable is the absolute path of the interpreter running this line,
so under uv run it is the workspace venv and the generator cannot
disagree with the environment it was launched in.
Path(__file__).resolve().parents[1] walks up from
labs/service_unit.py to the project root, so a clone in a different
directory produces the right answer with no edit. getpass.getuser()
checks the usual environment variables and falls back to the system password database,
so it still answers on a Pi that booted with nobody logged in. Your four lines will
read differently from mine; that is the point.
from labs.system_config import SystemConfig, load_config
CONFIG_PATH = PROJECT_ROOT / "configs" / "system_config.json"
# What each subsystem needs the operating system to have running first.
UNIT_REQUIREMENTS = {
"llm": ["ollama.service"],
"stt": ["sound.target"],
"tts": ["sound.target"],
}
def ordering_lines(cfg: SystemConfig) -> list[str]:
units = ["network.target"]
for comp in cfg.components:
if not comp.enabled:
continue
for unit in UNIT_REQUIREMENTS.get(comp.name, []):
if unit not in units:
units.append(unit)
lines = [f"After={' '.join(units)}"]
wants = [u for u in units if u.endswith(".service")]
if wants:
lines.append(f"Wants={' '.join(wants)}")
return lines
if __name__ == "__main__":
for line in ordering_lines(load_config(CONFIG_PATH)):
print(line)
$ uv run python labs/service_unit.py
After=network.target sound.target ollama.service
Wants=ollama.service
$ uv run python labs/service_unit.py # after setting "enabled": false on the llm component
After=network.target sound.target
This is the part a hand-written unit file can never keep true. The registry records
which subsystems exist and which are switched on; the table above records what each
one needs from the operating system. Compose them and the deployment follows the
configuration by itself: switch the language model off for a listening-only build and
the dependency on Ollama leaves the unit, with nothing to remember.
After= and Wants= are different promises.
After= is ordering only: if that unit is starting too, wait for it.
Wants= actually pulls the other service up, so it belongs on
ollama.service and not on sound.target, which is a milestone
in the boot sequence and not something you can want.
# labs/service_unit.py -- full file
import difflib
import getpass
import socket
import sys
from pathlib import Path
from labs.system_config import SystemConfig, load_config
PROJECT_ROOT = Path(__file__).resolve().parents[1]
CONFIG_PATH = PROJECT_ROOT / "configs" / "system_config.json"
ENTRY_SCRIPT = PROJECT_ROOT / "labs" / "wire_core.py"
HEALTH_CHECK = PROJECT_ROOT / "glados" / "health_check.py"
GENERATED = Path("/tmp/glados.service")
INSTALLED = Path("/etc/systemd/system/glados.service")
UNIT_REQUIREMENTS = {
"llm": ["ollama.service"],
"stt": ["sound.target"],
"tts": ["sound.target"],
}
SERVICE_TEMPLATE = """# Generated by labs/service_unit.py on {host}. Do not edit by hand.
[Unit]
Description=GLaDOS voice assistant
{ordering}
[Service]
Type=simple
User={user}
WorkingDirectory={work_dir}
Environment=PYTHONUNBUFFERED=1
Environment=GLADOS_CONFIG={config_path}
ExecStartPre={python} {health_check}
ExecStart={python} {script}
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
"""
def ordering_lines(cfg: SystemConfig) -> list[str]:
units = ["network.target"]
for comp in cfg.components:
if not comp.enabled:
continue
for unit in UNIT_REQUIREMENTS.get(comp.name, []):
if unit not in units:
units.append(unit)
lines = [f"After={' '.join(units)}"]
wants = [u for u in units if u.endswith(".service")]
if wants:
lines.append(f"Wants={' '.join(wants)}")
return lines
def render_unit(cfg: SystemConfig) -> str:
if sys.prefix == sys.base_prefix:
raise RuntimeError("not in the project venv: run with `uv run python labs/service_unit.py`")
return SERVICE_TEMPLATE.format(
host=socket.gethostname(),
ordering="\n".join(ordering_lines(cfg)),
user=getpass.getuser(),
work_dir=PROJECT_ROOT,
config_path=CONFIG_PATH,
python=sys.executable,
health_check=HEALTH_CHECK,
script=ENTRY_SCRIPT,
)
def check_installed(unit: str) -> int:
if not INSTALLED.exists():
print(f"{INSTALLED}: not installed yet")
return 1
current = INSTALLED.read_text()
if current == unit:
print(f"{INSTALLED}: matches this machine")
return 0
print("\n".join(difflib.unified_diff(
current.splitlines(), unit.splitlines(),
fromfile=str(INSTALLED), tofile="generated", lineterm="")))
return 1
def main() -> int:
unit = render_unit(load_config(CONFIG_PATH))
if "--check" in sys.argv:
return check_installed(unit)
GENERATED.write_text(unit)
print(unit)
print(f"Wrote {GENERATED}")
print(f"Install: sudo cp {GENERATED} {INSTALLED}")
print(" sudo systemctl daemon-reload")
print(" sudo systemctl enable --now glados")
return 0
if __name__ == "__main__":
sys.exit(main())
$ uv run python labs/service_unit.py --check
--- /etc/systemd/system/glados.service
+++ generated
@@ -10,7 +10,7 @@
Environment=GLADOS_CONFIG=/home/kryo/src/GladOS/configs/system_config.json
ExecStartPre=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/glados/health_check.py
-ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/voice_loop.py
+ExecStart=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/labs/wire_core.py
Restart=on-failure
RestartSec=10
Three additions earn their place. ExecStartPre runs the startup probe from
chapter 33 first: if a model file is missing or Ollama is not answering, the probe
exits non-zero, systemd never reaches ExecStart, and the reason lands in
the log instead of a half-started assistant sitting there deaf. The header comment
makes an installed unit name its own origin machine. And --check compares
the installed file against what this project would generate right now, printing a diff
and returning 1 when they differ; the run above came from a box whose unit still
pointed at the old entry script. That turns "is the deployed configuration current?"
into a command a timer can run instead of a question you answer by squinting.
Why this works: two supervisors, one boundary
systemd does not execute a unit file. It parses it into declarations and acts on them at
defined moments. Type=simple says the process it spawns is the
service, so it tracks that PID directly, which is correct for a Python program that
never forks. ExecStart is a command, not a shell line: no PATH
search, no ~ expansion, no pipes or redirections. That constraint is why an
absolute interpreter path is mandatory, and why this chapter computes one instead of
trusting anybody to type it. Restart=on-failure with
RestartSec=10 asks for another attempt ten seconds after a non-zero exit.
WantedBy= is read only when you run systemctl enable, which
symlinks her into the boot sequence. Nothing in the file is read again until
daemon-reload, so an edited unit stays invisible until you say so.
Now the layering question, because two supervisors are watching her and they watch different things. The watchdog from chapter 41 lives inside the process and watches progress: a turn finishes, a timestamp moves, and a background thread notices when the timestamps stop moving. systemd lives outside the process and watches its boundary: did it exit, with what status, and should something start it again. Neither can do the other's job. A wedged interpreter, blocked forever on a serial read, is a perfectly healthy service as far as systemd can tell, because the PID is alive and nothing exited. A machine that lost power at 03:00 is beyond the reach of a thread that no longer exists.
They connect at one point: the exit status. When in-process recovery has been tried and
the heartbeat is still silent, the watchdog callback stops improvising and calls
os._exit(1), which leaves immediately with a status of your choosing even
from a thread that cannot raise an exception into a blocked main thread. systemd sees a
non-zero exit, waits ten seconds, runs the health probe, and starts a fresh interpreter
with newly loaded models and newly opened devices. The watchdog decides that she is
stuck. systemd decides what happens next.
Type=notify plus WatchdogSec= lets a service ping systemd
over a socket and be killed if the pings stop, the same idea one level out. It needs a
notification handshake in your code and a binding to speak that protocol, and in
exchange it duplicates a detector you already have, with less knowledge of the pipeline
than yours has. Keeping Type=simple and letting the in-process watchdog
choose the exit status keeps the decision where the information is. Ubuntu Zero goes
deeper into unit types, sandboxing keys and timers if you want the wider tour; nothing
here depends on it.
The generator worked so well on the laptop that copying its output to the Pi felt like
the same operation: scp /tmp/glados.service pi@glados-pi:, install, enable,
done. Then:
pi@glados-pi:~$ systemctl status glados --no-pager
× glados.service - GLaDOS voice assistant
Loaded: loaded (/etc/systemd/system/glados.service; enabled)
Active: activating (auto-restart) (Result: exit-code)
Process: 1188 ExecStartPre=/home/kryo/src/GladOS/.venv/bin/python /home/kryo/src/GladOS/glados/health_check.py (code=exited, status=203/EXEC)
pi@glados-pi:~$ journalctl -u glados -n 4 --no-pager
glados-pi systemd[1188]: glados.service: Failed to locate executable /home/kryo/src/GladOS/.venv/bin/python: No such file or directory
glados-pi systemd[1188]: glados.service: Failed at step EXEC spawning /home/kryo/src/GladOS/.venv/bin/python: No such file or directory
glados-pi systemd[1]: glados.service: Control process exited, code=exited, status=203/EXEC
glados-pi systemd[1]: glados.service: Scheduled restart job, restart counter is at 7.
Captured on the bench; your hostnames, PIDs and timestamps will read differently. Exit
code 203 is systemd's own, produced before any of your code runs: the exec failed. The
path in the message answers the rest, since the Pi has no user kryo and no
such directory. Two details repay a second read. The restart counter climbing to 7 is a
self-heal applied to something no restart can fix, and an identical log line repeating
on a fixed interval is the signature of a configuration error, where a real crash tends
to vary. And the episode came from treating the unit file as the portable artifact when
the portable artifact is the generator. The correct deploy is uv sync and
uv run python labs/service_unit.py on the Pi, whose
sys.executable, hostname and username are the ones that matter there.
Running --check after the copy would have printed the mismatched
User=, path and host line before systemd ever tried.
Checkpoint, and a machine that gets up on its own
- I can name the three facts the generator reads from its own environment, and say
which registry field decides whether
Wants=ollama.serviceappears. - I can state what systemd does with
Type=simple,ExecStart,Restart=on-failureandWantedBy=, and which of them is read only at enable time. - I can explain why a frozen process looks healthy to systemd, and name the one value the in-process watchdog uses to hand the problem outward.
- Shown
status=203/EXEC, I know the failure happened before Python started, and I can find the offending path in one journalctl line. - I can tell whether the installed unit still matches this project without opening either file.
- I know why
ExecStartcannot contain a pipe, a wildcard or$PATHlookup, and where setup logic goes instead.
Exercise 1 — a service with no sudo. Add
render_user_unit() producing a unit for systemctl --user:
no User= line, WantedBy=default.target. Install it to
~/.config/systemd/user/glados.service and start it without root.
Copy the template, delete the User= line, change the install target,
and drop user from the format() call. Then
systemctl --user daemon-reload and
systemctl --user enable --now glados, with logs under
journalctl --user -u glados -f. One catch will bite you on a headless
machine: a user service normally stops when your last session ends, so on the Pi
run sudo loginctl enable-linger $USER to keep her running after you
log out. User services are the right choice when she needs your audio session and
your home directory and nothing privileged.
Exercise 2 — a brace that is not yours. Add an
ExecStartPre line that plays a boot chime:
/bin/bash -c 'aplay ${HOME}/sounds/boot.wav'. Predict what
render_unit() does before you run it.
It crashes with KeyError: 'HOME'. str.format() has no
idea that ${HOME} was addressed to a shell; it sees
{HOME}, looks for a keyword argument by that name, and finds none.
Escape a literal brace by doubling it, so the template line reads
'aplay ${{HOME}}/sounds/boot.wav' and the rendered file contains a
single ${HOME} for bash to expand at start time. The general lesson
outlives this file: when you template a configuration language, two syntaxes now
share one string, and every literal brace belongs to whichever one runs first.
Exercise 3 — let systemd grade your file. Misspell a key as
Restrat=on-failure, regenerate, and run
systemd-analyze verify /tmp/glados.service.
You get a line like
/tmp/glados.service:12: Unknown key 'Restrat' in section 'Service', ignoring.
Read the last word carefully. systemd does not reject a unit for an unknown key;
it logs and moves on, so a typo means the service simply never restarts, with no
error at the moment you install it and no clue when it eventually dies for good.
systemd-analyze verify is how you see those complaints without
installing anything, which makes it a natural companion to --check:
one asks whether the file is sane, the other asks whether it is current.
Pull the power, wait, plug it back in, and she comes up on her own with Ollama warmed and the health probe green. The software half of the deployment is now describable in one command that any machine can run for itself. The half still living in your head is the physical one: which pin drives the servo, what feeds 5 V, and which two wires you must never swap. Next chapter turns that into data the same way, so a connection can be checked by a program instead of remembered.