The Startup Health Check
Assembled and working are two different claims
Last chapter left you with a core that holds a listener, a brain, and a voice, and runs a
turn by calling them in order. Every attribute is filled in. Nothing is None.
The object prints beautifully. And none of that is evidence that a conversation will
survive contact with the hardware, because a callable can be present and still be wrong:
a model file moved, a server is down, a swapped engine now returns bytes where the loop
expects text.
Left alone, that discovery arrives at the worst possible time. You speak. The recorder
fills five seconds of buffer. Whisper spends two more turning it into text. The language
model burns another three composing a reply. Then, seven seconds into what felt like a
conversation, self.tts(reply) raises TypeError: 'NoneType' object is
not callable, and the traceback points at the voice loop, which is innocent. The
real fault was a misspelled key in a config file, decided before you ever opened your
mouth.
So this chapter builds a gate in front of the loop, and the rule behind it is blunt: a system refuses to start until every component has proven, on a throwaway input, that it can do its job. Three questions per component, asked in milliseconds, answered in one printed report.
Tests run on your laptop before you commit; a health check runs on the machine that is
about to do the work, in the moment before it does it. No test on your development box
knows that the Jetson in the hallway lost its network mount at 3 a.m., or that you
edited configs/personality.json last night and left a trailing comma.
Write both. This one costs about a hundred milliseconds and runs every start.
One result, one report, three probes
# glados/health_check.py
from dataclasses import dataclass
@dataclass
class CheckResult:
name: str
passed: bool
message: str = ""
def probe_stt(stt) -> CheckResult:
if stt is None:
return CheckResult("stt", False, "No STT callable injected")
result = stt("__probe__")
return CheckResult("stt", True, f"OK: returned {result!r}")
print(probe_stt(lambda path: "probe ok"))
print(probe_stt(None))
$ uv run python glados/health_check.py
CheckResult(name='stt', passed=True, message="OK: returned 'probe ok'")
CheckResult(name='stt', passed=False, message='No STT callable injected')
A probe has to report which component it looked at, whether it liked what it found, and
why. Returning (True, "...") would carry the same information and read as
check[1] at every use site; the dataclass from chapter 2 gives those two
fields names, and throws in the readable repr above for free. The
None guard comes first because it is the cheapest failure and the most
common one: a component that was never wired in at all.
from dataclasses import dataclass, field
@dataclass
class HealthReport:
checks: list[CheckResult] = field(default_factory=list)
@property
def overall_status(self) -> bool:
return all(c.passed for c in self.checks)
def add(self, result: CheckResult) -> None:
self.checks.append(result)
report = HealthReport()
report.add(CheckResult("stt", True, "ok"))
print("after stt:", report.overall_status)
report.add(CheckResult("tts", False, "missing"))
print("after tts:", report.overall_status)
$ uv run python glados/health_check.py
after stt: True
after tts: False
overall_status is a @property, not a field, and the two print
lines are there to show you why. It is computed from the checks, never set on its own,
so reading it after the second add gives an answer that accounts for the
second add. Store it as a plain field instead and you get the classic
stale-cache bug: all([]) is True, so a report built empty and
filled with failures would cheerfully report a healthy system. Note also
field(default_factory=list). A bare = [] default is evaluated
once at class definition and shared by every instance, so two reports would quietly
append into the same list.
def _probe_stt(core) -> CheckResult:
if core.stt is None:
return CheckResult("stt", False, "No STT callable injected")
try:
result = core.stt("__probe__")
if not isinstance(result, str):
return CheckResult("stt", False, f"Expected str, got {type(result).__name__}")
return CheckResult("stt", True, f"OK: returned {result[:40]!r}")
except Exception as e:
return CheckResult("stt", False, f"Raised: {e}")
def _probe_llm(core) -> CheckResult:
if core.llm is None:
return CheckResult("llm", False, "No LLM callable injected")
try:
result = core.llm("ping", [])
if not isinstance(result, str):
return CheckResult("llm", False, f"Expected str, got {type(result).__name__}")
return CheckResult("llm", True, f"OK: returned {result[:40]!r}")
except Exception as e:
return CheckResult("llm", False, f"Raised: {e}")
def _probe_tts(core) -> CheckResult:
if core.tts is None:
return CheckResult("tts", False, "No TTS callable injected")
try:
core.tts("__probe__")
return CheckResult("tts", True, "OK: callable executed without error")
except Exception as e:
return CheckResult("tts", False, f"Raised: {e}")
def run_checks(core) -> HealthReport:
report = HealthReport()
report.add(_probe_stt(core))
report.add(_probe_llm(core))
report.add(_probe_tts(core))
return report
def print_report(report: HealthReport) -> None:
print("\n=== GLaDOS Health Check ===")
for check in report.checks:
icon = "PASS" if check.passed else "FAIL"
print(f" [{icon}] {check.name}: {check.message}")
status = "ALL SYSTEMS GO" if report.overall_status else "SYSTEM NOT READY"
print(f"\n Status: {status}")
Three probes, one skeleton: present? runs without raising? returns the type the loop
depends on? The TTS probe stops after the second question on purpose, because speech
synthesis returns nothing useful; its contract is "does not explode." The broad
except Exception is deliberate too. A probe's job is to record a
failure, not to propagate one, and the failure box below shows what happens to a health
check that forgets that. Notice that run_checks returns data and
print_report does the printing: the same separation you have been holding
since chapter 2, which is what will let a future dashboard read the same report the
terminal does.
# glados/health_check.py — the gate and the demo
from glados.core import GladOSCore
def startup(core) -> HealthReport:
report = run_checks(core)
print_report(report)
if not report.overall_status:
failed = [c.name for c in report.checks if not c.passed]
raise RuntimeError(f"startup blocked: {', '.join(failed)} failed the probe")
return report
if __name__ == "__main__":
healthy = GladOSCore(
stt=lambda path: "probe ok",
llm=lambda text, history: "acknowledged",
tts=lambda text: None,
)
startup(healthy)
broken = GladOSCore(stt=None, llm=lambda text, history: 42, tts=None)
try:
startup(broken)
except RuntimeError as e:
print(f" {e}")
$ uv run python glados/health_check.py
=== GLaDOS Health Check ===
[PASS] stt: OK: returned 'probe ok'
[PASS] llm: OK: returned 'acknowledged'
[PASS] tts: OK: callable executed without error
Status: ALL SYSTEMS GO
=== GLaDOS Health Check ===
[FAIL] stt: No STT callable injected
[FAIL] llm: Expected str, got int
[FAIL] tts: No TTS callable injected
Status: SYSTEM NOT READY
startup blocked: stt, llm, tts failed the probe
The broken core is the interesting half. Its LLM is present and runs without complaint,
and it still fails, because 42 is not a string and the loop would have
handed that integer to the speech engine. Injectable collaborators are what make this
possible: because the core takes plain callables, a two-line lambda stands in for a
three-gigabyte model, and you can exercise both the healthy and the broken path in one
script that starts instantly. Call startup(core) as the first statement of
your entry point and a misconfiguration costs you a printed report instead of an
afternoon.
Why this works: ordered questions and a harmless input
Each probe is a short-circuit chain, and the order is load-bearing. The presence gate runs
first because calling None raises before any other question can be asked. The
callability gate runs second because you cannot inspect a return value that was never
produced. The contract gate runs last because it is the only one that assumes both
earlier answers were yes. Swap the type check ahead of the None guard and
your health check becomes the thing that crashes when a component is missing, which is
precisely the situation it exists to describe.
The sentinel string "__probe__" does quiet work here too. A real
transcriber treats its argument as a path and would raise
FileNotFoundError on any honest filename you invented, turning a working
component into a false failure; a real synthesizer handed real text would spend a second
generating audio nobody asked to hear. The sentinel keeps every probe cheap and free of
side effects, and the double underscores make it obvious in a log that no human typed it.
The generalization travels: validate at the boundary, before the expensive work, with the
smallest input that still exercises the wire.
Skipping the try/except feels reasonable while you write it. You are only
calling the component to see what comes back. Here is that version, run against an LLM
whose backend is not listening:
def _probe_llm(core) -> CheckResult:
result = core.llm("ping", []) # BUG: no try/except
return CheckResult("llm", isinstance(result, str), "checked")
def llm_that_fails(text, history):
raise ConnectionError("Ollama server not reachable at localhost:11434")
core = GladOSCore(stt=lambda p: "probe ok", llm=llm_that_fails, tts=lambda t: None)
print_report(run_checks(core))
$ uv run python glados/health_check.py
Traceback (most recent call last):
File "glados/health_check.py", line 71, in <module>
print_report(run_checks(core))
^^^^^^^^^^^^^^^^
File "glados/health_check.py", line 58, in run_checks
report.add(_probe_llm(core))
^^^^^^^^^^^^^^^^
File "glados/health_check.py", line 44, in _probe_llm
result = core.llm("ping", [])
^^^^^^^^^^^^^^^^^^^^
File "glados/health_check.py", line 48, in llm_that_fails
raise ConnectionError("Ollama server not reachable at localhost:11434")
ConnectionError: Ollama server not reachable at localhost:11434
Read what you lost. No report printed at all, so you never learned whether the
transcriber and the voice were fine; the process died on the first bad component and
told you about that one. The diagnostic tool became the crash. There is a second, worse
version of this: someone sees the traceback, decides health checks are noisy, and
deletes the call. Restore the try/except and the same broken backend
renders as [FAIL] llm: Raised: Ollama server not reachable at
localhost:11434, the other two probes still run, and the report tells you the
whole truth in one screen. A validator has to survive every failure it was built to
describe.
Checkpoint, and a config nobody checked
- I can name the three gates in a probe, in order, and say what breaks if I reverse any two of them.
- I can explain why
overall_statusrecomputes on every read, and what a stored field would report on an empty list of checks. - I can argue for
except Exceptioninside a probe while still calling it bad practice almost everywhere else. - I know why a probe passes
"__probe__"instead of a real audio path, and what a real transcriber would do with a plausible filename. - I can point at the exact line in an entry point where a bad config should stop being a runtime crash and start being a printed report.
- Handed a
[FAIL] llm: Expected str, got int, I can say which gate caught it and which two it cleared.
Exercise 1 — time every probe. Add a
latency_ms: float = 0.0 field to CheckResult and bracket
each probe with time.perf_counter(). Run it against a core whose TTS
sleeps for half a second. What does the report now tell you that it could not
before?
Capture start = time.perf_counter() at the top of the probe and
compute (time.perf_counter() - start) * 1000 on every return path,
including the failure ones. Giving the field a default keeps the stage-3 call sites
compiling unchanged. The new information is a passing component that is slowly
getting worse: a probe that used to answer in two milliseconds and now takes six
hundred is a model swapping to disk or a socket retrying, and it will feel like lag
in conversation long before it fails a gate.
Exercise 2 — write the report to disk. Add
save_report(report, path) using dataclasses.asdict, so a
failed start on the machine in the hallway leaves evidence. Check the JSON
carefully.
asdict walks the annotated fields and nothing else, so
overall_status is missing from the output until you add it to the
payload by hand: {"overall_status": report.overall_status, "checks":
[asdict(c) for c in report.checks]}. That omission is the property's design
showing its edge. Derived values live outside serialization, and any consumer of
the file either recomputes the verdict or reads the one you wrote deliberately.
Exercise 3 — an optional component. Write
_probe_memory(core) that stores and retrieves a test key, and treats a
missing memory component as a pass. Explain the asymmetry to yourself before you
write it.
Absent-but-optional and present-but-broken deserve different verdicts:
CheckResult("memory", True, "skipped: no memory component") for the
first, a False with the round-trip mismatch for the second. She can
hold a conversation with no long-term memory, so its absence must not block
startup, while a memory that accepts a write and returns something else on read is
actively lying to her. Use a sentinel key like "__probe_key__" and
delete it afterward, or your SQLite file from chapter 11 slowly fills with one
probe row per boot.
The gate now catches every component that is missing or broken. It says nothing at all about a component that is present, callable, correctly typed, and configured with a sample rate of 1600 or a path to a model file that was deleted last week. Those settings are read long before anyone calls a probe, and they fail just as late. Next chapter walks the config itself and collects every problem it finds into one list you can read before a single frame of audio is recorded.