GLaDOS Vol 4 · One System
ch 33 / 99
Chapter 33

The Startup Health Check

Assembled and working are two different claims

Last chapter left you with a core that holds a listener, a brain, and a voice, and runs a turn by calling them in order. Every attribute is filled in. Nothing is None. The object prints beautifully. And none of that is evidence that a conversation will survive contact with the hardware, because a callable can be present and still be wrong: a model file moved, a server is down, a swapped engine now returns bytes where the loop expects text.

Left alone, that discovery arrives at the worst possible time. You speak. The recorder fills five seconds of buffer. Whisper spends two more turning it into text. The language model burns another three composing a reply. Then, seven seconds into what felt like a conversation, self.tts(reply) raises TypeError: 'NoneType' object is not callable, and the traceback points at the voice loop, which is innocent. The real fault was a misspelled key in a config file, decided before you ever opened your mouth.

So this chapter builds a gate in front of the loop, and the rule behind it is blunt: a system refuses to start until every component has proven, on a throwaway input, that it can do its job. Three questions per component, asked in milliseconds, answered in one printed report.

◆ Note — this is not a test suite

Tests run on your laptop before you commit; a health check runs on the machine that is about to do the work, in the moment before it does it. No test on your development box knows that the Jetson in the hallway lost its network mount at 3 a.m., or that you edited configs/personality.json last night and left a trailing comma. Write both. This one costs about a hundred milliseconds and runs every start.

One result, one report, three probes

▣ Build · stage 1 — a probe that answers three things at once
# glados/health_check.py
from dataclasses import dataclass

@dataclass
class CheckResult:
    name: str
    passed: bool
    message: str = ""

def probe_stt(stt) -> CheckResult:
    if stt is None:
        return CheckResult("stt", False, "No STT callable injected")
    result = stt("__probe__")
    return CheckResult("stt", True, f"OK: returned {result!r}")

print(probe_stt(lambda path: "probe ok"))
print(probe_stt(None))
$ uv run python glados/health_check.py
CheckResult(name='stt', passed=True, message="OK: returned 'probe ok'")
CheckResult(name='stt', passed=False, message='No STT callable injected')

A probe has to report which component it looked at, whether it liked what it found, and why. Returning (True, "...") would carry the same information and read as check[1] at every use site; the dataclass from chapter 2 gives those two fields names, and throws in the readable repr above for free. The None guard comes first because it is the cheapest failure and the most common one: a component that was never wired in at all.

▣ Build · stage 2 — collect results, derive the verdict
from dataclasses import dataclass, field

@dataclass
class HealthReport:
    checks: list[CheckResult] = field(default_factory=list)

    @property
    def overall_status(self) -> bool:
        return all(c.passed for c in self.checks)

    def add(self, result: CheckResult) -> None:
        self.checks.append(result)

report = HealthReport()
report.add(CheckResult("stt", True, "ok"))
print("after stt:", report.overall_status)
report.add(CheckResult("tts", False, "missing"))
print("after tts:", report.overall_status)
$ uv run python glados/health_check.py
after stt: True
after tts: False

overall_status is a @property, not a field, and the two print lines are there to show you why. It is computed from the checks, never set on its own, so reading it after the second add gives an answer that accounts for the second add. Store it as a plain field instead and you get the classic stale-cache bug: all([]) is True, so a report built empty and filled with failures would cheerfully report a healthy system. Note also field(default_factory=list). A bare = [] default is evaluated once at class definition and shared by every instance, so two reports would quietly append into the same list.

▣ Build · stage 3 — the three gates, and a report you can read
def _probe_stt(core) -> CheckResult:
    if core.stt is None:
        return CheckResult("stt", False, "No STT callable injected")
    try:
        result = core.stt("__probe__")
        if not isinstance(result, str):
            return CheckResult("stt", False, f"Expected str, got {type(result).__name__}")
        return CheckResult("stt", True, f"OK: returned {result[:40]!r}")
    except Exception as e:
        return CheckResult("stt", False, f"Raised: {e}")

def _probe_llm(core) -> CheckResult:
    if core.llm is None:
        return CheckResult("llm", False, "No LLM callable injected")
    try:
        result = core.llm("ping", [])
        if not isinstance(result, str):
            return CheckResult("llm", False, f"Expected str, got {type(result).__name__}")
        return CheckResult("llm", True, f"OK: returned {result[:40]!r}")
    except Exception as e:
        return CheckResult("llm", False, f"Raised: {e}")

def _probe_tts(core) -> CheckResult:
    if core.tts is None:
        return CheckResult("tts", False, "No TTS callable injected")
    try:
        core.tts("__probe__")
        return CheckResult("tts", True, "OK: callable executed without error")
    except Exception as e:
        return CheckResult("tts", False, f"Raised: {e}")

def run_checks(core) -> HealthReport:
    report = HealthReport()
    report.add(_probe_stt(core))
    report.add(_probe_llm(core))
    report.add(_probe_tts(core))
    return report

def print_report(report: HealthReport) -> None:
    print("\n=== GLaDOS Health Check ===")
    for check in report.checks:
        icon = "PASS" if check.passed else "FAIL"
        print(f"  [{icon}] {check.name}: {check.message}")
    status = "ALL SYSTEMS GO" if report.overall_status else "SYSTEM NOT READY"
    print(f"\n  Status: {status}")

Three probes, one skeleton: present? runs without raising? returns the type the loop depends on? The TTS probe stops after the second question on purpose, because speech synthesis returns nothing useful; its contract is "does not explode." The broad except Exception is deliberate too. A probe's job is to record a failure, not to propagate one, and the failure box below shows what happens to a health check that forgets that. Notice that run_checks returns data and print_report does the printing: the same separation you have been holding since chapter 2, which is what will let a future dashboard read the same report the terminal does.

▣ Build · stage 4 — wire it to the startup path
# glados/health_check.py — the gate and the demo
from glados.core import GladOSCore

def startup(core) -> HealthReport:
    report = run_checks(core)
    print_report(report)
    if not report.overall_status:
        failed = [c.name for c in report.checks if not c.passed]
        raise RuntimeError(f"startup blocked: {', '.join(failed)} failed the probe")
    return report

if __name__ == "__main__":
    healthy = GladOSCore(
        stt=lambda path: "probe ok",
        llm=lambda text, history: "acknowledged",
        tts=lambda text: None,
    )
    startup(healthy)

    broken = GladOSCore(stt=None, llm=lambda text, history: 42, tts=None)
    try:
        startup(broken)
    except RuntimeError as e:
        print(f"  {e}")
$ uv run python glados/health_check.py

=== GLaDOS Health Check ===
  [PASS] stt: OK: returned 'probe ok'
  [PASS] llm: OK: returned 'acknowledged'
  [PASS] tts: OK: callable executed without error

  Status: ALL SYSTEMS GO

=== GLaDOS Health Check ===
  [FAIL] stt: No STT callable injected
  [FAIL] llm: Expected str, got int
  [FAIL] tts: No TTS callable injected

  Status: SYSTEM NOT READY
  startup blocked: stt, llm, tts failed the probe

The broken core is the interesting half. Its LLM is present and runs without complaint, and it still fails, because 42 is not a string and the loop would have handed that integer to the speech engine. Injectable collaborators are what make this possible: because the core takes plain callables, a two-line lambda stands in for a three-gigabyte model, and you can exercise both the healthy and the broken path in one script that starts instantly. Call startup(core) as the first statement of your entry point and a misconfiguration costs you a printed report instead of an afternoon.

Why this works: ordered questions and a harmless input

Each probe is a short-circuit chain, and the order is load-bearing. The presence gate runs first because calling None raises before any other question can be asked. The callability gate runs second because you cannot inspect a return value that was never produced. The contract gate runs last because it is the only one that assumes both earlier answers were yes. Swap the type check ahead of the None guard and your health check becomes the thing that crashes when a component is missing, which is precisely the situation it exists to describe.

The sentinel string "__probe__" does quiet work here too. A real transcriber treats its argument as a path and would raise FileNotFoundError on any honest filename you invented, turning a working component into a false failure; a real synthesizer handed real text would spend a second generating audio nobody asked to hear. The sentinel keeps every probe cheap and free of side effects, and the double underscores make it obvious in a log that no human typed it. The generalization travels: validate at the boundary, before the expensive work, with the smallest input that still exercises the wire.

⚠ Worked failure — the health check that needed a health check

Skipping the try/except feels reasonable while you write it. You are only calling the component to see what comes back. Here is that version, run against an LLM whose backend is not listening:

def _probe_llm(core) -> CheckResult:
    result = core.llm("ping", [])            # BUG: no try/except
    return CheckResult("llm", isinstance(result, str), "checked")

def llm_that_fails(text, history):
    raise ConnectionError("Ollama server not reachable at localhost:11434")

core = GladOSCore(stt=lambda p: "probe ok", llm=llm_that_fails, tts=lambda t: None)
print_report(run_checks(core))
$ uv run python glados/health_check.py
Traceback (most recent call last):
  File "glados/health_check.py", line 71, in <module>
    print_report(run_checks(core))
                 ^^^^^^^^^^^^^^^^
  File "glados/health_check.py", line 58, in run_checks
    report.add(_probe_llm(core))
               ^^^^^^^^^^^^^^^^
  File "glados/health_check.py", line 44, in _probe_llm
    result = core.llm("ping", [])
             ^^^^^^^^^^^^^^^^^^^^
  File "glados/health_check.py", line 48, in llm_that_fails
    raise ConnectionError("Ollama server not reachable at localhost:11434")
ConnectionError: Ollama server not reachable at localhost:11434

Read what you lost. No report printed at all, so you never learned whether the transcriber and the voice were fine; the process died on the first bad component and told you about that one. The diagnostic tool became the crash. There is a second, worse version of this: someone sees the traceback, decides health checks are noisy, and deletes the call. Restore the try/except and the same broken backend renders as [FAIL] llm: Raised: Ollama server not reachable at localhost:11434, the other two probes still run, and the report tells you the whole truth in one screen. A validator has to survive every failure it was built to describe.

Checkpoint, and a config nobody checked

✓ Checkpoint — what you can now do
  • I can name the three gates in a probe, in order, and say what breaks if I reverse any two of them.
  • I can explain why overall_status recomputes on every read, and what a stored field would report on an empty list of checks.
  • I can argue for except Exception inside a probe while still calling it bad practice almost everywhere else.
  • I know why a probe passes "__probe__" instead of a real audio path, and what a real transcriber would do with a plausible filename.
  • I can point at the exact line in an entry point where a bad config should stop being a runtime crash and start being a printed report.
  • Handed a [FAIL] llm: Expected str, got int, I can say which gate caught it and which two it cleared.
⚡ Exercises — try first, then reveal
Exercise 1 — time every probe. Add a latency_ms: float = 0.0 field to CheckResult and bracket each probe with time.perf_counter(). Run it against a core whose TTS sleeps for half a second. What does the report now tell you that it could not before?

Capture start = time.perf_counter() at the top of the probe and compute (time.perf_counter() - start) * 1000 on every return path, including the failure ones. Giving the field a default keeps the stage-3 call sites compiling unchanged. The new information is a passing component that is slowly getting worse: a probe that used to answer in two milliseconds and now takes six hundred is a model swapping to disk or a socket retrying, and it will feel like lag in conversation long before it fails a gate.

Exercise 2 — write the report to disk. Add save_report(report, path) using dataclasses.asdict, so a failed start on the machine in the hallway leaves evidence. Check the JSON carefully.

asdict walks the annotated fields and nothing else, so overall_status is missing from the output until you add it to the payload by hand: {"overall_status": report.overall_status, "checks": [asdict(c) for c in report.checks]}. That omission is the property's design showing its edge. Derived values live outside serialization, and any consumer of the file either recomputes the verdict or reads the one you wrote deliberately.

Exercise 3 — an optional component. Write _probe_memory(core) that stores and retrieves a test key, and treats a missing memory component as a pass. Explain the asymmetry to yourself before you write it.

Absent-but-optional and present-but-broken deserve different verdicts: CheckResult("memory", True, "skipped: no memory component") for the first, a False with the round-trip mismatch for the second. She can hold a conversation with no long-term memory, so its absence must not block startup, while a memory that accepts a write and returns something else on read is actively lying to her. Use a sentinel key like "__probe_key__" and delete it afterward, or your SQLite file from chapter 11 slowly fills with one probe row per boot.

The gate now catches every component that is missing or broken. It says nothing at all about a component that is present, callable, correctly typed, and configured with a sample rate of 1600 or a path to a model file that was deleted last week. Those settings are read long before anyone calls a probe, and they fail just as late. Next chapter walks the config itself and collects every problem it finds into one list you can read before a single frame of audio is recorded.