Full System Test
Every subsystem has been tested alone, and only alone
The eye behaved on an afternoon when nothing else was moving. The ring lit up while the arm sat unpowered. The microphones were measured in a quiet room, the speaker checked before the pack was even in the shell, and the arm homed on a bench supply that never sagged. Eleven chapters of hardware, eleven separate good days.
A machine is not the sum of its good days. Servos pull current in bursts that reach the light ring through a shared pack. A microphone that heard you at arm's length may be sitting next to a speaker cone now. Calibration numbers measured last week describe a bracket that has been unbolted and refitted twice since. None of that is visible from any single subsystem's own test, because each of those tests carefully removed the others.
So before the software stack goes back on top of this, one run has to answer one question with evidence: does every part of her work, right now, at the same time? The obvious way to write that run is six calls in a row, and it fails on contact with hardware. The first call raises, the program dies, and you have learned the state of exactly one subsystem out of six. Six faults become six round trips through a machine you have to power down between attempts.
Chapter 45 solved half of this for software tests and chapter 58 for the seams between them: wrap each check so a raised exception becomes a recorded result. This chapter puts that runner in front of real hardware and adds the two things hardware needs. Every row carries the number it measured next to the limit it was measured against, the way chapter 42 insisted. And the run ends in a word, the way chapter 59's acceptance report does. One rule holds it together: each check owns its own failure and returns a row instead of raising, every row starts as not-yet-run and can only be moved to PASS by a probe that measured something, and the verdict is GO only when all six rows say PASS.
Ninety seconds is a budget, chosen so the test is short enough to run before every session and long enough to contain a three second capture, a real arm motion and ten seconds of loaded rail sampling. Three of the six checks end by asking you a question, because a servo reports nothing back and a pixel has no way to tell you what colour it is. Those three are the reason this is a test you watch rather than a job you schedule. Every number quoted below came off one assembled body on one evening. Yours will differ, and the thresholds need review for your actual components before you use them. GO records this supervised functional check only. It does not certify fault-current protection, unseen voltage transients or an independent motion-power stop; establish those separately as chapter 70 requires before energizing the body.
One row per subsystem, and a default of not yet
# labs/hardware_check.py
import time
from dataclasses import dataclass
from typing import Callable
PASS, FAIL, SKIP = "PASS", "FAIL", "SKIP"
@dataclass
class Check:
"""One subsystem's row in the report. SKIP until a probe reports otherwise."""
name: str
budget_s: float
status: str = SKIP
detail: str = ""
elapsed_s: float = 0.0
def show(check: Check) -> Check:
print(f" [{check.status}] {check.name:<14}{check.elapsed_s:5.1f}s {check.detail}")
return check
def run_check(check: Check, probe: Callable[[], tuple[bool, str]]) -> Check:
"""Run one probe. Whatever it raises becomes this row's detail, not a traceback."""
start = time.perf_counter()
try:
ok, detail = probe()
check.status, check.detail = (PASS if ok else FAIL), detail
except Exception as exc:
check.status, check.detail = FAIL, f"{type(exc).__name__}: {exc}"
check.elapsed_s = time.perf_counter() - start
return show(check)
def skip(check: Check, why: str) -> Check:
check.status, check.detail = SKIP, why
return show(check)
if __name__ == "__main__":
def eye() -> tuple[bool, str]:
time.sleep(0.2)
return True, "eye_pan 1250-1700 us, eye_tilt 1350-1650 us"
def microphones() -> tuple[bool, str]:
time.sleep(3.1)
return False, "rms 0.00002, gate 0.00500"
def power() -> tuple[bool, str]:
raise FileNotFoundError(2, "No such file or directory", "/dev/spidev4.0")
for check, probe in ((Check("eye", 12.0), eye),
(Check("microphones", 10.0), microphones),
(Check("power", 22.0), power)):
run_check(check, probe)
print("the runner is still here")
$ uv run python -m labs.hardware_check
[PASS] eye 0.2s eye_pan 1250-1700 us, eye_tilt 1350-1650 us
[FAIL] microphones 3.1s rms 0.00002, gate 0.00500
[FAIL] power 0.0s FileNotFoundError: [Errno 2] No such file or directory: '/dev/spidev4.0'
the runner is still here
Three probes and three different endings, and the runner treats all of them as data. The middle one is the interesting case. It did not raise: it ran to completion, took its three seconds, and reported that what it heard was two hundred and fifty times quieter than the gate. A crash and a measurement that missed are different events for the person reading the report, and identical for the code that has to keep going.
Two smaller decisions carry weight later. The status field starts at
SKIP, which is chapter 58's habit of defaulting a result to not-passed and
making something prove otherwise. And run_check stamps
elapsed_s after the except as well as after the happy path, so
a check that dies at 19 seconds is distinguishable from one that dies instantly. The
first is a subsystem that half worked. The second is usually a missing file.
# labs/hardware_check.py — continued
BUDGET_S = 90.0
def tally(checks: list[Check]) -> dict[str, int]:
return {status: sum(1 for c in checks if c.status == status)
for status in (PASS, FAIL, SKIP)}
def verdict(checks: list[Check]) -> str:
"""GO needs every row to have passed. A check nobody ran is not a check that passed."""
return "GO" if all(c.status == PASS for c in checks) else "NO-GO"
def summarise(checks: list[Check]) -> str:
counts = tally(checks)
elapsed = sum(c.elapsed_s for c in checks)
unresolved = [c.name for c in checks if c.status != PASS]
call = verdict(checks)
print(f"\n{counts[PASS]} passed, {counts[FAIL]} failed, {counts[SKIP]} not run"
f" {elapsed:.1f}s of a {BUDGET_S:.0f}s budget")
print(f"{call} - " + ("every subsystem answered for itself" if not unresolved
else "unresolved: " + ", ".join(unresolved)))
return call
verdict asks whether everything passed. Written the other way round, as a
search for failures, it reads better and it is wrong, for a reason the worked failure at
the end of this chapter pays for in a committed report. Three states exist, and only
two of them are opinions about the hardware. SKIP is an opinion about the
test.
# labs/signoff_config.py
import json
from pathlib import Path
from typing import Any
HW_PATH = Path("configs/hardware.json")
CAL_PATH = Path("configs/calibration.json")
def load(path: Path) -> dict:
"""An absent or unreadable config is an empty dict."""
try:
return json.loads(path.read_text())
except FileNotFoundError:
return {}
except (OSError, json.JSONDecodeError) as exc:
print(f" {path} cannot be read as JSON and will be treated as empty: {exc}")
return {}
def need(config: dict, path: Path, *keys: str) -> Any:
"""Walk into a config, or raise naming both the key and the file it was missing from."""
node: Any = config
for depth, key in enumerate(keys, start=1):
if not isinstance(node, dict) or key not in node:
raise LookupError(f"{'.'.join(keys[:depth])} missing from {path}")
node = node[key]
return node
if __name__ == "__main__":
hw, cal = load(HW_PATH), load(CAL_PATH)
print(f"{len(hw)} sections in {HW_PATH}, {len(cal)} servos in {CAL_PATH}")
for keys in (("audio", "mic_name_substr"), ("arm", "channels"), ("audio", "speaker")):
try:
print(f" {'.'.join(keys):<24} -> {need(hw, HW_PATH, *keys)}")
except LookupError as exc:
print(f" {'.'.join(keys):<24} -> {exc}")
$ uv run python -m labs.signoff_config
6 sections in configs/hardware.json, 3 servos in configs/calibration.json
audio.mic_name_substr -> ReSpeaker
arm.channels -> {'shoulder': 0, 'elbow': 1, 'wrist': 2, 'gripper': 3}
audio.speaker -> audio.speaker missing from configs/hardware.json
$ mv configs configs.away && uv run python -m labs.signoff_config
0 sections in configs/hardware.json, 0 servos in configs/calibration.json
audio.mic_name_substr -> audio missing from configs/hardware.json
arm.channels -> arm missing from configs/hardware.json
audio.speaker -> audio missing from configs/hardware.json
The two halves degrade in opposite directions on purpose. load never
raises, because a diagnostic tool that cannot start on a machine with no config is
useless on precisely the machine you most need to diagnose. need always
raises, because the alternative is a default, and defaults here are how hardware gets
broken. Chapter 73 measured this eye at 1250 to 1700 microseconds; the library's
fallback of 1000 to 2000 would drive the pan servo hard into a printed yoke and hold it
there. A missing calibration file must stop the eye check, and it must stop nothing
else.
Notice what the error string carries: the key path and the file. That message ends up
in a JSON report someone reads on another machine in six months, so "missing from
configs/hardware.json" is the difference between a fix and an investigation. The
keys[:depth] slice is why the second run says audio instead of
audio.mic_name_substr. The whole section is gone, not just the field, and
the message says which.
The four checks that go first
# labs/signoff_checks.py
import time
from labs.calibrated import axis_limit_deg, calibrated_eye
from labs.led_ring import EyeRing
from labs.mood_state import MoodState
from labs.signoff_config import CAL_PATH, HW_PATH, need
EYE_DWELL_S = 1.2
RING_DWELL_S = 1.5
RING_MOODS = ("hostile", "satisfied", "neutral") # red, green, blue: all three dice
def watched(question: str) -> bool:
"""The part of a hardware test that no sensor on this machine can perform."""
return input(f" {question} [y/N]: ").strip().lower() == "y"
def probe_eye(hw: dict, cal: dict) -> tuple[bool, str]:
"""Both axes out to their measured stops and back to centre, slowly enough to follow."""
entries = {name: need(cal, CAL_PATH, name) for name in ("eye_pan", "eye_tilt")}
for name, entry in entries.items():
if entry.get("travel_deg") is None:
return False, f"{name} has no measured travel; calibrate before testing"
pan_limit = axis_limit_deg(entries["eye_pan"])
tilt_limit = axis_limit_deg(entries["eye_tilt"])
eye = calibrated_eye()
for pan, tilt in ((-pan_limit, 0.0), (pan_limit, 0.0), (0.0, -tilt_limit),
(0.0, tilt_limit), (0.0, 0.0)):
eye.look_at(pan, tilt)
time.sleep(EYE_DWELL_S)
detail = f"pan +-{pan_limit:.1f} deg, tilt +-{tilt_limit:.1f} deg, four stops and centre"
return watched("did the eye reach all four stops and return to centre?"), detail
def probe_ring(hw: dict, cal: dict) -> tuple[bool, str]:
"""Twenty four pixels through three moods and then dark."""
spi0 = need(hw, HW_PATH, "buses", "spi0")
if not spi0.get("exclusive"):
return False, "spi0 is not declared exclusive; the ring cannot share that line"
ring = EyeRing()
for mood in RING_MOODS:
ring.show_mood(MoodState(mood, 1.0))
time.sleep(RING_DWELL_S)
ring.off()
detail = f"24 pixels through {', '.join(RING_MOODS)}, dark"
return watched("did all 24 pixels light red, then green, then blue, then go dark?"), detail
Both probes take the same two config dictionaries whether they need them or not, so the
runner can call any of them the same way, and both return a pair: a boolean and the
sentence that will be in the report. Neither prints a verdict of its own, because
run_check owns that line.
The eye probe refuses in two distinct ways before it ever moves. A missing calibration
entry raises out of need and lands in the row as a
LookupError. An entry that exists but carries a null travel returns
False with a sentence about it, because that entry is real: chapter 73
writes exactly that record for a servo you declined to calibrate. The first is a broken
setup, the second is unfinished work, and the report should not call them the same thing.
The three ring moods are chosen for their hex values, not their meanings. Hostile is #ff4444, satisfied is #39ff14 and neutral is #4a9eff, so the sequence drives every red, green and blue die in all twenty four packages. A pixel with a dead green channel looks perfectly healthy showing hostile and gives itself away one and a half seconds later.
# labs/signoff_checks.py — continued
import numpy as np
import sounddevice as sd
from labs.audio_out import SAMPLE_RATE, apply_fades, make_sine_tone, scale_volume
from labs.mic_distance import SILENCE_GATE
from labs.mic_setup import SAMPLE_RATE as MIC_RATE, find_input_device, record, signal_report
LOOPBACK_RMS = 0.020 # a tone at 0.70 across a desk; a dead amplifier is nowhere near it
TONE_SECONDS = 2.0
def input_index(hw: dict) -> int:
"""Resolve the array quietly: this program's console output is the report."""
name = need(hw, HW_PATH, "audio", "mic_name_substr")
index = find_input_device(sd.query_devices(), name)
if index is None:
raise LookupError(f"no input device whose name contains {name!r}")
return index
def output_index(hw: dict) -> int:
name = need(hw, HW_PATH, "audio", "output_name_substr")
for index, dev in enumerate(sd.query_devices()):
if dev["max_output_channels"] > 0 and name.lower() in dev["name"].lower():
return index
raise LookupError(f"no output device whose name contains {name!r}")
def probe_microphones(hw: dict, cal: dict) -> tuple[bool, str]:
"""Three seconds of the room, judged by the gate the wake word already uses."""
index = input_index(hw)
print(" say something for three seconds")
peak, rms = signal_report(record(index, seconds=3))
detail = f"index {index}, peak {peak:.4f}, rms {rms:.4f}, gate {SILENCE_GATE:.4f}"
return rms >= SILENCE_GATE, detail
def probe_speaker(hw: dict, cal: dict) -> tuple[bool, str]:
"""Play a tone on the amplifier and listen for it on the array that just passed."""
heard: list[np.ndarray] = []
def keep(data: np.ndarray, frames: int, timing: object, status: object) -> None:
heard.append(data.copy()) # the driver reuses this buffer on the next call
tone = scale_volume(apply_fades(make_sine_tone(duration=TONE_SECONDS)), 0.7)
with sd.InputStream(device=input_index(hw), samplerate=MIC_RATE, channels=1,
dtype="float32", callback=keep):
sd.play(tone, samplerate=SAMPLE_RATE, device=output_index(hw), blocking=True)
time.sleep(0.2)
_, rms = signal_report(np.concatenate(heard) if heard else np.zeros(1, dtype=np.float32))
detail = f"440 Hz at 0.70, array heard rms {rms:.4f}, floor {LOOPBACK_RMS:.4f}"
return rms >= LOOPBACK_RMS, detail
The microphone gate is imported, not invented. SILENCE_GATE is the number
chapter 67 calibrated for deciding whether anyone is speaking, so a machine that passes
this check passes it at the level the pipeline will actually demand. A test with its own
private threshold can be green while the system it is testing is deaf.
The speaker check is the first place two subsystems are used at once, and it is legal here for one reason: the microphones ran first and passed. If the array had failed, this row would be measuring an unknown against an unknown, and the runner would not have started it. What the dependency buys is a speaker check with a number in it. Every other way of testing an amplifier ends in "did you hear that", and the room answers this one.
sd.play and sd.rec both write into one module-level stream
slot inside sounddevice, so calling them back to back stops the recording the instant
playback starts. An explicit InputStream is a separate object that the
playback call cannot touch, and the with block closes it even if the tone
fails to reach the amplifier. Inside the callback, data.copy() is not
caution: the driver hands the same buffer back on the next call, so keeping a reference
gives you a list whose entries all quietly become the last block recorded.
The arm, the rails under it, and one command
# labs/signoff_checks.py — continued
import threading
import spidev
from labs.arm_control import RoboticArm
from labs.arm_driver import ChannelDriver
from labs.power_monitor import SAMPLE_HZ, SPI_BUS, SPI_DEVICE, SPI_HZ, read_channel
from labs.power_sense import SENSES, adc_to_volts, capacity_percent
from labs.wire_drop import SERVO_MIN_V
ARM_DWELL_S = 2.0
MIN_CAPACITY_PCT = 20
REST_S, LOAD_S = 3.0, 10.0
def probe_arm(hw: dict, cal: dict) -> tuple[bool, str]:
"""Home, reach, grab, reach, release, home, relax, with time to see each one."""
channels = need(hw, HW_PATH, "arm", "channels")
arm = RoboticArm(ChannelDriver())
arm.home()
for move in (lambda: arm.reach(45.0, 90.0, 90.0), arm.grab,
lambda: arm.reach(120.0, 60.0, 90.0), arm.release, arm.home):
time.sleep(ARM_DWELL_S)
move()
arm.relax()
detail = f"{len(channels)} joints: home, reach, grab, reach, release, home, relax"
return watched("did all four joints move, and is the arm folded and limp now?"), detail
def sample(spi: spidev.SpiDev, seconds: float) -> dict[str, list[int]]:
"""Every sense channel, SAMPLE_HZ times a second, for this long."""
raws: dict[str, list[int]] = {name: [] for name in SENSES}
end = time.monotonic() + seconds
while time.monotonic() < end:
for name, sense in SENSES.items():
raws[name].append(read_channel(spi, sense.channel))
time.sleep(1 / SAMPLE_HZ)
return raws
def floor_volts(raws: dict[str, list[int]], rail: str) -> float:
"""The worst sample in a window: what the rail fell to, not what it averaged."""
return adc_to_volts(min(raws[rail]), SENSES[rail])
def top_volts(raws: dict[str, list[int]], rail: str) -> float:
return adc_to_volts(max(raws[rail]), SENSES[rail])
# labs/signoff_checks.py — continued
def probe_power(hw: dict, cal: dict) -> tuple[bool, str]:
"""Both rails at rest, then both rails with the ring full and the arm working."""
ring, arm = EyeRing(), RoboticArm(ChannelDriver()) # neither holds anything yet
spi = spidev.SpiDev()
spi.open(SPI_BUS, SPI_DEVICE)
spi.max_speed_hz = SPI_HZ
try:
rest = sample(spi, REST_S)
ring.show_mood(MoodState("neutral", 1.0))
stir = threading.Thread(target=arm.reach, args=(120.0, 60.0, 90.0), daemon=True)
stir.start()
loaded = sample(spi, LOAD_S)
stir.join()
finally:
arm.relax()
ring.off()
spi.close()
pack_rest, pack_low = top_volts(rest, "pack"), floor_volts(loaded, "pack")
rail_rest, rail_low = top_volts(rest, "motion"), floor_volts(loaded, "motion")
percent = capacity_percent(pack_rest)
detail = (f"pack {pack_rest:.2f}/{pack_low:.2f} V {percent}%, "
f"motion {rail_rest:.2f}/{rail_low:.2f} V, floor {SERVO_MIN_V:.2f} V")
return rail_low >= SERVO_MIN_V and percent >= MIN_CAPACITY_PCT, detail
The power check is the only one that tests the machine as a machine. A voltage read while nothing is happening says almost nothing: the rails are supposed to be fine at rest. What matters is the ten seconds when twenty four pixels are at full and three joints are accelerating together, and what matters inside those ten seconds is the worst observed sample, not the average. Fifty samples a second nominally gives about five hundred samples in ten seconds; reads and scheduling add delay. The roughly twenty millisecond gaps can miss a dip lasting only a few milliseconds, even over a longer window. A passing row means none of the recorded samples crossed the limit. Chapter 70's warning still applies: capture short transients with an appropriately configured oscilloscope or faster acquisition system before making claims about them.
The arm runs on a thread so the sampling loop keeps its rhythm while the joints move.
The finally is not decoration. If the ADC stops answering halfway through
the load window, the exception propagates to the runner and becomes a FAIL row, and
without that block it would leave the ring at full brightness and four servos holding
torque against a pose nobody chose. A check that fails must still put the hardware down.
Keep four files: hardware_check.py owns result states,
signoff_config.py owns configuration helpers,
signoff_checks.py owns probes, and the complete
signoff.py below owns orchestration. The probes and runner both import
the helper module, which imports neither of them. That removes the circular import
in either module entry path. Import every status used by the report, including
FAIL and SKIP. Probe imports stay inside
main, so python -m unittest labs.test_signoff can exercise
report construction without importing device libraries or running a probe.
# labs/signoff.py — complete file; helpers stay in signoff_config.py
import argparse
import json
from datetime import datetime
from functools import partial
from pathlib import Path
from labs.hardware_check import (BUDGET_S, FAIL, PASS, SKIP, Check, run_check, skip,
summarise, tally, verdict)
from labs.signoff_config import CAL_PATH, HW_PATH, load
REPORT_PATH = Path("glados/data/hardware_signoff.json")
BUDGETS = {"eye": 12.0, "light ring": 10.0, "microphones": 10.0,
"speaker": 12.0, "arm": 24.0, "power": 22.0}
NEEDS = {"speaker": ("microphones",), "power": ("light ring", "arm")}
def last_calibration(cal: dict) -> str:
dates = [entry["measured_at"] for entry in cal.values() if "measured_at" in entry]
return max(dates) if dates else "never"
def write_report(checks: list[Check], hw: dict, cal: dict, started: datetime) -> None:
counts = tally(checks)
REPORT_PATH.parent.mkdir(parents=True, exist_ok=True)
REPORT_PATH.write_text(json.dumps({
"started_at": started.isoformat(timespec="seconds"),
"body_revision": hw.get("revision", "unknown"),
"calibrated_at": last_calibration(cal),
"verdict": verdict(checks),
"budget_s": BUDGET_S,
"elapsed_s": round(sum(c.elapsed_s for c in checks), 1),
"passed": counts[PASS], "failed": counts[FAIL], "not_run": counts[SKIP],
"checks": [{"name": c.name, "status": c.status, "detail": c.detail,
"budget_s": c.budget_s, "elapsed_s": round(c.elapsed_s, 1)}
for c in checks],
}, indent=2) + "\n")
def main() -> None:
from labs.signoff_checks import (probe_arm, probe_eye, probe_microphones,
probe_power, probe_ring, probe_speaker)
probes = {"eye": probe_eye, "light ring": probe_ring, "microphones": probe_microphones,
"speaker": probe_speaker, "arm": probe_arm, "power": probe_power}
parser = argparse.ArgumentParser(description="GLaDOS full system test")
parser.add_argument("--skip", nargs="*", default=[], choices=list(probes),
help="subsystems that are off the machine right now")
args = parser.parse_args()
hw, cal = load(HW_PATH), load(CAL_PATH)
checks = [Check(name, BUDGETS[name]) for name in probes]
by_name = {c.name: c for c in checks}
started = datetime.now().astimezone()
print(f"GLaDOS body rev {hw.get('revision', 'unknown')}, full system test, "
f"{len(checks)} checks, {BUDGET_S:.0f}s budget")
for check in checks:
blocked = [n for n in NEEDS.get(check.name, ()) if by_name[n].status != PASS]
if check.name in args.skip:
skip(check, f"not run: --skip {check.name}")
elif blocked:
skip(check, f"not run: {' and '.join(blocked)} did not pass")
else:
run_check(check, partial(probes[check.name], hw, cal))
summarise(checks)
write_report(checks, hw, cal, started)
print(f"wrote {REPORT_PATH}")
if __name__ == "__main__":
main()
The following bench record is retained from the earlier build. The import and report corrections have offline checks, but no new hardware run accompanies this revision; these readings are not evidence that the revised program passed on your body. Run a new supervised sign-off after validating your hardware configuration.
$ uv run python -m labs.signoff # measured on the bench — yours will vary
GLaDOS body rev 1.0.0, full system test, 6 checks, 90s budget
did the eye reach all four stops and return to centre? [y/N]: y
[PASS] eye 11.4s pan +-25.5 deg, tilt +-17.0 deg, four stops and centre
did all 24 pixels light red, then green, then blue, then go dark? [y/N]: y
[PASS] light ring 9.6s 24 pixels through hostile, satisfied, neutral, dark
say something for three seconds
[PASS] microphones 6.7s index 1, peak 0.3120, rms 0.0912, gate 0.0050
[PASS] speaker 8.8s 440 Hz at 0.70, array heard rms 0.1642, floor 0.0200
did all four joints move, and is the arm folded and limp now? [y/N]: y
[PASS] arm 19.6s 4 joints: home, reach, grab, reach, release, home, relax
[PASS] power 17.3s pack 7.92/7.68 V 72%, motion 5.02/4.86 V, floor 4.60 V
6 passed, 0 failed, 0 not run 73.4s of a 90s budget
GO - every subsystem answered for itself
wrote glados/data/hardware_signoff.json
That is the whole body reporting for itself in seventy three seconds. The motion rail fell to 4.86 volts with the ring at full and three joints accelerating, which clears the 4.60 volt floor chapter 70 set for these servos by 0.26 volts, on a pack still holding seventy two percent. The array heard the tone at an RMS of 0.1642 against a floor of 0.0200, so the amplifier, the speaker, the microphones and the two metres of air between them all work in the same room at the same time.
$ mv configs configs.away && uv run python -m labs.signoff
GLaDOS body rev unknown, full system test, 6 checks, 90s budget
[FAIL] eye 0.0s LookupError: eye_pan missing from configs/calibration.json
[FAIL] light ring 0.0s LookupError: buses missing from configs/hardware.json
[FAIL] microphones 0.0s LookupError: audio missing from configs/hardware.json
[SKIP] speaker 0.0s not run: microphones did not pass
[FAIL] arm 0.0s LookupError: arm missing from configs/hardware.json
[SKIP] power 0.0s not run: light ring and arm did not pass
0 passed, 4 failed, 2 not run 0.1s of a 90s budget
NO-GO - unresolved: eye, light ring, microphones, speaker, arm, power
wrote glados/data/hardware_signoff.json
Same command, same six rows, one tenth of a second. Four rows name the section of config they went looking for, and two say plainly that nobody attempted them, because their dependencies had not passed. No traceback, no partial run, and a report on disk either way. A NO-GO report is the more useful of the two: it is the document you open tomorrow to remember what you were in the middle of.
// glados/data/hardware_signoff.json — the good run
{
"started_at": "2026-08-22T19:41:07-05:00",
"body_revision": "1.0.0",
"calibrated_at": "2026-08-22T09:31:55",
"verdict": "GO",
"budget_s": 90.0,
"elapsed_s": 73.4,
"passed": 6,
"failed": 0,
"not_run": 0,
"checks": [
{"name": "eye", "status": "PASS",
"detail": "pan +-25.5 deg, tilt +-17.0 deg, four stops and centre",
"budget_s": 12.0, "elapsed_s": 11.4},
{"name": "power", "status": "PASS",
"detail": "pack 7.92/7.68 V 72%, motion 5.02/4.86 V, floor 4.60 V",
"budget_s": 22.0, "elapsed_s": 17.3}
]
}
Two fields at the top do most of the long-term work. body_revision comes
from the pin map, so the report says which wiring arrangement it describes, and a
report taken against rev 1.0.0 makes no claim about the rev 1.1.0 you build next month.
calibrated_at is the newest measurement date in the calibration file,
because a sign-off is only as true as the numbers the checks ran against. A GO from a
body last calibrated three brackets ago is a GO about a machine that no longer exists.
Commit this file. The diff between two of them is the cheapest regression test this
build will ever have.
Why this works: three states, and an order that earns something
The whole design rests on refusing to let an exception mean anything. A raised exception is a control flow event: it unwinds until something catches it, and if nothing does it takes the process with it. That behaviour is correct for a program doing work and exactly backwards for a program whose only job is to find out what is broken. Catching at the check boundary converts the event into a row, one line down from where it happened, with the message intact. The runner above it never learns that anything went wrong. It counts.
The third state is what makes the count honest. With two states, every question has to be
answered yes or no, so a check that could not run has to lie in one direction or the
other. Call it a failure and you cannot tell a broken subsystem from an absent one, and
people start ignoring red rows. Call it a pass and the report claims something nobody
measured. SKIP lets the run say "I did not look", which is a true statement
about the test, and verdict then refuses to say GO because GO is a statement
about the hardware.
The ordering rule falls out of the same idea. Six checks that assume nothing about each other must each stand alone, and standing alone is expensive: the speaker can only be tested by asking a person, and the rails can only be read at rest. Order them so each check may lean on subsystems already marked PASS, and two of the six get better. When the dependency is not satisfied the check does not guess and does not fail: it skips, and the skip is visible in the verdict. That is how integration suites are ordered in general, and it is chapter 58's discipline applied to seams made of wire.
The arm is off the machine for an evening: a shoulder bracket is back on the printer,
so the four joints are in a box on the desk. Testing the head alone is reasonable, that
is what --skip is for, and the run looks fine. Here is
verdict as it was first written, which is how most people write it:
# labs/hardware_check.py — the version that shipped a false sign-off
def verdict(checks: list[Check]) -> str:
"""BUG: 'no failures' is not 'all passed' once a third status exists."""
return "NO-GO" if any(c.status == FAIL for c in checks) else "GO"
$ uv run python -m labs.signoff --skip arm
GLaDOS body rev 1.0.0, full system test, 6 checks, 90s budget
[PASS] eye 11.2s pan +-25.5 deg, tilt +-17.0 deg, four stops and centre
[PASS] light ring 9.4s 24 pixels through hostile, satisfied, neutral, dark
[PASS] microphones 6.6s index 1, peak 0.2884, rms 0.0771, gate 0.0050
[PASS] speaker 8.9s 440 Hz at 0.70, array heard rms 0.1590, floor 0.0200
[SKIP] arm 0.0s not run: --skip arm
[SKIP] power 0.0s not run: arm did not pass
4 passed, 0 failed, 2 not run 36.1s of a 90s budget
GO - every subsystem answered for itself
wrote glados/data/hardware_signoff.json
Read the last three lines together. Four passed. Two were never run. Verdict: GO. The
counts are right there, one line above a word that contradicts them, and the file on
disk records "verdict": "GO" next to "not_run": 2. Nothing
raised, nothing was measured wrongly, and every individual number in that report is
accurate. Only the conclusion is false.
The path from symptom to cause is short once you look at the right line. The verdict
disagrees with a count computed from the same list, so the bug is in whichever of the
two consults fewer states. any(c.status == FAIL ...) asks about one status
out of three and treats the other two as equivalent, which was true on the day it was
written, when only PASS and FAIL existed. Adding SKIP silently changed what
that line means, and it changed it in the direction that flatters the build.
The fix is all(c.status == PASS ...), and the general form is the part to
keep: derive an approval from the evidence that supports it, never from the absence of
evidence against it. Any function that decides yes or no by searching for a negative has
this bug waiting in it, and it arrives on the day somebody adds a third possibility. The
--skip flag survives the fix unharmed. It still lets you test the head with
the arm in a box; it just can no longer call that a GO.
Checkpoint, and a body that has run out of parts to add
- I can explain why
loadis forbidden to raise whileneedis required to, and what a defaulted calibration would do to the pan servo. - I can name the three statuses, say which two are claims about the hardware, and predict the verdict for any mix of them.
- Given the six checks, I can say which two depend on which others, and what each dependency buys the check that has it.
- I know why the power check keeps the lowest observed sample, and why a passing row cannot exclude millisecond dips between samples.
- I can explain why the speaker probe opens an
InputStreamby hand instead of callingsd.rec, and why its callback copies the buffer. - Handed a report whose verdict disagrees with its own counts, I can find the line that asked about one status out of three.
Exercise 1 — an unattended mode that can never say GO. Add
--unattended so the three watched checks skip instead of prompting, then
run it and read the verdict.
Give watched a module-level switch and have it return a skip rather
than a boolean. The cleanest version raises a small exception the runner already
knows how to turn into a row, but a flag on the probe list is enough:
WATCHED = ("eye", "light ring", "arm")
if args.unattended and check.name in WATCHED:
skip(check, "not run: unattended, nobody watching")
$ uv run python -m labs.signoff --unattended
[SKIP] eye 0.0s not run: unattended, nobody watching
[SKIP] light ring 0.0s not run: unattended, nobody watching
[FAIL] microphones 3.2s index 1, peak 0.0041, rms 0.0009, gate 0.0050
[SKIP] speaker 0.0s not run: microphones did not pass
[SKIP] arm 0.0s not run: unattended, nobody watching
[SKIP] power 0.0s not run: light ring and arm did not pass
0 passed, 1 failed, 5 not run 3.2s of a 90s budget
NO-GO - unresolved: eye, light ring, microphones, speaker, arm, power
An unattended run is now structurally incapable of printing GO, and that is the right outcome: three of these subsystems produce light or motion, and no sensor on this machine observes either. Watch the cascade in the middle as well. Nobody spoke, so the array failed its own gate, and the speaker row skipped behind it without being asked to decide anything. An empty room told the truth six times over.
Exercise 2 — trend two reports. Read the last two sign-off files and print, per check, whether the status changed and how the measured numbers moved.
Keep each run as hardware_signoff_<timestamp>.json, load the two
newest, and compare by name:
for name in {c["name"] for c in old["checks"]} & {c["name"] for c in new["checks"]}:
a = next(c for c in old["checks"] if c["name"] == name)
b = next(c for c in new["checks"] if c["name"] == name)
if a["status"] != b["status"] or a["detail"] != b["detail"]:
print(f"{name:<12} {a['status']} -> {b['status']}\n was: {a['detail']}\n now: {b['detail']}")
$ uv run python -m labs.signoff_trend
power PASS -> PASS
was: pack 7.92/7.68 V 72%, motion 5.02/4.86 V, floor 4.60 V
now: pack 7.61/7.19 V 50%, motion 5.01/4.71 V, floor 4.60 V
Both runs passed, and the second one is a warning anyway. The motion rail lost 0.15 volts of its loaded floor while the pack dropped from seventy two percent to fifty, so the sag is tracking the state of charge and the margin above 4.60 has halved. A pass or fail hides that. The numbers behind it do not, which is the argument for storing the measurement in the row and not only the verdict.
Exercise 3 (stretch) — make her announce her own result. Send the verdict through the voice path, so the last thing volume 7 does is spoken by the body that was just tested.
Compose the sentence from the report, never from a literal, and use the speaker the check just cleared:
report = json.loads(REPORT_PATH.read_text())
line = (f"{report['passed']} of {len(report['checks'])} subsystems nominal. "
f"Verdict: {report['verdict']}.")
speak(line, device=output_index(hw)) # the path volume 1 built
It is the whole volume in one demonstration. The numbers came from probes that touched real hardware, the hardware was calibrated by hand, the voice came from volume 1, and the sound arrives through an amplifier that was measured by her own microphones ninety seconds ago. Record it. That file is the first thing this machine has ever said about itself.
Look at what volume 7 added. A printed shell with a material spec that a validator checks before the slicer runs. A servo you can reason about, down to the pulse width and the deadband around it. An eye that eases through a clamped choke point, and a ring of twenty four pixels driven over one wire without corrupting a colour. A microphone array that resolves by name so a reboot cannot point the pipeline at a speaker, and an amplifier whose array dimensions and sample rate decide whether a sound exists at all. Four arm joints, each with its own measured band, behind one method that clamps and writes. A power budget with two converters, a wire gauge chosen from Ohm's law, and an ADC that turns the rail into a number. One pin map that outranks every constant in the build. A checklist that survived the evening you closed the laptop halfway through assembly. Calibration anchors measured against real stops. And now a run that exercises all of it at once and leaves a dated file saying so.
She physically exists. Six volumes of her lived in files and processes, and the only evidence of any of it was text on your screen. Now there is an object on the desk with a shell, an eye that tracks, a ring that changes colour with her mood, an arm that reaches and holds, ears that hear you from across the room and a voice loud enough to answer over a kitchen. It has a serial number of sorts, a revision, a calibration date and a sign-off. When something in this build says she moves, something in the room moves.
What she does not have yet is a brain that fits her body. Every model in this book has been running on a laptop, on the other end of a cable, or on a Pi that manages a small language model and a whisper transcription only by keeping both of them tiny. The Pi is the right controller for servos, pixels and an ADC, and it is the wrong computer for a mind. Volume 8 moves her onto a Jetson Orin Nano: flashing JetPack and proving CUDA is real, benchmarking the GPU against the numbers she runs at today, running Ollama on hardware that can hold a model several times larger, driving these same pins from a different board, migrating the whole stack across, and setting her up to start on boot and stay up. Same code, same configs, same sign-off run. A bigger mind in the body you just finished testing.