Chapter 43 — Testing the Seams

Every layer passes, and the light stays off

By now each part of her behaves. The transcriber turns a WAV into words. The brain answers. The voice model speaks. The memory store writes a fact and reads it back. The hardware layer toggles a pin and writes down what it expected before it did. Every one of those has been exercised on its own, and every one of them works on its own. Then you assemble the whole assistant, say "turn on the light," and the light stays off. No traceback. Nothing in the log marked ERROR. She answers you politely and the room stays dark.

Here is what actually happened, in exact detail, because most of the bugs left in this project look like this one. The transcriber returned " Turn on the light.\n": a leading space, because that is how the model emits a segment; a capital T; a period; a newline. The command extractor looks its input up in a dictionary keyed by lowercase phrases. Both halves are correct. Both halves would pass any test written against them alone. The lookup returns None, a None command means she treats the sentence as conversation, and the pin is never touched. The defect is not in either module. It sits in the six inches between them, where one module's output becomes another's input.

The obvious way to catch this is to talk to her and watch. That works exactly once, for exactly the sentence you happened to say, and it needs a human, a microphone and a quiet room every time you want to repeat it. So this chapter builds the alternative, and the rule it runs on is this: a seam test is the contract between two modules written as code — hand the first one this input and the second one must produce that output. When a seam test goes red it does not say "the assistant is broken." It names the handoff.

◆ Note — why the base class is not called TestCase

Pytest collects any class whose name starts with Test. A dataclass has a generated __init__, and pytest will not collect a class that takes constructor arguments; it prints PytestCollectionWarning: cannot collect test class 'TestCase' because it has a __init__ constructor and moves on. The warning scrolls past, the class is silently skipped, and one day you wonder why the suite reports fewer tests than you wrote. SeamTest avoids the collision and says what the object actually is.

One seam, then a runner that does not care which is which

▣ Build · stage 1 — write down one handoff and watch it fail

# labs/integration_tests.py
from collections.abc import Callable
from dataclasses import dataclass

COMMANDS = {"turn on the light": "light.on", "turn off the light": "light.off"}


@dataclass
class SeamTest:
    name: str
    description: str

    def run(self) -> tuple[bool, str]:
        raise NotImplementedError(f"{type(self).__name__} must implement run()")


@dataclass
class HeardToCommand(SeamTest):
    transcribe: Callable[[str], str]
    extract: Callable[[str], str | None]
    clip: str
    expect: str

    def run(self) -> tuple[bool, str]:
        heard = self.transcribe(self.clip)
        got = self.extract(heard)
        if got != self.expect:
            return False, f"heard {heard!r}, extractor gave {got!r}, expected {self.expect!r}"
        return True, f"heard {heard!r}, extractor gave {got!r}"


def fake_transcribe(clip: str) -> str:
    return " Turn on the light.\n"


def extract_command(heard: str) -> str | None:
    return COMMANDS.get(heard)


if __name__ == "__main__":
    seam = HeardToCommand(
        "heard_to_command",
        "a transcript becomes the command it names",
        fake_transcribe,
        extract_command,
        "glados/data/captured_audio.wav",
        "light.on",
    )
    passed, message = seam.run()
    print(f"  [{'PASS' if passed else 'FAIL'}] {seam.name}: {message}")

$ uv run python labs/integration_tests.py
  [FAIL] heard_to_command: heard ' Turn on the light.\n', extractor gave None, expected 'light.on'

The first seam test written for this book finds a real bug in one second, and the message contains the whole diagnosis: what came out of the first module, what the second one made of it, what you said it should have made of it. Two details in the code are doing quiet work. The transcriber arrives as a field, so the same test object runs against a canned string here and against the real model later without a line changing. And HeardToCommand is itself a dataclass, so it inherits the base's fields and appends its own: the constructor takes name, description, then the four fields declared here, in that order. The fix is one line, COMMANDS.get(heard.strip().lower().rstrip(".")), and the same run then reads:

$ uv run python labs/integration_tests.py
  [PASS] heard_to_command: heard ' Turn on the light.\n', extractor gave 'light.on'

▣ Build · stage 2 — two more seams and one loop over all of them

@dataclass
class ReplyToSpeech(SeamTest):
    reply: str
    to_speech: Callable[[str], str]

    def run(self) -> tuple[bool, str]:
        spoken = self.to_speech(self.reply)
        if not spoken.strip():
            return False, "normalizer emptied a reply that had words in it"
        markup = sorted({c for c in spoken if c in "*#`_"})
        if markup:
            return False, f"markup reached the voice model: {markup}"
        return True, f"{len(spoken)} speakable characters, no markup"


@dataclass
class MemoryRoundTrip(SeamTest):
    db_path: Path
    key: str
    value: str

    def run(self) -> tuple[bool, str]:
        self.db_path.parent.mkdir(parents=True, exist_ok=True)
        self.db_path.unlink(missing_ok=True)
        con = sqlite3.connect(self.db_path)
        con.execute("CREATE TABLE facts (key TEXT PRIMARY KEY, value TEXT)")
        con.execute("INSERT INTO facts VALUES (?, ?)", (self.key, self.value))
        con.commit()
        con.close()

        con = sqlite3.connect(self.db_path)
        row = con.execute("SELECT value FROM facts WHERE key = ?", (self.key,)).fetchone()
        con.close()
        if row is None:
            return False, f"{self.key!r} was gone after the reopen"
        if row[0] != self.value:
            return False, f"{self.key!r} came back as {row[0]!r}"
        return True, f"{self.key}={row[0]!r} survived close and reopen"


class IntegrationRunner:
    def __init__(self, tests: list[SeamTest]) -> None:
        self.tests = tests
        self.results: list[dict] = []

    def run_all(self) -> bool:
        for test in self.tests:
            passed, message = test.run()
            self.results.append({
                "name": test.name,
                "description": test.description,
                "passed": passed,
                "message": message,
            })
            print(f"  [{'PASS' if passed else 'FAIL'}] {test.name}: {message}")
        return all(r["passed"] for r in self.results)

$ uv run python labs/integration_tests.py
  [PASS] heard_to_command: heard ' Turn on the light.\n', extractor gave 'light.on'
  [PASS] reply_to_speech: 35 speakable characters, no markup
  [PASS] memory_round_trip: last_calibration='servo 2 at 88 degrees' survived close and reopen
All passed: True

Three handoffs, three classes, and a runner that treats them identically. Look at what run_all knows about its tests: their name, and that calling run() gives back a boolean and a string. Nothing else. It never asks which kind of seam it is holding, so a fourth kind is a new class and a list entry, with no edit to the loop. The memory case earns its place by being real rather than mimed: it writes to an actual SQLite file, closes the connection, opens a fresh one, and reads the row back, which is the only version of that test that would have caught a fact living in a transaction nobody committed.

▣ Build · stage 3 — a report the next run can be compared against

    def save_report(self, path: Path) -> None:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(self.results, indent=2))


def main() -> None:
    runner = IntegrationRunner(build_suite(fake_transcribe))
    all_passed = runner.run_all()
    runner.save_report(REPORT_PATH)
    print(f"Ran {len(runner.results)} seams. All passed: {all_passed}")

$ uv run python labs/integration_tests.py
  [PASS] heard_to_command: heard ' Turn on the light.\n', extractor gave 'light.on'
  [PASS] reply_to_speech: 35 speakable characters, no markup
  [PASS] memory_round_trip: last_calibration='servo 2 at 88 degrees' survived close and reopen
Ran 3 seams. All passed: True

$ head -7 glados/data/integration_report.json
[
  {
    "name": "heard_to_command",
    "description": "a transcript becomes the command it names",
    "passed": true,
    "message": "heard ' Turn on the light.\\n', extractor gave 'light.on'"
  },

The terminal output is for you, now. The JSON is for the version of you that runs this in three weeks and wants to know what changed, and for anything that reads results without a human in front of it. Storing description alongside the verdict is what makes the file readable months later: a bare "heard_to_command": false needs an archaeologist, while the sentence next to it states the contract that broke. mkdir(parents=True, exist_ok=True) means the first run on a fresh checkout writes the report instead of dying on a missing directory, and write_text truncates, so the file always describes the latest run and never accumulates stale entries.

A green suite that checked nothing

⚠ Worked failure — zero tests, and the gate says everything passed

You are about to touch the bench, so you want the hardware seams only. A filter on the name is the natural way to get them:

# run only the hardware seams before touching the bench
selected = [t for t in build_suite(fake_transcribe) if t.name.startswith("hardware_")]

runner = IntegrationRunner(selected)
all_passed = runner.run_all()
print(f"Ran {len(runner.results)} seams. All passed: {all_passed}")

$ uv run python labs/integration_tests.py
Ran 0 seams. All passed: True

No error. No warning. A confident green light over a suite that verified nothing at all, which is the most dangerous kind of pass because it wears the same face as a real one. Trace it back: no seam in the suite is named hardware_ anything, so selected is empty, so the loop body never executes, so self.results stays empty, and all([]) is True. That last one is not a Python quirk to be angry at. all() answers "did every element satisfy this?", and with no elements the answer is honestly yes; change it and all(x > 0 for x in []) inside every other program on your machine starts lying. The bug is the gate, which asked the wrong question. It wanted "did every seam pass?" when it should have wanted "did seams run, and did every one of them pass?" One bool() at the front of the return fixes it, and the same filter now prints Ran 0 seams. All passed: False.

▣ Build · stage 4 — the assembled suite

# labs/integration_tests.py — full file
import json
import sqlite3
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path

REPORT_PATH = Path("glados/data/integration_report.json")
SUITE_DB = Path("glados/data/suite_memory.db")

COMMANDS = {"turn on the light": "light.on", "turn off the light": "light.off"}


@dataclass
class SeamTest:
    name: str
    description: str

    def run(self) -> tuple[bool, str]:
        raise NotImplementedError(f"{type(self).__name__} must implement run()")


@dataclass
class HeardToCommand(SeamTest):
    transcribe: Callable[[str], str]
    extract: Callable[[str], str | None]
    clip: str
    expect: str

    def run(self) -> tuple[bool, str]:
        heard = self.transcribe(self.clip)
        got = self.extract(heard)
        if got != self.expect:
            return False, f"heard {heard!r}, extractor gave {got!r}, expected {self.expect!r}"
        return True, f"heard {heard!r}, extractor gave {got!r}"


@dataclass
class ReplyToSpeech(SeamTest):
    reply: str
    to_speech: Callable[[str], str]

    def run(self) -> tuple[bool, str]:
        spoken = self.to_speech(self.reply)
        if not spoken.strip():
            return False, "normalizer emptied a reply that had words in it"
        markup = sorted({c for c in spoken if c in "*#`_"})
        if markup:
            return False, f"markup reached the voice model: {markup}"
        return True, f"{len(spoken)} speakable characters, no markup"


@dataclass
class MemoryRoundTrip(SeamTest):
    db_path: Path
    key: str
    value: str

    def run(self) -> tuple[bool, str]:
        self.db_path.parent.mkdir(parents=True, exist_ok=True)
        self.db_path.unlink(missing_ok=True)
        con = sqlite3.connect(self.db_path)
        con.execute("CREATE TABLE facts (key TEXT PRIMARY KEY, value TEXT)")
        con.execute("INSERT INTO facts VALUES (?, ?)", (self.key, self.value))
        con.commit()
        con.close()

        con = sqlite3.connect(self.db_path)
        row = con.execute("SELECT value FROM facts WHERE key = ?", (self.key,)).fetchone()
        con.close()
        if row is None:
            return False, f"{self.key!r} was gone after the reopen"
        if row[0] != self.value:
            return False, f"{self.key!r} came back as {row[0]!r}"
        return True, f"{self.key}={row[0]!r} survived close and reopen"


class IntegrationRunner:
    def __init__(self, tests: list[SeamTest]) -> None:
        self.tests = tests
        self.results: list[dict] = []

    def run_all(self) -> bool:
        for test in self.tests:
            passed, message = test.run()
            self.results.append({
                "name": test.name,
                "description": test.description,
                "passed": passed,
                "message": message,
            })
            print(f"  [{'PASS' if passed else 'FAIL'}] {test.name}: {message}")
        return bool(self.results) and all(r["passed"] for r in self.results)

    def save_report(self, path: Path) -> None:
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(json.dumps(self.results, indent=2))


def fake_transcribe(clip: str) -> str:
    return " Turn on the light.\n"


def extract_command(heard: str) -> str | None:
    return COMMANDS.get(heard.strip().lower().rstrip("."))


def to_speech(reply: str) -> str:
    text = reply.replace("**", "").replace("*", "").replace("`", "")
    return " ".join(text.split())


def build_suite(transcribe: Callable[[str], str]) -> list[SeamTest]:
    return [
        HeardToCommand(
            "heard_to_command",
            "a transcript becomes the command it names",
            transcribe,
            extract_command,
            "glados/data/captured_audio.wav",
            "light.on",
        ),
        ReplyToSpeech(
            "reply_to_speech",
            "an LLM reply reaches the voice model without markup",
            "**Oh**, it's *you* again. `light.on` it is.",
            to_speech,
        ),
        MemoryRoundTrip(
            "memory_round_trip",
            "a stored fact survives closing the database",
            SUITE_DB,
            "last_calibration",
            "servo 2 at 88 degrees",
        ),
    ]


def main() -> None:
    runner = IntegrationRunner(build_suite(fake_transcribe))
    all_passed = runner.run_all()
    runner.save_report(REPORT_PATH)
    print(f"Ran {len(runner.results)} seams. All passed: {all_passed}")


if __name__ == "__main__":
    main()

build_suite takes the transcriber and nothing else, which is the seam between fast and slow. Called with fake_transcribe, the whole suite finishes in well under a second and you can run it after every edit. Called with the real transcriber pointed at a committed WAV, the same three contracts get checked against the actual model on the actual machine; that run is slower and its first message will quote whatever the model heard, so your line will read differently from the one above. The suite is identical either way. Only the collaborator changed.

Why this works: one loop, any number of contracts

Everything here rests on the runner refusing to know what it is running. It calls test.run() and relies on the promise that every subclass gives back a boolean and a string. That promise is the substitution principle from the object-oriented literature, doing plain work in twelve lines: any SeamTest subclass can stand in for any other wherever a SeamTest is expected. The alternative is a function per seam and a hand-written sequence of calls, and that version grows a line in the runner for every contract you add, which is how test suites get abandoned.

The base run() that raises is the enforcement half. Write a subclass, forget the method, and the inherited version fires with the class name baked into the message: NotImplementedError: WakeWordSeam must implement run(). Loud, immediate, and it names the file to open. A silent default that returned True would be the same species of lie as the empty suite.

There is a second reason the seams are classes and not functions, and it shows up the moment you have ten of them. HeardToCommand describes a kind of handoff; each instance describes one case of it. Ten sentences you want her to understand cost ten list entries and zero new code, each carrying its own clip and its own expected command. That split, one class per kind of contract and one instance per case, is what keeps an integration suite from turning into a thousand lines of nearly identical functions.

Checkpoint, and the interface that will not be assumed

✓ Checkpoint — what you can now do

Shown a system where every module's own tests pass and the behaviour is still wrong, I can name where the defect must be and write the test that proves it.
I can say what a dataclass subclass inherits from its base and what order the generated constructor takes the combined fields in.
I can explain why all([]) is True, why that is the right answer for all(), and why it is the wrong answer for a test gate.
I can add a fourth kind of seam without editing run_all, and state exactly what the runner relies on for that to keep working.
I can point at the line that makes this suite runnable in a second against fakes and on the bench against real models, without a second copy of the tests.

⚡ Exercises — try first, then reveal

Exercise 1 — break it on purpose. Put extract_command back to the plain COMMANDS.get(heard), run the suite, and predict both the terminal output and what integration_report.json holds afterwards.

The first line flips to [FAIL] heard_to_command: heard ' Turn on the light.\n', extractor gave None, expected 'light.on', the other two seams still pass because the runner keeps going after a failure, and the last line reads All passed: False. The report holds all three entries with "passed": false on the first, and its message is the diagnosis in text: the two strings that failed to match, side by side. Then put the fix back and watch the same file go green.

Exercise 2 — make the shell care. Add a summary() method that returns how many seams passed out of how many ran, and have main exit with a status code so a script can branch on the result.

summary() is two lines: passed = sum(1 for r in self.results if r["passed"]), then return f"{passed}/{len(self.results)} seams passed". For the exit code, end main with raise SystemExit(0 if all_passed else 1). Run it with a broken seam and echo $? prints 1; fix the seam and it prints 0. That single number is the whole point of a gate: something other than a human can now read the verdict, which is what makes it possible to refuse a deploy or refuse to power the servo rail.

Exercise 3 — run it against the real model. Record a short clip of yourself saying "turn on the light," commit it under glados/data/, and pass your real transcriber into build_suite instead of fake_transcribe.

The suite goes from under a second to a few seconds, and the first message now quotes what the model actually heard from your voice and your room, so expect something close to but not identical to the canned string. If it passes, the contract holds against the real pipeline. If it fails on punctuation or a politeness word the model added, you have found the next normalization rule your extractor needs, which is precisely the class of bug this chapter exists to catch. Keep the clip in the repository: a recorded input is what makes a slow test repeatable.

The suite trusts something it has no way to check. build_suite accepts any object with the right method name floating around on it, and finds out whether the transcriber really has one only when the test calls it, mid-run. Next comes the stricter version of that promise: STT, LLM and TTS contracts declared as abstract base classes, so a component missing a method is refused at the moment you try to build it.