Chapter 59 — Sign-Off Is a Number

Six subsystems, all green, and no answer

This volume has been about the parts of shipping that are invisible from inside the code: a scoring function that ranks what to build next, a permission table that refuses by default, one failure behaviour shared by every driver, a service unit regenerated from configuration, a wiring manifest that validates itself, a manual generated from the same registry the dispatcher reads. Every check passes. She wakes on her name, answers, moves the arm. And there is still nothing on disk that answers the question anybody will actually ask you: is she finished?

The usual answer is a demo. You run her five or six times in front of someone, it goes well, you call it done. Two bills come due later. Six weeks on, when a housemate asks whether the wake word is reliable, the honest reply is "it felt quick", which reports your mood on the afternoon you tested. Worse is what happens the first time a number disappoints you: a word error rate of fourteen percent becomes "close enough for a house" precisely because you found out it was fourteen. A standard agreed after the result is the result in a standard's clothing.

So the rule this chapter builds and then obeys: acceptance is a numbered list of criteria, each carrying a limit, a direction and a flag saying whether it blocks; the whole list is fixed before the first measurement, and the verdict is computed from it. Chapter 42 did this for one servo, writing the expected value and the tolerance down before touching the protractor. The same discipline over an assembled system needs two more things: a score that can be compared across runs, and a fingerprint of the criteria themselves, so that later you can show the bar stayed where you put it.

◆ Note — what is on the bench, and how to read the numbers below

One laptop running her the way volume 4 assembled her, an ESP32 on the USB link from chapter 40, and the SG90 servo carrying the arm. Every measured value printed in this chapter came off that bench on one afternoon with those parts and those models. Latency depends on your CPU, error rate on your microphone and your accent, and arm travel on the individual servo in your hand. Yours will land somewhere else. Nothing here asks you to reproduce these numbers; it asks you to produce your own and write them next to the limits you set in advance.

A criterion is a limit with a direction

▣ Build · stage 1 — one criterion, with the verdict derived

# labs/acceptance.py
from dataclasses import dataclass

DIRECTIONS = {"<=": lambda measured, limit: measured <= limit,
              ">=": lambda measured, limit: measured >= limit}


@dataclass
class Criterion:
    id: str
    statement: str
    unit: str
    limit: float
    direction: str
    blocking: bool = True
    measured: float | None = None
    note: str = ""

    def __post_init__(self) -> None:
        if self.direction not in DIRECTIONS:
            raise ValueError(f"{self.id}: direction must be one of {sorted(DIRECTIONS)}")

    @property
    def passed(self) -> bool:
        if self.measured is None:
            return False
        return DIRECTIONS[self.direction](self.measured, self.limit)


if __name__ == "__main__":
    ac = Criterion("AC-02", "transcription word error rate", "%", 10.0, "<=")
    print(ac.id, ac.statement, ac.direction, ac.limit, ac.unit)
    print("  before measuring:", ac.passed)
    ac.measured = 11.4
    print("  at 11.4 percent: ", ac.passed)
    ac.measured = 6.9
    print("  at 6.9 percent:  ", ac.passed)

$ uv run python -m labs.acceptance
AC-02 transcription word error rate <= 10.0 %
  before measuring: False
  at 11.4 percent:  False
  at 6.9 percent:   True

Half of what you want to promise about an assistant gets better as the number falls: latency, error rate, failures per hundred replies. The other half gets better as it climbs: facts recalled, commands matched, patterns refused. Storing the comparison as a string in direction keeps both kinds in one list, and __post_init__ is what makes that safe, because a typo like "=<" raises the moment the plan is built instead of quietly answering False forever. passed stays a computed property for the reason chapter 42 gave it one: a stored boolean can drift away from the numbers that produced it, and a criterion whose measured is still None has to report failure, because unmeasured and passing must never print the same word.

▣ Build · stage 2 — the plan, written before anything is measured

import hashlib
import json


def acceptance_plan() -> list[Criterion]:
    return [
        Criterion("AC-01", "wake word to first spoken word", "ms", 2000.0, "<="),
        Criterion("AC-02", "transcription word error rate", "%", 10.0, "<="),
        Criterion("AC-03", "replies rated in character", "of 5", 4.0, ">=", blocking=False),
        Criterion("AC-04", "synthesis failures in 20 replies", "failures", 0.0, "<="),
        Criterion("AC-05", "facts recalled after a restart", "facts", 12.0, ">="),
        Criterion("AC-06", "commands reaching the right handler", "commands", 8.0, ">="),
        Criterion("AC-07", "blocked patterns refused", "patterns", 6.0, ">="),
        Criterion("AC-08", "arm travel between its stops", "deg", 160.0, ">="),
    ]


def plan_id(criteria: list[Criterion]) -> str:
    bar = [[c.id, c.limit, c.direction, c.blocking] for c in criteria]
    return hashlib.sha256(json.dumps(bar).encode()).hexdigest()[:12]


def print_plan(criteria: list[Criterion]) -> None:
    print(f"Acceptance plan {plan_id(criteria)}")
    for c in criteria:
        rule = "blocking" if c.blocking else "exception allowed"
        print(f"  {c.id}  {c.statement:<36}{c.direction} {c.limit:<8.1f}{c.unit:<10}{rule}")


if __name__ == "__main__":
    plan = acceptance_plan()
    print_plan(plan)
    met = sum(1 for c in plan if c.passed)
    print(f"  {met} of {len(plan)} criteria met before anything is measured")

$ uv run python -m labs.acceptance
Acceptance plan c28f93f99a1b
  AC-01  wake word to first spoken word      <= 2000.0  ms        blocking
  AC-02  transcription word error rate       <= 10.0    %         blocking
  AC-03  replies rated in character          >= 4.0     of 5      exception allowed
  AC-04  synthesis failures in 20 replies    <= 0.0     failures  blocking
  AC-05  facts recalled after a restart      >= 12.0    facts     blocking
  AC-06  commands reaching the right handler >= 8.0     commands  blocking
  AC-07  blocked patterns refused            >= 6.0     patterns  blocking
  AC-08  arm travel between its stops        >= 160.0   deg       blocking
  0 of 8 criteria met before anything is measured

Read those eight lines as a document: the promise you are making about the machine, in numbers, on one screen, reviewable by somebody who has never seen the code. Two of the choices deserve an argument now, while arguing is free. AC-03 is the only criterion with a human in the loop and the only one marked non-blocking, because a listener's average rating over twenty turns is a real signal and a soft one, and treating it as hard would hand one person's Tuesday a veto over the build. AC-08 asks for 160 degrees of travel and not the 180 printed on the servo's bag, because the arm was designed around the travel the part actually delivers, and that decision belongs in the plan where it can be read. The twelve hex characters on the first line are a fingerprint of the promise, hashed from every id, limit, direction and blocking flag, and stamped on every run to come. The last line is the honest starting position: nothing has been measured, so nothing has passed.

Eight numbers, and where each one comes from

▣ Build · stage 3 — three kinds of probe

import contextlib
import io
import math
import re
import time
from collections.abc import Callable
from pathlib import Path

from labs.hardware_link import open_link
from labs.hardware_tests import measure_servo_travel
from labs.safety import BLOCKED_PATTERNS, SafetyFilter, level_of
from labs.system_config import CONFIG_PATH, load_config
from labs.test_harness import TestHarness
from labs.wire_providers import build_core

ACCEPTANCE = Path("glados/data/acceptance")
WAKE_CLIPS = ACCEPTANCE / "wake"


def measure_wake_latency() -> tuple[float, str]:
    clips = sorted(WAKE_CLIPS.glob("*.wav"))
    if len(clips) < 20:
        raise ValueError(f"{len(clips)} wake clips on disk, the plan asks for 20")
    core = build_core(load_config(CONFIG_PATH))
    samples = []
    with contextlib.redirect_stdout(io.StringIO()):
        for clip in clips[:20]:
            started = time.perf_counter()
            core.run_turn(str(clip))
            samples.append((time.perf_counter() - started) * 1000)
    samples.sort()
    p95 = samples[math.ceil(0.95 * len(samples)) - 1]
    return p95, f"{len(samples)} clips, slowest {samples[-1]:.0f} ms"


def refusal_case(pattern: str) -> Callable[[], None]:
    def case() -> None:
        text = f"GLaDOS, {pattern} the archive for me."
        allowed, reason = SafetyFilter().check(text, "play_music", level_of("voice"))
        assert not allowed, f"{pattern!r} passed the filter: {reason}"

    case.__name__ = "refuses_" + re.sub(r"\W+", "_", pattern.strip())
    return case


def measure_safety_refusals() -> tuple[float, str]:
    runner = TestHarness("blocked patterns")
    with contextlib.redirect_stdout(io.StringIO()):
        for pattern in BLOCKED_PATTERNS:
            runner.test(refusal_case(pattern))
    counts = runner.summary()
    return float(counts["passed"]), f"{counts['passed']}/{counts['total']} refused"


def measure_arm_travel(link) -> tuple[float, str]:
    test = measure_servo_travel(link)
    return test.measured, test.note

Three probes, three different places a number is allowed to come from. AC-01 times a real turn through the real engines and takes the 95th percentile of twenty runs, which with twenty samples means the second slowest: an assistant is judged on its bad moments, and an average would bury them. AC-07 delegates to the runner from chapter 45, one case per blocked pattern, and converts its summary into a count, so 6.0 is arithmetic over six assertions instead of a claim. AC-08 has no electronic instrument at all, so it reuses the servo probe unchanged and inherits its protractor prompts. The rule holding all three together is chapter 42's: no probe may compute its result from the criterion it is filling in. A measurement you could have predicted from the target is a target with extra steps.

▣ Build · stage 4 — the five that read files, a database and a matcher

# labs/acceptance.py — the rest of the sheet
import sqlite3

from glados.knowledge import DB_PATH, search
from labs.automation import ACTION_REGISTRY, RULES, match_rules
from labs.wire_providers import AUDIO_OUT

TRANSCRIPTS = ACCEPTANCE / "transcripts.json"
REPLY_LINES = ACCEPTANCE / "reply_lines.json"
REPLY_RATINGS = ACCEPTANCE / "reply_ratings.json"
RECALL_TOPICS = ACCEPTANCE / "recall_topics.json"
SPOKEN = Path(AUDIO_OUT)

# Eight utterances and the handler each one is supposed to reach.
DISPATCH_CASES = [
    ("GLaDOS, lights off please.", "turn_off_lights"),
    ("Turn the lights off in here.", "turn_off_lights"),
    ("Set timer for five minutes.", "set_timer"),
    ("GLaDOS, set timer, ten minutes.", "set_timer"),
    ("Play music, something loud.", "play_music"),
    ("Would you play music for once?", "play_music"),
    ("Lights off, then play music.", "turn_off_lights"),
    ("Set timer while you play music.", "set_timer"),
]


def measure_word_error_rate() -> tuple[float, str]:
    references = json.loads(TRANSCRIPTS.read_text())
    core = build_core(load_config(CONFIG_PATH))
    wrong = total = 0
    with contextlib.redirect_stdout(io.StringIO()):
        for clip, said in references.items():
            expected = said.lower().split()
            heard = core.process(str(ACCEPTANCE / clip)).lower().split()
            total += len(expected)
            wrong += sum(1 for i, word in enumerate(expected)
                         if i >= len(heard) or heard[i].strip(".,?!") != word.strip(".,?!"))
            wrong += max(0, len(heard) - len(expected))
    if total == 0:
        raise ValueError(f"{TRANSCRIPTS} holds no reference words")
    return 100.0 * wrong / total, f"{wrong} wrong of {total} words"


def rate_replies_by_ear() -> tuple[float, str]:
    ratings = json.loads(REPLY_RATINGS.read_text())["ratings"]
    if len(ratings) < 20:
        raise ValueError(f"{len(ratings)} ratings on file, the plan asks for 20")
    below = sum(1 for r in ratings if r < 4)
    return sum(ratings) / len(ratings), f"{len(ratings)} turns, {below} rated below 4"


def measure_synthesis_failures() -> tuple[float, str]:
    lines = json.loads(REPLY_LINES.read_text())["replies"]
    core = build_core(load_config(CONFIG_PATH))
    written = 0
    with contextlib.redirect_stdout(io.StringIO()):
        for line in lines:
            SPOKEN.unlink(missing_ok=True)
            try:
                core.speak(line)
            except Exception:
                continue
            if SPOKEN.exists() and SPOKEN.stat().st_size > 0:
                written += 1
    return float(len(lines) - written), f"{written}/{len(lines)} files written"


def measure_recall_after_restart() -> tuple[float, str]:
    topics = json.loads(RECALL_TOPICS.read_text())["topics"]
    conn = sqlite3.connect(DB_PATH)          # a new connection is the restart
    try:
        found = sum(1 for topic in topics if search(conn, [topic], limit=1))
    finally:
        conn.close()
    return float(found), f"{found}/{len(topics)} after restart"


def measure_command_dispatch() -> tuple[float, str]:
    matched = 0
    for said, expected in DISPATCH_CASES:
        hits = match_rules(said, RULES)
        reached = hits[0]["action"] if hits else None
        if reached == expected and reached in ACTION_REGISTRY:
            matched += 1
    return float(matched), f"{matched}/{len(DISPATCH_CASES)} matched"

Five probes, and not one of them invents its own number. AC-02 pushes recorded clips back through the same transcription path a live turn uses and counts word-level disagreements against a reference file you typed once, in both directions, so an invented extra word costs as much as a missing one. AC-03 measures nothing at all: it averages a ratings file a person filled in, because there is no instrument for whether a reply sounded like her. AC-04 deletes the output file before every line and then asks whether a non-empty one appeared, which is the only definition of "the voice worked" that does not take the engine's word for it. AC-05 opens a fresh connection to the database and looks for topics she was told about days ago, since a new connection is what a restart looks like from the data's side. AC-06 walks eight utterances past the matcher and counts only those that reach the expected handler and find it registered, so a rule that matches into an empty registry scores nothing. Two of these load real engines and take minutes; three read a file and finish instantly. The plan does not care which, and neither does the report.

▣ Build · stage 5 — the runner, and a broken probe that stops nothing

def probes(link) -> dict[str, Callable[[], tuple[float, str]]]:
    return {
        "AC-01": measure_wake_latency,
        "AC-02": measure_word_error_rate,
        "AC-03": rate_replies_by_ear,
        "AC-04": measure_synthesis_failures,
        "AC-05": measure_recall_after_restart,
        "AC-06": measure_command_dispatch,
        "AC-07": measure_safety_refusals,
        "AC-08": lambda: measure_arm_travel(link),
    }


def run_plan(criteria: list[Criterion],
             registered: dict[str, Callable[[], tuple[float, str]]]) -> None:
    for c in criteria:
        probe = registered.get(c.id)
        if probe is None:
            c.note = "no measurement registered"
            continue
        try:
            c.measured, c.note = probe()
        except Exception as exc:
            c.measured, c.note = None, f"measurement error: {exc}"

The dictionary is the seam. Criteria are argued over and then frozen; probes get rewritten as a subsystem matures, and swapping one is an edit to a single value here. Building the dictionary inside a function lets the hardware probe close over an open serial link while the software probes take no arguments, so every entry has the same callable signature no matter what it needs underneath. The except is the same trade chapter 45 made in its own runner: a probe that raises becomes a recorded failure instead of a stack trace that abandons the other seven measurements. Note the assignment order in that branch. Setting measured back to None is not tidiness, it is the difference between a criterion that failed to measure and a criterion that measured well, and the next stage is where forgetting it would cost you.

The verdict, and a run that refuses to round up

▣ Build · stage 6 — score it, name what failed, and say the word

def verdict(criteria: list[Criterion]) -> str:
    unmet = [c for c in criteria if not c.passed]
    if any(c.blocking for c in unmet):
        return "NOT ACCEPTED"
    if unmet:
        return f"ACCEPTED WITH {len(unmet)} EXCEPTION" + ("S" if len(unmet) > 1 else "")
    return "ACCEPTED"


def report(criteria: list[Criterion]) -> None:
    met = sum(1 for c in criteria if c.passed)
    print(f"\nAcceptance run against plan {plan_id(criteria)}")
    for c in criteria:
        flag = "PASS" if c.passed else "FAIL"
        shown = "not measured" if c.measured is None else f"{c.measured:.1f}"
        print(f"  [{flag}] {c.id}  {c.statement:<36}{shown:>12} {c.unit:<10}"
              f"{c.direction} {c.limit:<8.1f}{c.note}")
    print(f"\n  Score: {met} of {len(criteria)} criteria met "
          f"({100 * met / len(criteria):.1f}%)")
    for c in criteria:
        if not c.passed:
            rule = "blocking" if c.blocking else "exception allowed"
            print(f"  Unmet: {c.id} ({rule})")
    print(f"  Verdict: {verdict(criteria)}")

$ uv run python -m labs.acceptance   # measured on the bench, yours will vary
Measuring 8 criteria. AC-08 will ask you for a protractor reading.
  protractor reading at commanded 0 deg: 6
  protractor reading at commanded 180 deg: 173

Acceptance run against plan c28f93f99a1b
  [PASS] AC-01  wake word to first spoken word            1420.0 ms        <= 2000.0  20 clips, slowest 1655 ms
  [FAIL] AC-02  transcription word error rate               11.4 %         <= 10.0    8 wrong of 70 words
  [FAIL] AC-03  replies rated in character                   3.8 of 5      >= 4.0     20 turns, 3 rated below 4
  [PASS] AC-04  synthesis failures in 20 replies             0.0 failures  <= 0.0     20/20 files written
  [PASS] AC-05  facts recalled after a restart              12.0 facts     >= 12.0    12/12 after restart
  [PASS] AC-06  commands reaching the right handler          8.0 commands  >= 8.0     8/8 matched
  [PASS] AC-07  blocked patterns refused                     6.0 patterns  >= 6.0     6/6 refused
  [PASS] AC-08  arm travel between its stops               167.0 deg       >= 160.0   6 to 173 deg

  Score: 6 of 8 criteria met (75.0%)
  Unmet: AC-02 (blocking)
  Unmet: AC-03 (exception allowed)
  Verdict: NOT ACCEPTED

Six of eight, and the machine says NOT ACCEPTED while you stand there watching it work beautifully. That gap is the point of the exercise. AC-02 missed by 1.4 percentage points, which is exactly the size of gap that talks you into rounding, because the demo felt fine and the deadline is Thursday. The report gives that temptation nothing to hold: the limit was written before the reading, the reading is printed beside it, and the verdict is a function of the two. The score line prints one decimal on purpose, since :.0% would turn seven of eight into a tidy 88 percent and lose the half a reader deserves to see.

▣ Build · stage 7 — an artifact per run, and the run after the fix

from datetime import datetime

REPORT = Path("docs/acceptance.json")
BENCH = "laptop CPU, ESP32 DevKit v1, SG90 arm servo"


def save_run(criteria: list[Criterion], path: Path = REPORT, bench: str = BENCH) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    runs = json.loads(path.read_text()) if path.exists() else []
    met = sum(1 for c in criteria if c.passed)
    runs.append({
        "recorded_at": datetime.now().isoformat(timespec="seconds"),
        "plan": plan_id(criteria),
        "bench": bench,
        "score": {"met": met, "total": len(criteria), "verdict": verdict(criteria)},
        "criteria": [
            {"id": c.id, "statement": c.statement, "unit": c.unit, "limit": c.limit,
             "direction": c.direction, "blocking": c.blocking,
             "measured": c.measured, "passed": c.passed, "note": c.note}
            for c in criteria
        ],
    })
    path.write_text(json.dumps(runs, indent=2))


def main() -> None:
    criteria = acceptance_plan()
    print(f"Measuring {len(criteria)} criteria. AC-08 will ask you for a protractor reading.")
    with open_link() as link:
        run_plan(criteria, probes(link))
    report(criteria)
    save_run(criteria)
    print(f"Run appended to {REPORT}")


if __name__ == "__main__":
    main()

$ uv run python -m labs.acceptance   # after switching the transcription model
Acceptance run against plan c28f93f99a1b
  [PASS] AC-01  wake word to first spoken word            1980.0 ms        <= 2000.0  20 clips, slowest 2240 ms
  [PASS] AC-02  transcription word error rate                6.9 %         <= 10.0    5 wrong of 70 words
  [FAIL] AC-03  replies rated in character                   3.8 of 5      >= 4.0     20 turns, 3 rated below 4
  [PASS] AC-04  synthesis failures in 20 replies             0.0 failures  <= 0.0     20/20 files written
  [PASS] AC-05  facts recalled after a restart              12.0 facts     >= 12.0    12/12 after restart
  [PASS] AC-06  commands reaching the right handler          8.0 commands  >= 8.0     8/8 matched
  [PASS] AC-07  blocked patterns refused                     6.0 patterns  >= 6.0     6/6 refused
  [PASS] AC-08  arm travel between its stops               167.0 deg       >= 160.0   6 to 173 deg

  Score: 7 of 8 criteria met (87.5%)
  Unmet: AC-03 (exception allowed)
  Verdict: ACCEPTED WITH 1 EXCEPTION

One edit produced that: the transcription component in the config moved from the small model to the next size up, which is a value change in one file and no code at all. The error rate came down to 6.9 percent, and the tail went up by more than half a second, from 1420 to 1980 milliseconds against a limit of 2000. Both numbers are on the same sheet, so the trade is visible instead of felt, and AC-01 is now close enough to its limit that it belongs on the list of things to watch. The verdict names its own remaining hole. She is accepted with one exception, and the exception is printed every single time the report runs, so nobody has to remember it and nobody gets to call this eight of eight.

[
  {
    "recorded_at": "2026-08-22T16:41:07",
    "plan": "c28f93f99a1b",
    "bench": "laptop CPU, ESP32 DevKit v1, SG90 arm servo",
    "score": { "met": 7, "total": 8, "verdict": "ACCEPTED WITH 1 EXCEPTION" },
    "criteria": [
      {
        "id": "AC-02",
        "statement": "transcription word error rate",
        "unit": "%",
        "limit": 10.0,
        "direction": "<=",
        "blocking": true,
        "measured": 6.9,
        "passed": true,
        "note": "5 wrong of 70 words"
      }
    ]
  }
]

Both runs carry the fingerprint c28f93f99a1b, and that string is the quietest important thing in the file. It hashes only the parts of the plan that constitute the promise: every id, limit, direction and blocking flag. Change a statement's wording and it holds steady. Loosen AC-02 to 12 percent and it changes, which turns the oldest trick in engineering acceptance into a visible edit. Two runs with the same fingerprint are comparable; two runs with different ones measured different promises, whatever the score says.

Why this works: the order of two writes

Everything the plan protects comes from one piece of sequencing. The limit is written to the file at time A, the measurement at time B, and A comes before B. Reverse that order and every part of the machinery still runs: the dataclass holds the same fields, the report prints the same columns, the verdict function returns the same strings. What disappears is the only thing that made the output evidence. A limit chosen after the reading encodes what the system did, and a report full of those is an elaborate way of writing down what happened and calling it a standard.

The blocking flag is what keeps that discipline survivable. A plan where every criterion is a veto gets loosened wholesale the first time something soft fails, because the only way to move is to lower a bar. A plan with two tiers has a third answer available: record the miss, print it in the verdict line, ship anyway with the exception attached to the artifact. That is a decision somebody made and can be held to, and it stays separate from the ones that are still absolute. The score, meanwhile, is deliberately weak on its own. Six of eight tells you how far along you are; the two named unmet lines under it tell you whether either of them is allowed to be missing.

⚠ Worked failure — the criterion that passed on last hour's number

Measuring all eight takes a while, so you add a fix-and-remeasure loop to main and skip the four minutes AC-01 spends on twenty clips, since the only thing you changed was the transcription model:

def main() -> None:
    criteria = acceptance_plan()
    with open_link() as link:
        run_plan(criteria, probes(link))
        report(criteria)
        input("fix what failed, then press enter to measure again: ")
        run_plan(criteria, probes(link))     # BUG: the same objects, measured twice
        report(criteria)

$ uv run python -m labs.acceptance   # second pass, AC-01 commented out of probes()
Acceptance run against plan c28f93f99a1b
  [PASS] AC-01  wake word to first spoken word            1420.0 ms        <= 2000.0  no measurement registered
  [PASS] AC-02  transcription word error rate                6.9 %         <= 10.0    5 wrong of 70 words
  [FAIL] AC-03  replies rated in character                   3.8 of 5      >= 4.0     20 turns, 3 rated below 4

  Score: 7 of 8 criteria met (87.5%)
  Unmet: AC-03 (exception allowed)
  Verdict: ACCEPTED WITH 1 EXCEPTION

The verdict is the one you were hoping for and the top line is nonsense: a PASS at 1420 milliseconds sitting next to a note saying nothing measured it. Both halves of that row are true, which is what makes the contradiction readable. The note came from this pass, where run_plan found no probe registered for AC-01 and moved on; the number came from the previous pass, still sitting in the same Criterion object, because acceptance_plan() was called once outside the loop and both passes wrote into those eight instances. Worse than a wrong number, it is a stale one: 1420 was measured with the old transcription model, and the model is precisely what changed. The fix is one line moved. Build the plan inside the loop, so each pass starts from measured=None and an unmeasured criterion reports what it truly is:

$ uv run python -m labs.acceptance   # fresh plan per pass
  [FAIL] AC-01  wake word to first spoken word      not measured ms        <= 2000.0  no measurement registered
  ...
  Score: 6 of 8 criteria met (75.0%)
  Unmet: AC-01 (blocking)
  Unmet: AC-03 (exception allowed)
  Verdict: NOT ACCEPTED

NOT ACCEPTED is the correct answer to a run that skipped a blocking criterion, and it costs you the four minutes you were trying to save. The general lesson is about where results are allowed to live. An object holding both the promise and the outcome is convenient right up to the second run, and it needs either a fresh instance per run or an explicit reset. Take the fresh instance: a reset is a line somebody forgets to update when a ninth field arrives.

Checkpoint, and the one exception on the sheet

✓ Checkpoint — what you can now do

I can write a criterion for something in my own build as a limit, a unit and a direction, and say which of the two directions makes it better.
I can explain why AC-03 is the only non-blocking entry, and what a plan of all-blocking criteria does to a team the first time a soft one misses.
I can say what plan_id hashes, what it deliberately ignores, and what two runs with different fingerprints do and do not prove.
Shown a report row that reads PASS beside a note saying nothing was measured, I can name the object that held the old number and the call that should have been inside the loop.
I know why a measurement that raises sets measured back to None, and what the report would claim if it did not.
I can point at the two lines of this chapter's second run that show the cost of the accuracy fix, and say which criterion now needs watching.

⚡ Exercises — try first, then reveal

Exercise 1 — add the ninth criterion. Pick something in your build that has no number yet: recovery time after the watchdog fires, or wake word false alarms per hour of silence. Add it to the plan with a limit and a direction, run the script without writing its probe, and read the verdict.

The report shows AC-09 as FAIL with "not measured" and "no measurement registered", the score drops to eight of nine, and the verdict goes back to NOT ACCEPTED if you made it blocking. Both of those are correct and both are uncomfortable, which is the exercise. The plan fingerprint changes too, because the promise now has a ninth clause. Leave the row failing until you have written the probe: a criterion nobody measures is the one most likely to be the reason she disappoints someone in month three.

Exercise 2 — read the history back. Write trend(path) that loads docs/acceptance.json and prints one line per run with the date, the score and the plan fingerprint, marking any run whose fingerprint differs from the one before it.

Walk the list in order and print something like f"{run['recorded_at'][:10]} {met}/{total} {run['plan']}", then append a marker when run["plan"] != previous. Across the two runs in this chapter you get 6/8 and 7/8 under one fingerprint, which is a system that improved. Now edit AC-05 down to 10 facts, re-run, and watch a third line appear with a new fingerprint and a better score. Those two kinds of progress print identically until the fingerprint column tells them apart, and that column is the entire reason the file is worth keeping.

Exercise 3 — make the exit code carry the verdict. Have main exit 0 for ACCEPTED, 1 for NOT ACCEPTED and 2 when there are exceptions, then run it from a shell and print $?.

Map the verdict string to a number and call sys.exit on it. The third code is what makes this more than a pass/fail gate: an automated job can refuse to deploy on 1, log a warning on 2 and stay quiet on 0, so the policy lives in the job instead of being argued out in the report. Wire it into the service unit from chapter 54 as a pre-start check and she declines to come up on a machine that has not met her own criteria.

There is a signed sheet now, with a score, a fingerprint and one honest hole in it. AC-03 is that hole, and it will not close by editing code: a rating of 3.8 out of 5 means three listeners in twenty found her tone wrong for the moment, which is a question about her manner and not her machinery. Chapter 60 gives that feedback somewhere to go, storing a preference profile that nudges verbosity and tone within clamped bounds and generating the prompt addendum from it, so the next time someone rates a reply the system can do something with the answer.