Performance Benchmarks

# labs/benchmark.py — copied to the board, run on the python3 JetPack shipped
import json
import urllib.request

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "llama3.2:3b"
NS_PER_MS = 1_000_000

# Her system prompt, copied out of configs/personality.json, so the request costs
# what one of her turns costs instead of what a demo question costs.
PREAMBLE = (
    "You are GLaDOS, the rogue AI from Aperture Science. You are sardonic, "
    "passive-aggressive and darkly witty. Answer in at most two sentences.\n\n"
    "User: Good morning.\n"
    "GLaDOS: It is morning. One of us is pleased about that.\n\n"
)
PROMPT = PREAMBLE + "User: Is the front door locked?\nGLaDOS:"


def measure(prompt: str, timeout: float = 180.0) -> dict:
    """One non-streamed generation, reduced to what a benchmark run reports."""
    body = json.dumps({"model": MODEL, "prompt": prompt, "stream": False}).encode("utf-8")
    request = urllib.request.Request(
        OLLAMA_URL,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as reply:
        r = json.loads(reply.read().decode("utf-8"))
    decode_ms = r["eval_duration"] / NS_PER_MS
    return {
        "tokens": r["eval_count"],
        "prompt_tokens": r["prompt_eval_count"],
        "decode_ms": round(decode_ms, 1),
        "stage_ms": round(r["total_duration"] / NS_PER_MS, 1),
        "decode_tps": round(r["eval_count"] / (decode_ms / 1000), 1),
    }


if __name__ == "__main__":
    print(measure(PROMPT))

$ ssh glados-jetson python3 benchmark.py   # measured on the bench — yours will vary
{'tokens': 31, 'prompt_tokens': 96, 'decode_ms': 1949.0, 'stage_ms': 7205.0, 'decode_tps': 15.9}

Two numbers come out of this call and they answer different questions. decode_tps divides the generated tokens by the generation duration, the pairing chapter 79 established, and it describes the model on this silicon. stage_ms is total_duration, everything the request cost from arrival to reply, and it is what the collector from chapter 49 would have timed if it had wrapped the call. The budget written in chapter 61 is in milliseconds, so a benchmark that reports only tokens a second cannot be checked against it. Notice the gap between the two on this first run: 1,949 milliseconds of generation inside a 7,205 millisecond stage, because the weights were read off the NVMe drive first.

The prompt is her actual preamble with one line of the conversation attached. Chapter 79 showed that prompt length barely moves the decode rate, so a demo question would have produced a similar decode_tps and a much smaller stage_ms. Benchmark the request she will really make.

# labs/benchmark.py — continued
NUM_RUNS = 6
WARMUP_RUNS = 1


def run_series(n: int = NUM_RUNS) -> list[dict]:
    """Every run that came back, in order. A run that raises prints and is skipped."""
    runs: list[dict] = []
    for i in range(1, n + 1):
        label = "warm-up" if i <= WARMUP_RUNS else "measured"
        try:
            r = measure(PROMPT)
        except Exception as exc:
            print(f"  run {i}/{n}  {label:8}  FAILED: {exc}")
            continue
        runs.append(r)
        print(f"  run {i}/{n}  {label:8}  {r['decode_tps']:5.1f} t/s{r['stage_ms']:9.1f} ms")
    return runs


if __name__ == "__main__":
    run_series()

$ ssh glados-jetson python3 benchmark.py   # measured on the bench — yours will vary
  run 1/6  warm-up    15.9 t/s   7205.0 ms
  run 2/6  measured   18.7 t/s   1836.0 ms
  run 3/6  measured   19.2 t/s   1693.0 ms
  run 4/6  measured   18.5 t/s   1907.0 ms
  run 5/6  measured   18.1 t/s   1720.0 ms
  run 6/6  measured   17.5 t/s   1882.0 ms

Read run 1 against runs 2 through 6 before reading anything else. It is slower in both columns, and the two columns are slow for different reasons. Its stage cost includes the 4.9 seconds of disk read that the later runs skip, because Ollama keeps the model resident for five minutes after a request. Its decode rate is low for a second reason: the GPU clocks were idling when the request arrived and spent the first fraction of a second climbing. That run is not a bad measurement of this board. It is a good measurement of a different situation, the first question after a long silence, and mixing it into a mean answers neither question.

The function keeps every run in full, not the one number the summary will use later. A list of six floats is smaller and it throws away the ability to ask a new question of an old benchmark. Tomorrow you will want to know whether the slow runs were slow in generation or slow in loading, and only the record can say.

# labs/benchmark.py — continued
def summarize(values: list[float]) -> dict[str, float]:
    """A mean says where the runs sat; min and max say how far they wandered."""
    return {
        "mean": round(sum(values) / len(values), 1),
        "min": min(values),
        "max": max(values),
    }


if __name__ == "__main__":
    runs = run_series()
    for key in ("decode_tps", "stage_ms"):
        values = [r[key] for r in runs]
        print(f"  {key}")
        print(f"    all six       {summarize(values)}")
        print(f"    warm-up out   {summarize(values[WARMUP_RUNS:])}")

$ ssh glados-jetson python3 benchmark.py   # tail of the same run
  decode_tps
    all six       {'mean': 18.0, 'min': 15.9, 'max': 19.2}
    warm-up out   {'mean': 18.4, 'min': 17.5, 'max': 19.2}
  stage_ms
    all six       {'mean': 2707.2, 'min': 1693.0, 'max': 7205.0}
    warm-up out   {'mean': 1807.6, 'min': 1693.0, 'max': 1907.0}

One discarded run moves the reported rate from 18.0 to 18.4, which sounds like rounding and is not. It moves the stage summary from a mean of 2,707 milliseconds with a worst case of 7,205 to a mean of 1,808 with a worst case of 1,907. The ceiling chapter 61 set for the model stage is 3,000 milliseconds. Keep the warm-up in and this board fails that ceiling on its worst run by more than a factor of two; take it out and the worst run clears it by 1,093 milliseconds. The same six requests, the same silicon, two opposite verdicts, and the only difference is whether a cold-start measurement was allowed to describe a warm system.

The spread is the reason min and max are reported at all. A mean of 18.4 is compatible with five runs clustered inside half a token a second, and equally compatible with a board that started at 19 and was down to 12 by the last run while its case heated up. Those are different boards and you would want to know which one you own. Chapter 61 refused to print a percentile from 14 samples; five samples cannot support one either, so this file reports the ends of the range it actually saw and claims nothing about the ninety-fifth.

# labs/benchmark.py — continued
import os
import time

BOARD = "jetson-orin-nano-super"
POWER_MODE = "MAXN SUPER 25W"          # read off `nvpmodel -q`, copied in by hand
RESULTS_DIR = "glados/data/benchmarks"


def build_record(runs: list[dict]) -> dict:
    """Everything a future reader needs to decide whether this run is comparable."""
    return {
        "board": BOARD,
        "power_mode": POWER_MODE,
        "model": MODEL,
        "prompt_tokens": runs[0]["prompt_tokens"],
        "runs": NUM_RUNS,
        "warmup_runs": WARMUP_RUNS,
        "decode_tps": summarize([r["decode_tps"] for r in runs][WARMUP_RUNS:]),
        "stage_ms": summarize([r["stage_ms"] for r in runs][WARMUP_RUNS:]),
        "per_run_tps": [r["decode_tps"] for r in runs],
        "started": time.strftime("%Y-%m-%dT%H:%M:%S"),
    }


def save(record: dict) -> str:
    os.makedirs(RESULTS_DIR, exist_ok=True)
    stamp = record["started"].replace(":", "-")
    path = f"{RESULTS_DIR}/{record['board']}-{stamp}.json"
    with open(path, "w") as handle:
        json.dump(record, handle, indent=2)
    return path


if __name__ == "__main__":
    print(f"  wrote {save(build_record(run_series()))}")

$ ssh glados-jetson cat glados/data/benchmarks/jetson-orin-nano-super-2026-08-14T20-31-07.json
{
  "board": "jetson-orin-nano-super",
  "power_mode": "MAXN SUPER 25W",
  "model": "llama3.2:3b",
  "prompt_tokens": 96,
  "runs": 6,
  "warmup_runs": 1,
  "decode_tps": {
    "mean": 18.4,
    "min": 17.5,
    "max": 19.2
  },
  "stage_ms": {
    "mean": 1807.6,
    "min": 1693.0,
    "max": 1907.0
  },
  "per_run_tps": [
    15.9,
    18.7,
    19.2,
    18.5,
    18.1,
    17.5
  ],
  "started": "2026-08-14T20:31:07"
}

One file per run, named for the board and the moment, never overwritten. That choice is what makes a benchmark an archive instead of a reading, and every field in it exists to answer a question a future comparison will raise. Which board. Which power mode, because the same silicon at 7 W is a different machine. How long the prompt was, because a benchmark against a 900-token prompt is not the same experiment. How many runs, and how many were discarded, so nobody has to guess whether the mean includes a cold start. And per_run_tps keeps the raw values, warm-up included, so a later reader who disagrees with the reduction can redo it instead of re-running the board.

The script writes into glados/data/benchmarks beside itself on the Jetson, since her workspace does not land there until chapter 82. Copy the file back to the repository with scp and it joins the record of the build.

# labs/compare_boards.py — runs on the repository machine, not the board
import glob
import json

RESULTS_DIR = "glados/data/benchmarks"


def load_all(pattern: str = f"{RESULTS_DIR}/*.json") -> list[dict]:
    records: list[dict] = []
    for path in sorted(glob.glob(pattern)):
        with open(path) as handle:
            records.append(json.load(handle))
    return sorted(records, key=lambda r: r["started"])


def compare(records: list[dict], model: str) -> None:
    """Decode rate only: it is the column that survives a change of prompt."""
    same = [r for r in records if r["model"] == model]
    print(model)
    for r in same:
        d = r["decode_tps"]
        print(f"  {r['board']:<24}{r['started']}  mean {d['mean']:5.1f} t/s"
              f"   spread {d['min']:.1f}-{d['max']:.1f}")
    baseline, latest = same[0], same[-1]
    factor = latest["decode_tps"]["mean"] / baseline["decode_tps"]["mean"]
    print(f"  {latest['board']} is {factor:.1f}x the decode rate of {baseline['board']}")


if __name__ == "__main__":
    compare(load_all(), "llama3.2:3b")

$ uv run python -m labs.compare_boards   # both records measured on the bench — yours will vary
llama3.2:3b
  raspberry-pi-4b         2026-07-29T21:14:03  mean   2.7 t/s   spread 2.5-2.8
  jetson-orin-nano-super  2026-08-14T20:31:07  mean  18.4 t/s   spread 17.5-19.2
  jetson-orin-nano-super is 6.8x the decode rate of raspberry-pi-4b

There is the receipt for the money. Chapter 75 predicted the ratio from the bandwidth row alone, 12.8 GB/s against 102, and the measurement lands on 6.8. Only the decode column is compared, on purpose: stage milliseconds depend on how long the prompt was and whether the weights were resident, and two evenings apart those were not identical. The decode rate is a property of the model and the silicon, which is what makes it the honest column to carry across boards.

# labs/turn_budget.py
from labs.latency_budget import STAGES, LatencyBudget

# p95 per stage from the evening recorded in chapter 61, on the old board.
MEASURED_P95_MS = {"stt": 1042.8, "llm": 4270.5, "tts": 512.4, "arm": 1180.6}


def with_llm(p95: dict[str, float], llm_ms: float) -> dict[str, float]:
    """The same turn with one stage replaced by a newer measurement."""
    return {**p95, "llm": llm_ms}


def turn_ms(p95: dict[str, float]) -> float:
    return sum(p95[stage] for stage in STAGES)


def report(before: dict[str, float], after: dict[str, float]) -> None:
    budget = LatencyBudget()
    print(f"{'stage':<6}{'before ms':>11}{'after ms':>10}{'ceiling':>9}  verdict")
    rows = [(s, before[s], after[s], budget.for_stage(s)) for s in STAGES]
    rows.append(("turn", turn_ms(before), turn_ms(after), budget.total_ms))
    for name, was, now, ceiling in rows:
        verdict = "ok" if now <= ceiling else f"OVER by {now - ceiling:.1f}"
        print(f"{name:<6}{was:>11.1f}{now:>10.1f}{ceiling:>9.0f}  {verdict}")


if __name__ == "__main__":
    after = with_llm(MEASURED_P95_MS, 1907.0)     # the worst of five measured runs
    report(MEASURED_P95_MS, after)
    print(f"model {18.4 / 2.7:.1f}x faster, "
          f"turn {turn_ms(MEASURED_P95_MS) / turn_ms(after):.1f}x faster")

$ uv run python -m labs.turn_budget
stage   before ms  after ms  ceiling  verdict
stt        1042.8    1042.8     1500  ok
llm        4270.5    1907.0     3000  ok
tts         512.4     512.4      800  ok
arm        1180.6    1180.6      600  OVER by 580.6
turn       7006.3    4642.8     5900  ok
model 6.8x faster, turn 1.5x faster

The last line is the chapter. The model got 6.8 times faster and the conversation got 1.5 times faster, because the model was never the whole turn. Substituting the worst of five measured runs for a p95 is deliberately pessimistic and it is the only substitution honest at five samples. The three unchanged rows are unchanged for a plain reason: transcription, speech and the arm still run on the old board, and they move here in chapter 82. Reading the table is a different exercise from reading a tokens-per-second figure. The llm row was 61 percent of the whole turn by itself and is now 41 percent of a shorter one, so the next second you want back is not in the model at all.

The arm row also stays over its ceiling, and it stayed over it through a hardware migration that never touched a servo. Chapter 61 marked that row thin at 14 samples and thin it remains. A GPU cannot fix a pin, which is where chapter 81 starts.