The Lab Notebook
Did that change actually help?
Volume 3 has been one tuning decision after another: a normalization rule added in
chapter 23, a mood threshold nudged in chapter 24, and behind all of it the standing
questions every reader of this book eventually asks. Would the small
Whisper model beat base on your microphone? Does a longer personality
prompt cut the out-of-character replies, or just slow her down? Each of these is an
experiment: a hypothesis, a few parameters you change, a number you measure. After a
dozen of them, memory blurs. You re-run tests you already ran, you misremember which
configuration won, and, worst of all, you "remember" results that flatter the change
you were hoping would work.
The usual fixes rot. Terminal scrollback gets cleared. A scratch file keeps the numbers
but loses which parameters produced them. And your head quietly rewrites the hypothesis
after it has seen the result, which is the cardinal sin of experimentation:
"I always suspected base was good enough" is a sentence memory
manufactures on demand. Six months from now, "I think it was around ninety-something
percent" defends no decision at all.
The fix is to make each experiment a first-class object with a life cycle. This chapter's rule: create the experiment, with its hypothesis, before you run anything; record the result afterward, in a separate step; and let a file on disk, never your memory, be the record. Chapter 2 gave her configuration a dataclass. This chapter gives your decisions one.
Stamp, save, record
# labs/experiments.py
import uuid
from dataclasses import dataclass
from datetime import datetime
@dataclass
class Experiment:
name: str
hypothesis: str
parameters: dict
results: dict | None = None
status: str = "pending"
id: str = ""
created_at: str = ""
def __post_init__(self) -> None:
if not self.id:
self.id = str(uuid.uuid4())[:8]
if not self.created_at:
self.created_at = datetime.now().isoformat()
if __name__ == "__main__":
exp = Experiment(
name="whisper model size",
hypothesis="small beats base on word accuracy without pushing latency past 2s",
parameters={"model": "small", "compute_type": "int8", "beam_size": 5},
)
print(f"id: {exp.id} status: {exp.status}")
print(f"created_at: {exp.created_at}")
$ uv run python labs/experiments.py
id: 7c1f22a9 status: pending
created_at: 2026-08-21T09:14:03.512847
Your id and timestamp will differ; that is the point of them. The interesting move
is __post_init__. The dataclass machinery writes __init__
for you from the field annotations, assigns the plain defaults, and then hands
control to __post_init__, where real code can run once per instance.
A fresh UUID and the current time have to be computed there, at construction, because
a field default is evaluated exactly once, when the class is defined. Also note
results: dict | None = None: no result exists yet, and the record says
so honestly instead of holding an empty dict that looks like a measurement of
nothing.
import json
from dataclasses import asdict
from pathlib import Path
EXPERIMENTS_DIR = Path("glados/data/experiments")
def save_experiment(exp: Experiment) -> Path:
EXPERIMENTS_DIR.mkdir(parents=True, exist_ok=True)
path = EXPERIMENTS_DIR / f"{exp.id}.json"
path.write_text(json.dumps(asdict(exp), indent=2))
return path
$ cat glados/data/experiments/7c1f22a9.json
{
"name": "whisper model size",
"hypothesis": "small beats base on word accuracy without pushing latency past 2s",
"parameters": {
"model": "small",
"compute_type": "int8",
"beam_size": 5
},
"results": null,
"status": "pending",
"id": "7c1f22a9",
"created_at": "2026-08-21T09:14:03.512847"
}
asdict() walks the dataclass recursively (nested dicts included) and
returns plain Python that json.dumps understands, so keep the contents
of parameters and results JSON-native: strings, numbers,
booleans, lists, dicts. Naming the file by id gives every experiment a stable
address; two experiments can never clobber each other, and the next function can
find any record with nothing but its id. Python's None lands in the
file as JSON null, exactly the "no result yet" marker a
pending record wants.
def record_result(exp_id: str, results: dict) -> Experiment:
path = EXPERIMENTS_DIR / f"{exp_id}.json"
data = json.loads(path.read_text())
exp = Experiment(**data)
exp.results = results
exp.status = "complete"
save_experiment(exp)
return exp
def main() -> None:
exp = Experiment(
name="whisper model size",
hypothesis="small beats base on word accuracy without pushing latency past 2s",
parameters={"model": "small", "compute_type": "int8", "beam_size": 5},
)
save_experiment(exp)
print(f"created {exp.id} (status: {exp.status})")
done = record_result(exp.id, {"word_accuracy": 0.94, "latency_ms": 1840})
print(f"recorded {done.id} (status: {done.status})")
print(f"results: {done.results}")
if __name__ == "__main__":
main()
$ uv run python labs/experiments.py
created 4a70f0cd (status: pending)
recorded 4a70f0cd (status: complete)
results: {'word_accuracy': 0.94, 'latency_ms': 1840}
The accuracy and latency here are one bench run on one machine; when you run the
real comparison, your numbers will differ. What holds everywhere is the signature:
record_result takes an id, not the live object, because in
real use the run happens later, sometimes in a different process, with only the id
written on a sticky note. Loading with Experiment(**data) proves the
round-trip works: the JSON keys match the field names, so the dict unpacks straight
into the constructor, and __post_init__ leaves the loaded id and
timestamp alone because they are already truthy. The function can change
results and status and nothing else. That asymmetry is
deliberate: no code path exists for editing a hypothesis after the fact.
Python could pickle the object in one line, but a pickle is opaque bytes tied to
today's class definition. JSON stays readable in any editor six months from now,
diffs cleanly, and survives the class growing new fields. Commit
glados/data/experiments/ along with your code: each record is a few
hundred bytes, and the day you need to defend a model choice, the audit trail is
already sitting in your history.
Why this works: defaults run once, __post_init__ runs every time
The mechanism under this chapter generalizes to every dataclass you will ever write.
When Python executes the class body, it evaluates each default value exactly once and
stores it on the class; every future instance that omits the field receives that same
stored object. For an immutable value like "pending", sharing is harmless:
nobody can change a string in place. For anything that must be fresh per instance (a
new UUID, the current time, an empty list) sharing would be a quiet disaster, so the
rule is to compute those in __post_init__, which the generated
__init__ calls once for every object constructed.
The sentinel pattern in stage 1 follows from that rule: default the field to something
falsy (""), and have __post_init__ fill it only when it is
empty. Fresh objects get stamped; objects rebuilt from disk keep their original id and
timestamp. One method serves both construction paths, and it is the same guard that
made Experiment(**data) in stage 3 safe. Dataclasses go one step further
for mutable types: a shared dict default is dangerous enough that the decorator
refuses to build the class at all.
Defaulting results to an empty dict feels natural, since it spares
every reader a None check:
@dataclass
class Experiment:
name: str
hypothesis: str
parameters: dict
results: dict = {} # looks harmless
status: str = "pending"
$ uv run python labs/experiments.py
Traceback (most recent call last):
File "labs/experiments.py", line 7, in <module>
@dataclass
^^^^^^^^^
File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
ValueError: mutable default <class 'dict'> for field results is not allowed: use default_factory
Read the traceback's top frame: the error points at the @dataclass
line itself, not at any call site, and main() never ran. That location
is the diagnosis. The failure happens while the decorator is building the class,
because one shared dict as a default would mean every experiment silently writing
into every other experiment's results, so dataclasses ban mutable defaults
(dict, list, set) outright. The error even
names both exits: field(default_factory=dict) when you truly want a
fresh empty dict per instance, or, as this chapter chose, None, because
"no result yet" and "empty result" deserve different spellings.
Checkpoint, hypothesis first
- I can explain when a dataclass field default is evaluated, why that is safe for
"pending", and why it forces the UUID and timestamp into__post_init__. - I can trace the two-step life cycle (create as
pending, record tocomplete) and point to the line where the hypothesis becomes unchangeable. - I know why
record_resulttakes an id instead of the object, and what theExperiment(**data)round-trip proves about the file. - I can predict what a fresh experiment's
resultsfield serializes to in JSON, and whyNonebeats{}as its default. - Handed the mutable-default
ValueError, I can read from the traceback that the class itself failed to build, and name both fixes.
Exercise 1 — the index you will actually use. Write
list_experiments(): scan glados/data/experiments/*.json
and print one line per record with id, status, and name. Run it after the stage 3
demo.
Glob, load, format: for path in sorted(EXPERIMENTS_DIR.glob("*.json")),
then data = json.loads(path.read_text()) and print
f"{data['id']} {data['status']:<9} {data['name']}". The
:<9 pads pending and complete to the
same width so the names line up in a column. With the demo's record on disk you
get one complete line; create a second experiment without recording
a result and a pending line joins it.
Exercise 2 — the unfinished-business report. Add
list_pending() returning the ids of every experiment still awaiting
results. What does a long pending list tell you about your own habits?
Same scan with a filter: keep records where data["status"] ==
"pending", collect the ids, return the list. The report is a mirror.
A few pending entries mean experiments queued honestly; a dozen mean you have
been creating hypotheses and never running them, or running them and recording
the outcome nowhere, which is the exact failure this chapter exists to end.
Either way the file system now tells you, instead of you having to notice.
Exercise 3 — settle the model question. Create two
experiments, base versus small, with the same hypothesis
and your own machine's numbers from re-running chapter 5's transcription on the
same WAV. Record both, then write compare_experiments(ids, metric)
printing a table of model, status, and one metric.
Load each id, pull data["parameters"].get("model", "?") and
(data["results"] or {}).get(metric, "-"); the or {}
guard keeps a still-pending record from crashing the table. The measurements
are yours, so the winner is too: on many machines small transcribes
noticeably better and takes two to three times as long, and now the trade-off
sits in two files you can cite instead of one impression you would have had to
trust.
Decisions about her are now records you can query. The same move applies one level up: the project itself is a pile of goals (finish the prosody pass, order the servos, wire the permission system) living in your head, exactly where hypotheses used to live. Next chapter turns the roadmap into dataclasses too: goals with priorities, deadlines, and subtasks, and a planner that can answer "what should I work on next?" from data instead of mood.