Chapter 05 — Hearing You

From a wall of floats to words

Chapter 4 left you holding 80,000 floats. Useful, but she cannot reason about a numpy array; to respond to a human she needs the sentence as text she can hand to a language model. That conversion, audio to text, is speech-to-text, and it is the entire listening half of the voice loop.

The tempting route is a cloud transcription API, and for this project it is the wrong one three times over: a per-request bill, a network round-trip inside every exchange, and a privacy hole you can never quite close, because every word your assistant hears leaves the building. An assistant that lives in your house should keep its ears in the house.

So the model runs on your machine. faster-whisper is a fast reimplementation of OpenAI's Whisper speech model; with int8 quantization the base size fits in about 150 MB and turns five seconds of speech into text in a couple of seconds on an ordinary CPU. No network, no bill, no audience. How a model like Whisper works inside is AI Zero's territory; here it is a tool, and the craft is in feeding it correctly and reading what it actually returns. And what it hands back, as you will see, is no plain string.

▣ Build · stage 1 — load the model once, at module level

# labs/stt_mic.py
from faster_whisper import WhisperModel

MODEL_SIZE = "base"
model = WhisperModel(MODEL_SIZE, device="cpu", compute_type="int8")
print(f"Loaded Whisper '{MODEL_SIZE}' on CPU (int8).")

$ uv run python labs/stt_mic.py
Loaded Whisper 'base' on CPU (int8).

First uv add faster-whisper, and expect the first run to pause while the weights download. The placement is the lesson of this stage: constructing the model reads 150 MB off disk, so it happens once, at module level, when the file loads. Chapter 10's loop will call transcribe() against this one already-loaded model on every exchange. Load-once, use-many is the pattern for every model in this book, and violating it is the difference between a one-second reply and a twelve-second one.

▣ Build · stage 2 — record and persist, exactly as taught

import sounddevice as sd
import soundfile as sf

SAMPLE_RATE = 16000
CHANNELS = 1
DURATION = 5
AUDIO_FILE = "glados/data/heard.wav"

print(f"Speak now... ({DURATION} seconds)")
audio = sd.rec(int(DURATION * SAMPLE_RATE), samplerate=SAMPLE_RATE,
               channels=CHANNELS, dtype="float32")
sd.wait()
sf.write(AUDIO_FILE, audio, SAMPLE_RATE)
print("Saved", AUDIO_FILE)

Chapter 4's capture, condensed: the rate Whisper was trained on, the barrier before the save. Why a file at all, when transcribe() could take an array? Because the WAV is an inspectable artifact. When a transcript comes back wrong, you play the file and instantly know which side of the boundary the bug lives on: garbled audio means a capture problem, clean audio with a wrong transcript means a transcription problem. Decoupled stages are debuggable stages.

▣ Build · stage 3 — transcribe, and join the stream

print("Got it. Transcribing...")
segments, info = model.transcribe(AUDIO_FILE, language="en")
text = " ".join(s.text.strip() for s in segments)
print(f"\nYou said: {text}")

$ uv run python labs/stt_mic.py
Loaded Whisper 'base' on CPU (int8).
Speak now... (5 seconds)
Saved glados/data/heard.wav
Got it. Transcribing...

You said: Hello GLaDOS, can you hear me yet?

Your transcript is whatever you said; the mechanics are the same for everyone. transcribe() returns a pair: info (detected language, duration) and segments, a lazy generator of segment objects, each carrying a .text fragment of the sentence. The join walks the generator, strips each piece, and assembles one string. Passing language="en" skips auto-detection and saves time on every call: the right trade for an assistant that lives in one language.

Why this works: the work happens when you look

The surprise in this API: model.transcribe(...) returns almost instantly, before any audio has been decoded. The actual model runs as you consume the generator. Lazy evaluation lets a consumer stream partial results from long recordings, and it carries one consequence that bites everyone exactly once: a generator can be walked a single time.

# exhaust.py — the one-shot property, isolated
segments = (s for s in ["a", "b"])
print("first pass:", list(segments))
print("second pass:", list(segments))

$ uv run python labs/exhaust.py
first pass: ['a', 'b']
second pass: []

The first iteration does the work and yields everything; the second sees an exhausted stream and yields nothing, without complaint. So if you iterate once to count segments and again to build the text, your transcript comes back silently empty. The rule behind every Whisper consumer you will write: capture what you need on the first walk. Our join does exactly that, and stores the string.

⚠ Worked failure — why won't it join my segments?

The most common Whisper mistake: forgetting that segments yields objects. You read "join the pieces" and reach for the direct join. Reproduced here with a stand-in class, so you can trigger it without recording anything:

class Segment:
    def __init__(self, text):
        self.text = text

segments = [Segment(" Hello"), Segment(" there.")]
text = " ".join(segments)    # forgot .text
print(f"You said: {text}")

$ uv run python labs/fail_join.py
Traceback (most recent call last):
  File "labs/fail_join.py", line 6, in <module>
    text = " ".join(segments)    # forgot .text
TypeError: sequence item 0: expected str instance, Segment found

The same species of error as chapter 3's chunk join, and the same reading: "expected str, found Segment" names the missing attribute access. The fix reaches into each object first: " ".join(s.text.strip() for s in segments). Two chapters, two APIs, one lesson. Libraries hand you containers, and the join wants what is inside them.

◆ Note — picking a model size

Whisper comes in sizes: tiny, base, small, medium, large-v3. Each step buys accuracy with latency and memory. On a typical laptop CPU, tiny transcribes five seconds of speech well under a second but stumbles on names and noise; base takes a couple of seconds and holds up in a quiet room; small is noticeably better and noticeably slower. The honest numbers are the ones you measure on your own hardware (exercise 2), and the decision returns in volume 8, where the Jetson's GPU moves the whole trade-off curve. Start with base; change it when a measurement, not a mood, says to.

Checkpoint, halfway to a conversation

✓ Checkpoint — what you can now do

I can trace the three-stage STT pipeline (record to array, persist to 16 kHz mono WAV, transcribe to segments) and say what each stage hands the next.
I can explain why the model loads at module level and what it would cost the voice loop if it loaded per call.
I know transcribe() does its work lazily, and I can state the one-walk rule and the silent failure that breaking it produces.
I can name the trade each Whisper size makes, and I know which number to measure before changing it.
Handed a wrong transcript, my first move is playing the WAV, because it splits the pipeline in half.

⚡ Exercises — try first, then reveal

Exercise 1 — read the info you've been dropping. Print info.language, info.language_probability and info.duration after a transcription. Then drop language="en" and transcribe again. What changed, and what did it cost?

With the hint, the probability reports 1.0 and detection is skipped; without it, Whisper spends a moment classifying the language first (usually confidently right for clear English). info.duration should match your five seconds: a third way to cross-check the capture, after the frame count and the RMS.

Exercise 2 — race the sizes. Wrap the transcribe-and-join in time.perf_counter() and run the same WAV through tiny, base and small. Record seconds and transcript quality for each. Which would you give her?

On most laptop CPUs you will see something like tiny <1s, base ~1–3s, small ~3–8s for a five-second clip, with your machine setting the real numbers. Quality differences show on proper nouns and fast speech. There is no universal right answer; there is a right answer for your hardware and your patience, and now you have the table to defend it. Keep the timing harness; chapter 10 wants it.

Exercise 3 — transcribe her own voice. Feed chapter 3's glados_says.wav to the transcriber. How close is the round trip, text → speech → text?

Usually very close: synthesized speech is clean, evenly paced and noise-free, which is Whisper's favorite diet. Punctuation and capitalization may drift. You have just run her mouth into her ears, and the fact that the loop closes at all is the proof that both halves speak the same language: samples in files, at declared rates.

The machine hears you and can quote you back. Both halves of the conversation exist; neither sounds like her, and nothing thinks yet. The next two chapters fix the voice — first the dataset, because a cloned voice is only as good as the recordings behind it, and hers are scattered across a video game's files waiting to be collected.