Speaking While She Thinks

# labs/streaming_voice.py
TOKENS = ["Oh", ". It", "'s", " you", ". I", " was", " expecting", " someone", " taller"]

if __name__ == "__main__":
    for i, token in enumerate(TOKENS):
        print(f"{i:>2}  {token!r}")
    print(f"{len(TOKENS)} tokens, {sum(len(t) for t in TOKENS)} characters, "
          f"{len(''.join(TOKENS).split())} words")

$ uv run python labs/streaming_voice.py
 0  'Oh'
 1  '. It'
 2  "'s"
 3  ' you'
 4  '. I'
 5  ' was'
 6  ' expecting'
 7  ' someone'
 8  ' taller'
9 tokens, 44 characters, 8 words

A fixed list stands in for the network so the buffering logic can be argued about on its own, and it is a fair stand-in: those are the cuts a model really makes. Nine tokens carry eight words. Leading spaces belong to the token that follows them, an apostrophe and its suffix arrive as their own piece, and token 1 holds the end of one sentence and the start of the next in a single string. Hand any one of these to a synthesizer and you get a word fragment read flat.

Prosody is why the unit has to be a sentence. A synthesizer decides its pitch contour from the whole text it is given: the voice falls at a full stop, lifts at a question mark, and stretches the syllable before a comma. All of that is planned before the first sample of audio is written, from punctuation the model has not typed yet when token 0 arrives. Feed the synthesizer one word and it has nothing to plan with.

import re

SENTENCE_END = re.compile(r"[.!?]")

def split_completed(buffer: str) -> tuple[str, str]:
    """Everything through the last sentence end, and the unfinished tail."""
    last = None
    for match in SENTENCE_END.finditer(buffer):
        last = match
    if last is None:
        return "", buffer
    return buffer[:last.end()], buffer[last.end():]

if __name__ == "__main__":
    buffer = ""
    for i, token in enumerate(TOKENS):
        buffer += token
        done, buffer = split_completed(buffer)
        if done.strip():
            print(f"token {i}: speak {done.strip()!r}, tail {buffer!r}")
    print(f"stream ended, buffer holds {buffer.strip()!r}")

$ uv run python labs/streaming_voice.py
token 1: speak 'Oh.', tail ' It'
token 4: speak "It's you.", tail ' I'
stream ended, buffer holds 'I was expecting someone taller'

Two decisions are doing all the work. The loop keeps the last match instead of the first because a buffer can hold several finished sentences at once when tokens arrive in a burst, and cutting at the first would leave completed text sitting in the buffer waiting for a boundary that already went by. And the cut is a slice, not a reset. Setting buffer = "" after speaking would have thrown away the " It" that rode in on token 1, and the loss is invisible: she would say "s you." and nobody would ever see an error.

Now read the last line of that output, because it is the whole trap of stream processing in one sentence. The tokens ran out while thirty characters were still sitting in the buffer, unspoken. Every loop that consumes a stream in pieces ends this way, and the code that handles it has to live after the loop, not inside it.

ABBREVIATIONS = {"dr", "mr", "mrs", "ms", "st", "vs", "approx"}
MIN_SPEAK_CHARS = 12

def is_boundary(text: str, match: re.Match) -> bool:
    """True when this .!? really ends a sentence, given what has arrived so far."""
    i = match.start()
    if match.group() != ".":
        return True
    if i > 0 and text[i - 1].isdigit() and text[i + 1:i + 2].isdigit():
        return False
    word = re.search(r"([A-Za-z]+)$", text[:i])
    return not (word and word.group(1).lower() in ABBREVIATIONS)

def split_completed(buffer: str, minimum: int = MIN_SPEAK_CHARS) -> tuple[str, str]:
    """Everything through the last decided sentence end, and the tail to keep."""
    last = None
    for match in SENTENCE_END.finditer(buffer):
        if match.end() >= len(buffer):
            break
        if is_boundary(buffer, match):
            last = match
    if last is None or last.end() < minimum:
        return "", buffer
    return buffer[:last.end()], buffer[last.end():]

CASES = [
    "Oh. It's you. The chamber is ready",
    "Dr. Chell is late.",
    "The core reads 3.14 volts. Barely",
    "Really? Fine",
]

if __name__ == "__main__":
    for text in CASES:
        done, tail = split_completed(text)
        print(f"{text!r}\n  speak {done.strip()!r}  hold {tail!r}")

$ uv run python labs/streaming_voice.py
"Oh. It's you. The chamber is ready"
  speak "Oh. It's you."  hold ' The chamber is ready'
'Dr. Chell is late.'
  speak ''  hold 'Dr. Chell is late.'
'The core reads 3.14 volts. Barely'
  speak 'The core reads 3.14 volts.'  hold ' Barely'
'Really? Fine'
  speak ''  hold 'Really? Fine'

Four cases, four different refusals to be naive. The first ships two sentences in one chunk, which is correct: both are finished, and a synthesizer given both reads them with the pause between them intact. The third case shows the decimal rule earning its keep, since a period with a digit on each side is inside a number and splitting there makes her say "three point" and then, as a fresh sentence, "one four volts."

The second case is the abbreviation set, and the set is short on purpose. Words that are sometimes a real ending stay out: "no" abbreviates "number" and also finishes thousands of ordinary sentences, so adding it would swallow the end of every reply where she says no. A list like this trades one class of error for another and you pick the one you would rather hear. Trained sentence segmenters do better on prose full of citations, but they expect a finished document; this splitter runs on a partial string that grows by three characters at a time.

The third rule is the one nobody predicts. A boundary sitting at the very end of the buffer is undecidable, because "3." and "3.14" look identical until the next token arrives, so the loop breaks there and waits. That rule fires on nearly every reply, since a finished answer usually ends with a period, and it means the last sentence she generates is always left in the buffer. The fourth case adds the prosody floor: "Really?" is a real boundary, but seven characters is a scrap, and MIN_SPEAK_CHARS holds it back to join whatever comes next.

import json
from collections.abc import Iterable, Iterator

RAW_LINES = [
    b'{"response":"Oh","done":false}',
    b'{"response":". It","done":false}',
    b'{"response":"\'s","done":false}',
    b'{"response":" you","done":false}',
    b'{"response":". I","done":false}',
    b'{"response":" was","done":false}',
    b'{"response":" expecting","done":false}',
    b'{"response":" someone","done":false}',
    b'{"response":" taller","done":false}',
    b'{"response":"","done":true,"eval_count":9,"eval_duration":471000000}',
]

def sentences_from(lines: Iterable[bytes], stats: dict | None = None) -> Iterator[str]:
    """Yield speakable chunks as NDJSON lines arrive; flush the tail at the end."""
    buffer = ""
    for raw in lines:
        line = raw.decode("utf-8").strip()
        if not line:
            continue
        chunk = json.loads(line)
        buffer += chunk.get("response", "")
        done, buffer = split_completed(buffer)
        if done.strip():
            yield done.strip()
        if chunk.get("done"):
            if stats is not None:
                stats.update(chunk)
            break
    if buffer.strip():
        yield buffer.strip()

if __name__ == "__main__":
    stats: dict = {}
    for sentence in sentences_from(RAW_LINES, stats):
        print(f"[speak] {sentence}")
    seconds = stats["eval_duration"] / 1e9
    print(f"[stats] {stats['eval_count']} tokens in {seconds:.3f} s = "
          f"{stats['eval_count'] / seconds:.1f} tok/s")

$ uv run python labs/streaming_voice.py
[speak] Oh. It's you.
[speak] I was expecting someone taller
[stats] 9 tokens in 0.471 s = 19.1 tok/s

A streamed Ollama body is NDJSON: one complete, independent JSON object per line, with no enclosing array and no commas between them. That format exists precisely so a reader can act on line three without having seen line four, which a JSON array cannot offer, since an array is only parseable once its closing bracket arrives. Decode one line, parse it, take response, repeat.

The last object is different from all the others. Its response is empty, its done is true, and it carries the counters: 9 tokens generated in 471 million nanoseconds, which is 0.471 seconds, which is 19.1 tokens a second. Those are free measurements of exactly the stage the budget in chapter 61 caps at 3,000 milliseconds, arriving in the same reply as the text. Copying them into stats costs one line and gives the caller a throughput number it did not have to ask for.

The yield is not decoration either. A function returning a list would have to finish before the caller saw any of it, which is the blocking design wearing a different name.

import sys
import time
import urllib.request

from glados.core import GladOSCore
from labs.system_config import build_default_config
from labs.wire_core import build_core

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "llama3.2:3b"
SYSTEM_PROMPT = (
    "You are GLaDOS from Portal. Answer in two or three short sentences, "
    "in character: precise, sardonic, mildly threatening."
)

def stream_lines(prompt: str, timeout: float = 180.0) -> Iterator[bytes]:
    body = json.dumps({"model": MODEL, "system": SYSTEM_PROMPT,
                       "prompt": prompt, "stream": True}).encode("utf-8")
    request = urllib.request.Request(
        OLLAMA_URL, data=body,
        headers={"Content-Type": "application/json"}, method="POST")
    with urllib.request.urlopen(request, timeout=timeout) as stream:
        yield from stream

def answer_streaming(core: GladOSCore, prompt: str,
                     stats: dict | None = None) -> tuple[str, float, float]:
    """Speak each chunk as it completes. Returns (text, first_ms, total_ms)."""
    started = time.monotonic()
    first_ms, spoken = 0.0, []
    for sentence in sentences_from(stream_lines(prompt), stats):
        if not spoken:
            first_ms = (time.monotonic() - started) * 1000
        core.speak(sentence)
        spoken.append(sentence)
    return " ".join(spoken), first_ms, (time.monotonic() - started) * 1000

def answer_blocking(core: GladOSCore, prompt: str) -> tuple[str, float, float]:
    """The pipeline as it stands: every token first, then one synthesis."""
    started = time.monotonic()
    text = "".join(json.loads(line).get("response", "")
                   for line in stream_lines(prompt) if line.strip())
    first_ms = (time.monotonic() - started) * 1000
    core.speak(text.strip())
    return text.strip(), first_ms, (time.monotonic() - started) * 1000

def report(label: str, result: tuple[str, float, float]) -> None:
    text, first_ms, total_ms = result
    print(f"{label:<10} first sentence out {first_ms:6.0f} ms   total {total_ms:6.0f} ms")
    print(f"           {text}")

def main() -> None:
    prompt = " ".join(sys.argv[1:]) or "What is behind the door?"
    core = build_core(build_default_config())
    stats: dict = {}
    report("blocking", answer_blocking(core, prompt))
    report("streaming", answer_streaming(core, prompt, stats))
    seconds = stats["eval_duration"] / 1e9
    print(f"{stats['eval_count']} tokens in {seconds:.3f} s = "
          f"{stats['eval_count'] / seconds:.1f} tok/s")

if __name__ == "__main__":
    main()

$ uv run python labs/streaming_voice.py "What is behind the door?"   # her wording and your timings will differ
blocking   first sentence out   2314 ms   total  14680 ms
           Oh. It's you. The door is where you left it, and so, apparently, is your curiosity.
streaming  first sentence out    623 ms   total  13520 ms
           Oh. It's you. Behind the door is another door. I find that funnier than you will.
40 tokens in 2.061 s = 19.4 tok/s

Two runs, two different replies, and the replies are not the point. The first column is. Blocking waits 2,314 milliseconds before a single character reaches the voice, because that is how long the model took to finish; streaming gets there in 623, because that is how long the model took to finish saying "Oh. It's you." Audio starts one synthesis after each of those numbers, and both designs pay that same synthesis, so the gap between the columns is pure removed silence: about 1.7 seconds of a room waiting.

Now read the second column and notice how little happened there. The total moved by about a second, and most of what she costs is the plain fact that saying a sentence out loud takes as long as it takes. If total time had been your metric you would have called this change a rounding error and thrown it away.

One detail in stream_lines is easy to walk past. The with block sits inside a generator, so the HTTP connection stays open across every yield and closes only when the generator is exhausted. That is what lets the caller read a token, spend two seconds speaking, and come back for the next one. It also means abandoning the generator halfway leaves a socket open until Python collects it, which matters the moment you add a way to interrupt her.

REPLY = ("Oh. It's you. The chamber is ready, though I use the word loosely. "
         "Do try not to die in the first room.")

def fake_tokens(text: str) -> list[str]:
    """Stand-in for the model's tokenizer: whole words, leading space attached."""
    return re.findall(r"\s*\S+", text)

def first_chunk(tokens: list[str], minimum: int) -> tuple[int, str]:
    buffer = ""
    for i, token in enumerate(tokens):
        buffer += token
        done, buffer = split_completed(buffer, minimum)
        if done.strip():
            return i, done.strip()
    return len(tokens), buffer.strip()

if __name__ == "__main__":
    tokens = fake_tokens(REPLY)
    print(f"{len(tokens)} tokens, {len(REPLY)} characters")
    print(f"{'minimum':>8}  {'token':>5}  first chunk")
    for minimum in (1, 12, 30):
        i, chunk = first_chunk(tokens, minimum)
        print(f"{minimum:>8}  {i:>5}  {chunk!r}")

$ uv run python labs/streaming_voice.py
22 tokens, 103 characters
 minimum  token  first chunk
       1      1  'Oh.'
      12      3  "Oh. It's you."
      30     13  "Oh. It's you. The chamber is ready, though I use the word loosely."

The knob has a price list now. At a minimum of 1 she speaks after two tokens, and what she speaks is the word "Oh" with a full stop on it, which a synthesizer renders as a clipped grunt with no room to bend the pitch down at the end. At 30 she sounds like a person reading a paragraph, and eleven more tokens have to arrive first. At the 19.4 tokens a second stage 5 measured, each token is about 52 milliseconds, so those eleven tokens are worth a little over half a second of extra silence.

That is the entire tradeoff of this chapter, in one table: time to first audio against how natural the first thing she says sounds. There is no setting that wins both, and the right value depends on your synthesizer, since a model that plans pitch over long spans suffers more from short chunks than one that works phrase by phrase. Twelve characters is a defensible default because it clears greetings and one-word reactions while still cutting at the first real sentence. Change it, listen to ten replies, and keep the one you stop noticing.