GLaDOS Vol 1 · Her Voice
ch 07 / 99
Chapter 07

Voice cloning

Two roads to her voice

Start with a rights-cleared reference recording and its exact transcript. Zero-shot synthesis uses that pair with pretrained weights, avoiding a training job before the recording has been tested.

F5-TTS is a pretrained voice-cloning model that already learned how human speech works from thousands of hours of generic audio. That prior knowledge unlocks zero-shot cloning: hand it one clean reference clip plus that clip's exact transcript, and it speaks new text in that voice on the first run. No training. The cost is consistency: long outputs can drift in timbre, and rare sounds occasionally come out wrong, because the model is improvising the voice from a single example every time.

Fine-tuning updates the pretrained weights using your authorized dataset. It can improve consistency for that data, but standard F5-TTS inference still uses reference audio and text after fine-tuning. Keep this rule: both routes condition on a reference pair; fine-tuning changes the weights, not the need for that pair. Start with zero-shot to judge the recording before paying for training. Neither route grants rights to a voice, recording or checkpoint. The wrapper below works with compatible fine-tuned checkpoints too.

Preflight cheap, load once, speak forever

▣ Build · stage 1 — preflight the reference before anything heavy
# labs/glados_voice.py
import wave
from pathlib import Path

REF_FILE = "glados/data/voice/reference.wav"
REF_TEXT = "This is my reference recording for the local assistant."

def reference_info(ref_file: str) -> dict[str, float]:
    path = Path(ref_file)
    if not path.exists():
        raise FileNotFoundError(f"Reference clip not found: {ref_file}")
    with wave.open(str(path), "rb") as wav:
        return {"seconds": round(wav.getnframes() / wav.getframerate(), 2),
                "sample_rate": float(wav.getframerate())}

if __name__ == "__main__":
    info = reference_info(REF_FILE)
    print(f"Reference OK: {info['seconds']}s @ {int(info['sample_rate'])} Hz")
$ uv run python labs/glados_voice.py
Reference OK: 4.9s @ 24000 Hz

Use the recording you made or obtained with consent in chapter 6, and change REF_TEXT if you recorded different words. The retained run-outs used an earlier reference; their durations are exemplars, not measurements of your new file. Why check before loading? Because instantiating the cloning model downloads roughly two gigabytes and pins your machine for tens of seconds, and if the path is wrong or the file is not really a WAV you want to fail in milliseconds with a clear message instead. The standard-library wave module reads just the header: chapter 3 knowledge, now doing gatekeeping duty. Pick your reference deliberately (a clean four-to-ten-second clip, one consenting voice alone, no background music) and make sure REF_TEXT matches the audio word for word, punctuation included. The model aligns the reference audio against that text to learn the voice, and a sloppy transcript smears everything downstream.

▣ Build · stage 2 — wrap the model behind one function
from f5_tts.api import F5TTS

OUTPUT = "glados_says.wav"

def say(tts: F5TTS, text: str) -> float:
    wav, sr, _ = tts.infer(
        ref_file=REF_FILE,
        ref_text=REF_TEXT,
        gen_text=text,
        file_wave=OUTPUT,
        remove_silence=True,
    )
    return len(wav) / sr

Two details carry weight. The import is f5_tts.api, because the public interface lives in that submodule; import from the bare package and there is no F5TTS to be found (the failure box reproduces it). And infer() returns a three-tuple of waveform, sample rate and spectrogram; we keep two, discard the spectrogram with _, and return the duration. Wrapping the call in say() means nothing else in the program ever has to remember that signature — chapter 10 imports this exact function.

▣ Build · stage 3 — the interactive loop
import sounddevice as sd
import soundfile as sf

def main() -> None:
    info = reference_info(REF_FILE)
    print(f"Reference OK: {info['seconds']}s @ {int(info['sample_rate'])} Hz")
    tts = F5TTS()
    print("Type something for GLaDOS to say (blank line to quit).")
    while True:
        text = input("GLaDOS > ").strip()
        if not text:
            break
        seconds = say(tts, text)
        print(f"Generated {seconds:.2f}s of audio. Playing...")
        data, samplerate = sf.read(OUTPUT)
        sd.play(data, samplerate)
        sd.wait()

if __name__ == "__main__":
    main()
$ uv run python labs/glados_voice.py
Reference OK: 4.9s @ 24000 Hz
Type something for GLaDOS to say (blank line to quit).
GLaDOS > You found this chapter. I'm genuinely impressed. And a little disappointed.
Generated 5.84s of audio. Playing...
GLaDOS >

The program checks the recording before loading the model, then reuses the loaded instance for each sentence. Measure generation time on your machine.

◆ Note — outputs in this volume vary by nature

From here on, much of what this book shows you is the output of neural models, and model output is not byte-stable: your generation times, your drift, even her exact intonation on a given sentence will differ from run to run and machine to machine. The run-outs on these pages are honest captures of a working build; treat them as the pattern to match, not bytes to diff. Numbers that must agree exactly (frame counts, file sizes, error messages) will keep agreeing exactly.

Why this works: conditioning, not memory

F5-TTS conditions generation on reference audio represented as mel-spectrogram frames, a sequence of sound-energy measurements across frequency bands, together with the reference and requested text. It fills the requested audio region using those inputs; this path does not first reduce the voice to a dedicated speaker embedding. A clean recording and matching transcript give the model a consistent audio/text example. Noise, mismatched words and long outputs can still hurt pronunciation or timbre.

Fine-tuning adjusts the network's learned parameters with more examples; it may improve a target voice or domain and can also overfit poor data. Standard F5TTS.infer still takes ref_file, ref_text and gen_text. A compatible checkpoint selected with ckpt_file leaves say()'s reference arguments in place; its model configuration and vocabulary must also match. Check the version you installed against the upstream API and conditioning implementation. Compare held-out recordings before deciding training helped. Its duration and success depend on your dataset and hardware, so an overnight run is no guarantee.

⚠ Worked failure — but it's right there, it's an audio file

You exported your own recording as glados.mp3, then pointed REF_FILE at it. The preflight chokes before the model loads:

print(reference_info("glados.mp3"))
$ uv run python labs/fail_ref.py
Traceback (most recent call last):
  File "labs/fail_ref.py", line 15, in <module>
    print(reference_info("glados.mp3"))
  File "labs/fail_ref.py", line 9, in reference_info
    with wave.open(str(path), "rb") as wav:
  File "/usr/lib/python3.11/wave.py", line 253, in initfp
    raise Error('file does not start with RIFF id')
wave.Error: file does not start with RIFF id

The file exists, so the existence check passes; then wave.open reads the first four bytes looking for the WAV format's RIFF marker and finds MP3 data instead. Two lessons in one traceback. Extension is not format: renaming to .wav would change nothing, since the marker lives in the bytes. And this is the preflight doing its job: the same wrong file handed straight to the model produces a far longer, far stranger error after a two-gigabyte download. Convert properly (ffmpeg -i glados.mp3 -ar 24000 -ac 1 glados.wav) or, better, export your authorized recording as PCM WAV to begin with.

The import mistake from stage 2 fails just as instructively: from f5_tts import F5TTS raises ImportError: cannot import name 'F5TTS' from 'f5_tts'. When a library's README and its package layout disagree, the layout wins; api is where this one keeps its public face.

Checkpoint, in her voice

✓ Checkpoint — what you can now do
  • I can explain F5-TTS conditioning on reference mel frames and text, and name the reference arguments standard inference still needs after fine-tuning.
  • I can state the zero-shot/fine-tune trade in one sentence each, and name the moment in a project where each road is right.
  • I know why the reference transcript must match the audio word for word, and how mismatched words can damage the generated speech.
  • I can defend the preflight pattern: what it costs, what it prevents, and why it runs before the model constructor.
  • I can read "does not start with RIFF id" and know immediately that a file's extension lied about its format.
⚡ Exercises — try first, then reveal
Exercise 1 — reference roulette. Generate the same sentence against three different reference clips: a clean short line, a long rambling one, and one with background ambience. Rank the outputs by ear. What made the difference?

Compare several runs before ranking the clips. A clean short reference often helps, but a particular result need not follow that order. Listen for copied background noise, pronunciation errors and changes in timbre. Record the clip, transcript and settings alongside each result so you can repeat the comparison.

Exercise 2 — the drift experiment. Generate one sentence, then a full paragraph of five or six sentences. Listen to the paragraph's last sentence against the first. What changed, and which road fixes it?

Listen for changes in pitch or timbre without assuming every run drifts. Try a better reference or shorter generated chunks before training. Fine-tuning may help, but judge it on held-out text with the same reference pair and settings. Save the before-and-after audio with a note of what you heard; a checkpoint alone does not prove the problem is fixed.

Exercise 3 — wire her into chapter 3's seat. Give this file a speak(text) -> float that hides the model and reference behind module state, mirroring chapter 3's Piper interface. What would swapping the two engines cost a caller?

Nothing but an import line: both expose text in, audio out, duration returned. That interface symmetry is deliberate and is the whole reason chapter 10 can choose its voice engine with a config flag. When two implementations share a surface, the rest of the program stops caring which one is loaded, a theme volume 6 turns into a discipline with device drivers.