Voice cloning
Two roads to her voice
Start with a rights-cleared reference recording and its exact transcript. Zero-shot synthesis uses that pair with pretrained weights, avoiding a training job before the recording has been tested.
F5-TTS is a pretrained voice-cloning model that already learned how human speech works from thousands of hours of generic audio. That prior knowledge unlocks zero-shot cloning: hand it one clean reference clip plus that clip's exact transcript, and it speaks new text in that voice on the first run. No training. The cost is consistency: long outputs can drift in timbre, and rare sounds occasionally come out wrong, because the model is improvising the voice from a single example every time.
Fine-tuning updates the pretrained weights using your authorized dataset. It can improve consistency for that data, but standard F5-TTS inference still uses reference audio and text after fine-tuning. Keep this rule: both routes condition on a reference pair; fine-tuning changes the weights, not the need for that pair. Start with zero-shot to judge the recording before paying for training. Neither route grants rights to a voice, recording or checkpoint. The wrapper below works with compatible fine-tuned checkpoints too.
Preflight cheap, load once, speak forever
# labs/glados_voice.py
import wave
from pathlib import Path
REF_FILE = "glados/data/voice/reference.wav"
REF_TEXT = "This is my reference recording for the local assistant."
def reference_info(ref_file: str) -> dict[str, float]:
path = Path(ref_file)
if not path.exists():
raise FileNotFoundError(f"Reference clip not found: {ref_file}")
with wave.open(str(path), "rb") as wav:
return {"seconds": round(wav.getnframes() / wav.getframerate(), 2),
"sample_rate": float(wav.getframerate())}
if __name__ == "__main__":
info = reference_info(REF_FILE)
print(f"Reference OK: {info['seconds']}s @ {int(info['sample_rate'])} Hz")
$ uv run python labs/glados_voice.py
Reference OK: 4.9s @ 24000 Hz
Use the recording you made or obtained with consent in chapter 6, and change
REF_TEXT if you recorded different words. The retained run-outs used an
earlier reference; their durations are exemplars, not measurements of your new file.
Why check before loading? Because instantiating the cloning model
downloads roughly two gigabytes and pins your machine for tens of seconds, and if
the path is wrong or the file is not really a WAV you want to fail in milliseconds
with a clear message instead. The standard-library wave module reads
just the header: chapter 3 knowledge, now doing gatekeeping duty. Pick your
reference deliberately (a clean four-to-ten-second clip, one consenting voice alone,
no background music) and make sure REF_TEXT matches the audio word
for word, punctuation included. The model aligns the reference audio against that
text to learn the voice, and a sloppy transcript smears everything downstream.
from f5_tts.api import F5TTS
OUTPUT = "glados_says.wav"
def say(tts: F5TTS, text: str) -> float:
wav, sr, _ = tts.infer(
ref_file=REF_FILE,
ref_text=REF_TEXT,
gen_text=text,
file_wave=OUTPUT,
remove_silence=True,
)
return len(wav) / sr
Two details carry weight. The import is f5_tts.api, because the public
interface lives in that submodule; import from the bare package and there is no
F5TTS to be found (the failure box reproduces it).
And infer() returns a three-tuple of waveform, sample rate and
spectrogram; we keep two, discard the spectrogram with _, and return
the duration. Wrapping the call in say() means nothing else in the
program ever has to remember that signature — chapter 10 imports this exact
function.
import sounddevice as sd
import soundfile as sf
def main() -> None:
info = reference_info(REF_FILE)
print(f"Reference OK: {info['seconds']}s @ {int(info['sample_rate'])} Hz")
tts = F5TTS()
print("Type something for GLaDOS to say (blank line to quit).")
while True:
text = input("GLaDOS > ").strip()
if not text:
break
seconds = say(tts, text)
print(f"Generated {seconds:.2f}s of audio. Playing...")
data, samplerate = sf.read(OUTPUT)
sd.play(data, samplerate)
sd.wait()
if __name__ == "__main__":
main()
$ uv run python labs/glados_voice.py
Reference OK: 4.9s @ 24000 Hz
Type something for GLaDOS to say (blank line to quit).
GLaDOS > You found this chapter. I'm genuinely impressed. And a little disappointed.
Generated 5.84s of audio. Playing...
GLaDOS >
The program checks the recording before loading the model, then reuses the loaded instance for each sentence. Measure generation time on your machine.
From here on, much of what this book shows you is the output of neural models, and model output is not byte-stable: your generation times, your drift, even her exact intonation on a given sentence will differ from run to run and machine to machine. The run-outs on these pages are honest captures of a working build; treat them as the pattern to match, not bytes to diff. Numbers that must agree exactly (frame counts, file sizes, error messages) will keep agreeing exactly.
Why this works: conditioning, not memory
F5-TTS conditions generation on reference audio represented as mel-spectrogram frames, a sequence of sound-energy measurements across frequency bands, together with the reference and requested text. It fills the requested audio region using those inputs; this path does not first reduce the voice to a dedicated speaker embedding. A clean recording and matching transcript give the model a consistent audio/text example. Noise, mismatched words and long outputs can still hurt pronunciation or timbre.
Fine-tuning adjusts the network's learned parameters with more examples; it may improve
a target voice or domain and can also overfit poor data. Standard
F5TTS.infer still takes ref_file, ref_text and
gen_text. A compatible checkpoint selected with ckpt_file
leaves say()'s reference arguments in place; its model configuration and
vocabulary must also match. Check the version you installed against the
upstream API
and conditioning implementation.
Compare held-out recordings before deciding training helped. Its duration and success
depend on your dataset and hardware, so an overnight run is no guarantee.
You exported your own recording as glados.mp3, then
pointed REF_FILE at it. The preflight chokes before the model loads:
print(reference_info("glados.mp3"))
$ uv run python labs/fail_ref.py
Traceback (most recent call last):
File "labs/fail_ref.py", line 15, in <module>
print(reference_info("glados.mp3"))
File "labs/fail_ref.py", line 9, in reference_info
with wave.open(str(path), "rb") as wav:
File "/usr/lib/python3.11/wave.py", line 253, in initfp
raise Error('file does not start with RIFF id')
wave.Error: file does not start with RIFF id
The file exists, so the existence check passes; then wave.open reads
the first four bytes looking for the WAV format's RIFF marker and finds
MP3 data instead. Two lessons in one traceback. Extension is not format: renaming
to .wav would change nothing, since the marker lives in the bytes.
And this is the preflight doing its job: the same wrong file handed straight to the
model produces a far longer, far stranger error after a two-gigabyte download.
Convert properly (ffmpeg -i glados.mp3 -ar 24000 -ac 1 glados.wav) or,
better, export your authorized recording as PCM WAV to begin with.
The import mistake from stage 2 fails just as instructively:
from f5_tts import F5TTS raises
ImportError: cannot import name 'F5TTS' from 'f5_tts'. When a library's
README and its package layout disagree, the layout wins; api is where
this one keeps its public face.
Checkpoint, in her voice
- I can explain F5-TTS conditioning on reference mel frames and text, and name the reference arguments standard inference still needs after fine-tuning.
- I can state the zero-shot/fine-tune trade in one sentence each, and name the moment in a project where each road is right.
- I know why the reference transcript must match the audio word for word, and how mismatched words can damage the generated speech.
- I can defend the preflight pattern: what it costs, what it prevents, and why it runs before the model constructor.
- I can read "does not start with RIFF id" and know immediately that a file's extension lied about its format.
Exercise 1 — reference roulette. Generate the same sentence against three different reference clips: a clean short line, a long rambling one, and one with background ambience. Rank the outputs by ear. What made the difference?
Compare several runs before ranking the clips. A clean short reference often helps, but a particular result need not follow that order. Listen for copied background noise, pronunciation errors and changes in timbre. Record the clip, transcript and settings alongside each result so you can repeat the comparison.
Exercise 2 — the drift experiment. Generate one sentence, then a full paragraph of five or six sentences. Listen to the paragraph's last sentence against the first. What changed, and which road fixes it?
Listen for changes in pitch or timbre without assuming every run drifts. Try a better reference or shorter generated chunks before training. Fine-tuning may help, but judge it on held-out text with the same reference pair and settings. Save the before-and-after audio with a note of what you heard; a checkpoint alone does not prove the problem is fixed.
Exercise 3 — wire her into chapter 3's seat. Give this
file a speak(text) -> float that hides the model and reference
behind module state, mirroring chapter 3's Piper interface. What would swapping
the two engines cost a caller?
Nothing but an import line: both expose text in, audio out, duration returned. That interface symmetry is deliberate and is the whole reason chapter 10 can choose its voice engine with a config flag. When two implementations share a surface, the rest of the program stops caring which one is loaded, a theme volume 6 turns into a discipline with device drivers.