Voice recordings

# labs/scrape_stage1.py
from bs4 import BeautifulSoup

# A tiny slice of what a Portal wiki page looks like.
html = """
<a href="https://wiki.example/audio/Glados_escape01.wav">Oh. It's you.</a>
<a href="/w/index.php?title=File:Glados_demo.wav">upload page</a>
<a href="https://wiki.example/audio/Glados_cake02.wav">The cake is a lie.</a>
"""
soup = BeautifulSoup(html, "html.parser")
for a in soup.find_all("a", href=True):
    print(a["href"])

$ uv run python labs/scrape_stage1.py
https://wiki.example/audio/Glados_escape01.wav
/w/index.php?title=File:Glados_demo.wav
https://wiki.example/audio/Glados_cake02.wav

uv add requests beautifulsoup4 first. We start against a three-line stand-in for the real page, because before filtering anything you have to see what pages actually contain. find_all("a", href=True) returns every anchor tag that has an href; the tag's URL is a["href"] and its visible text is a.get_text(strip=True). Look at the middle line: it ends in .wav and it is an upload page, not audio. That one link is the whole reason the next stage exists.

links = []
for a in soup.find_all("a", href=True):
    href = a["href"]
    if href.endswith(".wav") and href.startswith("http"):
        links.append((href, a.get_text(strip=True)))

print(f"Kept {len(links)} of {len(soup.find_all('a', href=True))} links")
for href, transcript in links:
    print(f"  {transcript!r} -> {href}")

$ uv run python labs/scrape_stage2.py
Kept 2 of 3 links
  "Oh. It's you." -> https://wiki.example/audio/Glados_escape01.wav
  'The cake is a lie.' -> https://wiki.example/audio/Glados_cake02.wav

Why two conditions? endswith(".wav") alone keeps the upload page; adding startswith("http") demands an absolute URL, the only kind a downloader can fetch. And the append stores a pair: URL and link text, because on these pages the link text is the transcript, and the (clip, transcript) pair is the dataset row chapter 7 needs. Scrape the label at the same moment you scrape the file; going back for transcripts later is misery.

import csv
from pathlib import Path

def load_existing(csv_path: Path) -> set[str]:
    if not csv_path.exists():
        return set()
    with open(csv_path, newline="") as f:
        reader = csv.reader(f, delimiter="|")
        return {row[0] for row in reader if row and row[0] != "filename"}

csv_path = Path("glados/data/metadata.csv")
existing = load_existing(csv_path)
print(f"Already have: {sorted(existing)}")
for name in ["Glados_escape01.wav", "Glados_turret07.wav"]:
    if name in existing:
        print(f"  skip {name}")
    else:
        print(f"  + new {name}")

$ uv run python labs/scrape_stage3.py   # after a previous partial run
Already have: ['Glados_cake02.wav', 'Glados_escape01.wav']
  skip Glados_escape01.wav
  + new Glados_turret07.wav

The manifest, metadata.csv, is a pipe-delimited filename|transcript file, and it is the source of truth for "what made it into the dataset." Loading it into a set gives instant membership checks, and the header-row filter keeps the literal word filename out of your skip list. Now a crash mid-run costs nothing: the next run reads the manifest and resumes where the last one stopped. Chapter 1 called this idempotence, on six directories; here it earns its keep on three hundred network requests.

# labs/scrape_glados.py
import argparse, csv, time, urllib.request
from pathlib import Path
from urllib.parse import urlparse
import requests
from bs4 import BeautifulSoup

def scrape_wav_links(url: str) -> list[tuple[str, str]]:
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    return [(a["href"], a.get_text(strip=True))
            for a in soup.find_all("a", href=True)
            if a["href"].endswith(".wav") and a["href"].startswith("http")]

def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--urls", nargs="+", required=True)
    p.add_argument("--output", type=Path, default=Path("glados/data/voice"))
    args = p.parse_args()
    args.output.mkdir(parents=True, exist_ok=True)
    csv_path = args.output / "metadata.csv"
    existing = load_existing(csv_path)

    new_rows = 0
    with open(csv_path, "a", newline="") as f:
        writer = csv.writer(f, delimiter="|")
        if not existing:
            writer.writerow(["filename", "transcript"])
        for url in args.urls:
            for href, transcript in scrape_wav_links(url):
                name = Path(urlparse(href).path).name
                if name in existing or not transcript:
                    continue
                urllib.request.urlretrieve(href, args.output / name)
                writer.writerow([name, transcript])
                existing.add(name)
                new_rows += 1
                time.sleep(0.1)   # be polite to the wiki
    print(f"Added {new_rows} clips; manifest now {len(existing)} entries.")

if __name__ == "__main__":
    main()

$ uv run python labs/scrape_glados.py --urls https://theportalwiki.com/wiki/GLaDOS_voice_lines
Added 312 clips; manifest now 312 entries.

This retained capture illustrates the earlier wiki import, not permission to repeat it. Point --urls at your authorized source, or use the self-recorded manifest without running the scraper. (load_existing comes along from stage 3.) One clean reference pair is enough to try zero-shot synthesis. A larger authorized set can support fine-tuning, but clip count alone says nothing about recording quality. The time.sleep(0.1) between downloads illustrates pacing; it does not establish an acceptable rate for every site. Follow the source's access rules and stop if the server refuses or rate-limits requests.