Chapter 16 — Calling for Help

The failures you never see

She runs unattended for hours now, waiting on her name. When the wake word fires you are standing right there, and problems announce themselves. The dangerous moments are the other kind: the Ollama connection dropping at 2 a.m., a sensor going quiet, an automation looping on an error while the house sleeps. If nothing records those events and nothing surfaces them, you learn about the failure days later, when you ask a question and get silence, with no trail explaining when the silence started.

The reflex fix is print() everywhere, and it fails three ways. Prints vanish when the terminal closes, so there is no record. Every print looks equally important, so the critical drowns in the chatty. And a print cannot escalate: a dead language model at 2 a.m. should reach your phone, not scroll past a console nobody is watching. Noise, no signal, no reach.

The fix has two organs and one principle. Python's logging module persists every event with a severity level; smtplib escalates the rare serious ones by email; and routing is data: a config dict decides which levels escalate and whether escalation is armed at all, so the same alert() call behaves differently in development and production without one line of logic changing. Chapter 9 made her character data. Today her panic threshold gets the same treatment.

Log, gate, route

▣ Build · stage 1 — every event, to file and console at once

# labs/alerts.py
import logging
from pathlib import Path

LOG_PATH = Path("glados/data/alerts.log")
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)

logging.basicConfig(
    level=logging.DEBUG,
    format="[%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler(LOG_PATH),
        logging.StreamHandler(),
    ],
)
logger = logging.getLogger("glados.alerts")

logger.info("GLaDOS system started.")
logger.warning("Wake word confidence below threshold.")
logger.critical("Ollama connection lost.")

$ uv run python labs/alerts.py
[INFO] GLaDOS system started.
[WARNING] Wake word confidence below threshold.
[CRITICAL] Ollama connection lost.

One basicConfig call fans every record out to two handlers: the FileHandler is the durable record you grep after the fact, and the StreamHandler is the live view while you develop. The mkdir runs before the first write because a FileHandler pointed at a missing directory raises on a fresh checkout, exactly the kind of first-run failure chapter 9's loader taught you to design away. And each message now carries a level, which is the entire upgrade over print: importance became machine-readable.

▣ Build · stage 2 — email, gated before any network I/O

import smtplib
from email.mime.text import MIMEText

config = {
    "enabled": False,
    "smtp_host": "smtp.gmail.com",
    "smtp_port": 587,
    "username": "you@example.com",
    "password": "app-password",
    "to": "you@example.com",
    "min_level": "CRITICAL",
}

def send_email_alert(subject: str, body: str) -> None:
    if not config["enabled"]:
        logger.debug("Email disabled; skipping send of %r.", subject)
        return
    msg = MIMEText(body)
    msg["Subject"] = f"[GLaDOS Alert] {subject}"
    msg["From"] = config["username"]
    msg["To"] = config["to"]
    with smtplib.SMTP(config["smtp_host"], config["smtp_port"]) as server:
        server.starttls()
        server.login(config["username"], config["password"])
        server.send_message(msg)

$ uv run python labs/alerts.py
[DEBUG] Email disabled; skipping send of 'CRITICAL: Ollama down'.

The guard is the first line on purpose: with enabled: False the function returns before a message is constructed or a socket is opened, so development needs no credentials and touches no network. When you do arm it, use an app-specific password from your mail provider, never your real one, and load the secret from an environment variable rather than the file you are about to commit. The skip is logged at DEBUG: visible when you want the full story, silent in the everyday view. Even declining to act is an event.

▣ Build · stage 3 — one call, routed by rank

LEVEL_ORDER = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]

def alert(level: str, message: str) -> None:
    level = level.upper()
    log_fn = getattr(logger, level.lower(), logger.info)
    log_fn(message)
    min_idx = LEVEL_ORDER.index(config["min_level"])
    cur_idx = LEVEL_ORDER.index(level) if level in LEVEL_ORDER else 0
    if cur_idx >= min_idx:
        send_email_alert(f"{level}: {message[:60]}", message)

if __name__ == "__main__":
    alert("INFO", "GLaDOS system started.")
    alert("WARNING", "Wake word confidence below threshold.")
    alert("CRITICAL", "Ollama connection lost - LLM unavailable.")

$ uv run python labs/alerts.py
[INFO] GLaDOS system started.
[WARNING] Wake word confidence below threshold.
[CRITICAL] Ollama connection lost - LLM unavailable.
[DEBUG] Email disabled; skipping send of 'CRITICAL: Ollama connection lost - LLM unavai'.

Read the output as a routing table settling: three alerts logged, and only the CRITICAL one attempted escalation (then hit the disabled gate and said so). The comparison works on list positions, because severity is an ordering, and .index() turns names into ranks so "at least as serious as the threshold" becomes a numeric >=. The getattr picks the matching logger method with logger.info as the fallback for an unknown level; misuse degrades politely instead of crashing the caller, which is the correct temperament for infrastructure that runs while you sleep.

Why this works: severity is an ordinal

Levels look like labels and behave like numbers. DEBUG through CRITICAL is an ordered scale, and every routing question ("log it? escalate it? page a phone?") reduces to comparing a rank against a threshold. Keeping the order in one list, LEVEL_ORDER, makes the scale explicit and editable; keeping the threshold in config makes the policy deployable. Development runs with escalation off and DEBUG visible. Production runs with escalation armed at CRITICAL. Same code, different dict, and that separation is why alerting systems from this forty lines up to industrial pagers all have the same skeleton. Observability Zero takes this idea to fleets; her single machine only needs the honest core.

⚠ Worked failure — the alert that alerted no one

Escalation armed, threshold set, and a config edit later the 2 a.m. email never comes. The log tells the story the next morning:

config["min_level"] = "critical"   # edited by hand, lowercase

$ uv run python labs/alerts.py
Traceback (most recent call last):
  File "labs/alerts.py", line 44, in alert
    min_idx = LEVEL_ORDER.index(config["min_level"])
ValueError: 'critical' is not in list

.index() is case-sensitive and the list holds uppercase, so the first alert of any level crashes the alert system itself: the watchdog is the thing that died. The one-line fix is normalizing at the boundary, config["min_level"].upper(), and the deeper habit is the one this book keeps returning to: every value that crosses from config into code gets validated or normalized at the crossing. Chapter 33's startup validation turns that habit into a checklist she runs on herself at boot, before anything else trusts the file.

Checkpoint, on call

✓ Checkpoint — what you can now do

I can name the three ways print fails as an alerting system and which organ replaces each.
I can explain why the email guard is the function's first line and what it buys during development.
I can trace an alert through the rank comparison and predict which levels escalate at any threshold.
I know why the skip is logged at DEBUG instead of silently returning.
I normalize config values at the boundary, because I have seen the watchdog die of a lowercase letter.

⚡ Exercises — try first, then reveal

Exercise 1 — wire it to the bus. Subscribe alert to a new system_error event and publish one from the voice loop's exception handler. What did the bus buy you here?

The loop knows nothing about logging, email, or thresholds; it shouts system_error and walks away, exactly as chapter 15 promised. Swap the alert layer, add a second listener that flashes a light, and the loop never changes. Failure reporting became a subscriber, which is where it belongs.

Exercise 2 — a real escalation, once. Arm the config with an app password, set the threshold to WARNING, and trigger one. Then put everything back. What arrived, and what would 3 a.m. you want changed about it?

An email with the level, the first sixty characters, and the full message. Most people immediately want two edits: a timestamp in the body and the machine's name in the subject, both one-line changes. The exercise's real lesson is the drill itself; an escalation path you have never tested is a wish, not a system.

Exercise 3 — the flood. Put an alert inside a loop that fails every two seconds and let it run five minutes. Read the log, then your inbox. What is missing from this chapter's design?

Rate limiting: 150 identical CRITICALs and, if armed, 150 emails. The fix is deduplication with a cooldown, remembering the last send time per message and suppressing repeats inside the window, plus a counter so the log still shows the true rate. That upgrade is genuinely needed, arrives with the watchdog in chapter 41, and now you know why it exists before you meet it.

She keeps records, and she can raise her voice when something breaks. So far, though, everything she does is a reaction to you. The next chapter starts the other half of an assistant's job: automations, rules that fire on their own, written as data so that adding the twenty-first is as safe as adding the first.