A Monitor That Outlives Startup

# labs/health_monitor.py
import shutil
from dataclasses import dataclass

DISK_WARN_PCT = 85.0
GIB = 1024 ** 3

@dataclass(frozen=True)
class HealthStatus:
    name: str
    ok: bool
    detail: str
    value: float | None = None

def check_disk(path: str = "/") -> HealthStatus:
    usage = shutil.disk_usage(path)
    pct = usage.used / usage.total * 100
    return HealthStatus("disk", pct < DISK_WARN_PCT,
                        f"{pct:.1f}% used, {usage.free / GIB:.1f} GiB free",
                        round(pct, 1))

if __name__ == "__main__":
    print(check_disk())

$ uv run python -m labs.health_monitor   # your drive will read differently
HealthStatus(name='disk', ok=True, detail='71.2% used, 67.1 GiB free', value=71.2)

Chapter 33's CheckResult carried a name, a verdict and a message, and for a probe that runs once that was enough. A standing monitor needs a fourth field, because detail is a sentence for a human and you cannot do arithmetic on a sentence. value is the same reading as a number, and every trend in this chapter is computed from it. It defaults to None for a reason the next stage makes concrete.

frozen=True costs one word and buys something specific. A status is a record of one instant, and it is about to be appended to a history that other code will read minutes later; freezing it means nothing downstream can quietly edit the past. Note also that the threshold lives in a named constant and the verdict is computed from it. That comparison is the seam every policy in this chapter reads.

import json
import urllib.request

MEM_WARN_PCT = 85.0
OLLAMA_TAGS_URL = "http://127.0.0.1:11434/api/tags"
PROBE_TIMEOUT = 3.0

def check_memory(path: str = "/proc/meminfo") -> HealthStatus:
    try:
        info: dict[str, int] = {}
        with open(path) as f:
            for line in f:
                parts = line.split()
                if len(parts) >= 2:
                    info[parts[0].rstrip(":")] = int(parts[1])
        total, available = info["MemTotal"], info["MemAvailable"]
    except (OSError, KeyError, ValueError) as e:
        return HealthStatus("memory", False, f"unreadable: {e.__class__.__name__}: {e}")
    pct = (total - available) / total * 100
    return HealthStatus("memory", pct < MEM_WARN_PCT,
                        f"{pct:.1f}% used, {available / (1024 * 1024):.1f} GiB available",
                        round(pct, 1))

def check_ollama(url: str = OLLAMA_TAGS_URL) -> HealthStatus:
    try:
        with urllib.request.urlopen(url, timeout=PROBE_TIMEOUT) as reply:
            models = json.loads(reply.read()).get("models", [])
    except Exception as e:
        return HealthStatus("ollama", False, f"{e.__class__.__name__}: {e}")
    return HealthStatus("ollama", True, f"{len(models)} model(s) available",
                        float(len(models)))

$ uv run python -m labs.health_monitor   # your box, your model list
HealthStatus(name='memory', ok=True, detail='41.8% used, 4.3 GiB available', value=41.8)
HealthStatus(name='ollama', ok=True, detail='6 model(s) available', value=6.0)

$ sudo systemctl stop ollama; uv run python -m labs.health_monitor
HealthStatus(name='ollama', ok=False, detail='URLError: <urlopen error [Errno 111] Connection refused>', value=None)

Read MemAvailable, never MemFree. Linux spends every spare page on cache and gives it back the instant a program wants it, so MemFree on a healthy machine that has been up for a week is a number near zero that means nothing at all. MemAvailable is the kernel's own estimate of what a new process could actually get, which is the only figure a memory alarm should ever compare against.

The two except clauses differ on purpose. Reading a file has a small, nameable set of ways to go wrong, so check_memory lists them and lets a genuine bug in its own parsing crash loudly during development. An HTTP call to another process has an open-ended set: refused connections, timeouts, resets, redirects, HTML error pages that fail to parse as JSON. Enumerating those is a losing game, and for a probe, any failure to determine health is itself the health signal. Note what the failed status carries: a detail string and value=None. A failed check has no measurement. Writing 0.0 there would put a fake reading into the history and drag every average and every trend toward a number nothing ever measured.

import time

CHECKS = (check_ollama, check_disk, check_memory)

def run_checks() -> list[HealthStatus]:
    return [check() for check in CHECKS]

def render(checks: list[HealthStatus], at: float) -> str:
    lines = [f"[{time.strftime('%H:%M:%S', time.localtime(at))}] health"]
    for c in checks:
        lines.append(f"  {'ok  ' if c.ok else 'FAIL'} {c.name:<7} {c.detail}")
    failed = [c.name for c in checks if not c.ok]
    lines.append(f"  degraded: {', '.join(failed)}" if failed else "  all nominal")
    return "\n".join(lines)

if __name__ == "__main__":
    print(render(run_checks(), time.time()))

$ uv run python -m labs.health_monitor   # with ollama stopped
[21:04:11] health
  FAIL ollama  URLError: <urlopen error [Errno 111] Connection refused>
  ok   disk    71.2% used, 67.1 GiB free
  ok   memory  41.8% used, 4.3 GiB available
  degraded: ollama

That list comprehension is the whole argument for returning failure as a value. The model server is down, and the disk and memory readings are still on the screen. Let check_ollama raise instead and the comprehension unwinds on its first element, the other two checks never execute, and the one moment you most need a full picture of the machine is the moment the tool gives you a traceback about one part of it.

render touches three fields and knows nothing about sockets, /proc, or partitions. Adding a fourth check is one function and one entry in CHECKS; the renderer, the history and the alerting below all keep working untouched. That is the payoff for making every check return the same type.

from collections import deque

HISTORY_POLLS = 720      # 12 hours at one poll a minute
MIN_TREND_HOURS = 2.0    # below this a slope is noise

class History:
    def __init__(self, keep: int = HISTORY_POLLS) -> None:
        self._keep = keep
        self._series: dict[str, deque[tuple[float, float]]] = {}

    def record(self, at: float, status: HealthStatus) -> None:
        if status.value is None:
            return
        self._series.setdefault(status.name, deque(maxlen=self._keep)).append((at, status.value))

    def latest(self, name: str) -> float | None:
        series = self._series.get(name)
        return series[-1][1] if series else None

    def rate_per_hour(self, name: str) -> float | None:
        series = self._series.get(name)
        if series is None or len(series) < 2:
            return None
        (t0, v0), (t1, v1) = series[0], series[-1]
        hours = (t1 - t0) / 3600
        if hours < MIN_TREND_HOURS:
            return None
        return (v1 - v0) / hours

    def hours_until(self, name: str, limit: float) -> float | None:
        rate = self.rate_per_hour(name)
        if rate is None or rate <= 0:
            return None
        return (limit - self._series[name][-1][1]) / rate

if __name__ == "__main__":
    h, at = History(), 1_756_000_000.0
    for i in range(12):                       # twelve hourly readings, half a point apart
        pct = 88.0 + 0.5 * i
        h.record(at + i * 3600,
                 HealthStatus("disk", pct < DISK_WARN_PCT, f"{pct:.1f}% used", pct))
    print(f"latest        {h.latest('disk'):.1f} %")
    print(f"rate          {h.rate_per_hour('disk'):+.2f} %/hour")
    print(f"hours to 100  {h.hours_until('disk', 100.0):.1f}")

$ uv run python -m labs.health_monitor
latest        93.5 %
rate          +0.50 %/hour
hours to 100  13.0

Every number there is checkable by hand. Twelve readings from 88.0 rising by 0.5 each end at 93.5, spread across eleven hours, so the slope is 5.5 divided by 11, exactly half a point an hour. The remaining 6.5 points at that rate is thirteen hours. That last line is the sentence chapter 33's probe could never produce and a restart can never produce: not "the disk is fine", but "the disk stops working before dinner".

One deque per check name, each capped at 720 entries, is chapter 49's ring buffer doing the same job for a different measurement. record drops a status whose value is None, so an hour of connection failures leaves a gap in the series and no false readings in it. MIN_TREND_HOURS is the guard that keeps the monitor quiet for its first two hours after a restart: a slope drawn through four minutes of readings rounded to one decimal place is mostly rounding, and it will happily predict a full disk by teatime. Two endpoints are the crudest possible slope, and one spike at either end tilts the whole line. Averaging the first five and the last five readings costs three more lines and fixes that; the demo above answers identically either way, so make the change when your readings are noisy rather than because it looks more serious.

@dataclass(frozen=True)
class Policy:
    fail_polls: int                     # consecutive failures before this counts as a fault
    clear_polls: int                    # consecutive passes before the fault is over
    level: str                          # severity when it fires
    limit: float | None = None          # the value at which the subsystem stops working
    urgent_hours: float | None = None   # inside this many hours of the limit, wake someone

POLICIES = {
    "ollama": Policy(fail_polls=2, clear_polls=2, level="CRITICAL"),
    "disk":   Policy(fail_polls=5, clear_polls=5, level="WARNING",
                     limit=100.0, urgent_hours=24.0),
    "memory": Policy(fail_polls=5, clear_polls=5, level="WARNING",
                     limit=100.0, urgent_hours=2.0),
}
DEFAULT_POLICY = Policy(fail_polls=3, clear_polls=3, level="WARNING")

class Debouncer:
    """Turns a stream of per-poll verdicts into rare state changes."""

    def __init__(self, policies: dict[str, Policy]) -> None:
        self._policies = policies
        self._streak: dict[str, int] = {}
        self._faulted: dict[str, bool] = {}

    def faulted(self, name: str) -> bool:
        return self._faulted.get(name, False)

    def update(self, status: HealthStatus) -> str | None:
        policy = self._policies.get(status.name, DEFAULT_POLICY)
        prior = self._streak.get(status.name, 0)
        streak = max(prior, 0) + 1 if status.ok else min(prior, 0) - 1
        self._streak[status.name] = streak
        if not self.faulted(status.name) and -streak >= policy.fail_polls:
            self._faulted[status.name] = True
            return "fault"
        if self.faulted(status.name) and streak >= policy.clear_polls:
            self._faulted[status.name] = False
            return "clear"
        return None

FLAPPING = (84.6, 85.4, 84.9, 85.7, 84.8, 86.2, 87.4, 88.1, 89.0, 88.6,
            84.9, 83.5, 83.9, 83.1, 82.8)

if __name__ == "__main__":
    d, raw = Debouncer(POLICIES), 0
    print("poll  memory  raw    event")
    for i, pct in enumerate(FLAPPING, start=1):
        status = HealthStatus("memory", pct < MEM_WARN_PCT, f"{pct:.1f}% used", pct)
        event = d.update(status)
        raw += 0 if status.ok else 1
        print(f"{i:>4}  {pct:>6.1f}  {'ok' if status.ok else 'FAIL':<5}  {event or '-'}")
    print(f"\nmessages without a debouncer: {raw}")

$ uv run python -m labs.health_monitor
poll  memory  raw    event
   1    84.6  ok     -
   2    85.4  FAIL   -
   3    84.9  ok     -
   4    85.7  FAIL   -
   5    84.8  ok     -
   6    86.2  FAIL   -
   7    87.4  FAIL   -
   8    88.1  FAIL   -
   9    89.0  FAIL   -
  10    88.6  FAIL   fault
  11    84.9  ok     -
  12    83.5  ok     -
  13    83.9  ok     -
  14    83.1  ok     -
  15    82.8  ok     clear

messages without a debouncer: 7

Fifteen minutes of readings, seven of them over the line, and exactly two events. The trick is the signed streak: a passing poll makes it positive and counts up, a failing poll makes it negative and counts down, and either one resets the other. Polls 2 and 4 are real threshold crossings and produce nothing, because a single sample on the far side of a line is not evidence of anything. Polls 6 through 10 are five failures in a row, and the fifth is the one the policy was waiting for.

fail_polls and clear_polls are separate numbers so you can make one direction harder than the other. They are equal here and unequal in most real deployments, where a fault clears slowly on purpose so a subsystem that recovers for ninety seconds and dies again does not send you a recovery notice it has to take back. The cost of all this is stated plainly: five polls at sixty seconds each means the monitor is five minutes late to every real fault. That is the trade, and ollama takes the other side of it with fail_polls=2, because two minutes of her being unable to answer at all matters more than a rare false alarm.

# labs/health_monitor.py — the standing monitor
from labs.alerts import LEVEL_ORDER, alert

POLL_SECONDS = 60.0
HEARTBEAT_POLLS = 60

def judge(status: HealthStatus, event: str, history: History) -> tuple[str, str]:
    policy = POLICIES.get(status.name, DEFAULT_POLICY)
    if event == "clear":
        return "INFO", f"{status.name} recovered: {status.detail}"
    level, parts = policy.level, [f"{status.name} degraded: {status.detail}"]
    rate = history.rate_per_hour(status.name)
    if rate is not None:
        parts.append(f"trending {rate:+.2f}/hour")
    hours = history.hours_until(status.name, policy.limit) if policy.limit else None
    if hours is not None:
        hours = round(hours, 1)
        parts.append(f"{hours} h from {policy.limit:.0f}")
        if policy.urgent_hours is not None and hours < policy.urgent_hours:
            level = "CRITICAL"
    return level, ", ".join(parts)

class Monitor:
    """History, hysteresis and severity for one machine."""

    def __init__(self, policies: dict[str, Policy] | None = None) -> None:
        self.history = History()
        self.debounce = Debouncer(policies or POLICIES)
        self._sent: dict[str, str] = {}

    def observe(self, at: float, status: HealthStatus) -> tuple[str, str] | None:
        self.history.record(at, status)
        event = self.debounce.update(status)
        if event == "clear":
            self._sent.pop(status.name, None)
            return judge(status, "clear", self.history)
        if not self.debounce.faulted(status.name):
            return None
        level, message = judge(status, "fault", self.history)
        sent = self._sent.get(status.name)
        if sent is not None and LEVEL_ORDER.index(level) <= LEVEL_ORDER.index(sent):
            return None
        self._sent[status.name] = level
        return level, message

    def poll(self, at: float) -> tuple[list[HealthStatus], int]:
        checks, spoken = run_checks(), 0
        for status in checks:
            verdict = self.observe(at, status)
            if verdict is not None:
                alert(*verdict)
                spoken += 1
        return checks, spoken

def main(interval: float = POLL_SECONDS) -> None:
    monitor, n = Monitor(), 0
    alert("INFO", f"health monitor up, polling every {interval:.0f}s")
    while True:
        at = time.time()
        checks, spoken = monitor.poll(at)
        n += 1
        if spoken or n % HEARTBEAT_POLLS == 1:
            print(render(checks, at), flush=True)
        time.sleep(max(0.0, interval - (time.time() - at)))

if __name__ == "__main__":
    main()

observe holds the rule that decides who gets woken. A fault sends one message at the policy's level. While it stays faulted it sends nothing more, unless the level it would send now outranks the level already sent, and the ranking comes from chapter 16's ordered list of severities. So a disk that crosses its threshold reports once at WARNING, goes quiet, and speaks a second time only when the trend says the drive is full inside a day. Two messages for one problem, hours apart, each one carrying information the previous one did not have. Note the round before the comparison: the number printed in the message is the same number the rule tested, so no one has to work out why a log line reading 24.0 hours escalated under a twenty-four hour threshold.

Two details in the loop earn their lines. HEARTBEAT_POLLS prints one full report an hour when nothing is happening, so a quiet journal still proves the monitor is running; without it, a monitor that silently died and a monitor with nothing to report look identical from the outside. The sleep subtracts the time the poll itself took, because otherwise a three-second Ollama timeout makes every cycle 63 seconds long, and a day that should hold 1,440 samples holds about 1,370 instead. Sleeping for the interval and sleeping until the next interval are different instructions, and only one of them keeps a schedule.

# /etc/systemd/system/glados-monitor.service
[Unit]
Description=GLaDOS health monitor
After=network-online.target

[Service]
Type=simple
User=glados
WorkingDirectory=/home/glados/GladOS
ExecStart=/home/glados/.local/bin/uv run python -m labs.health_monitor
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

$ ssh glados-jetson journalctl -u glados-monitor --since '2026-08-21 20:00' -o cat   # captured on her Jetson; yours will differ
[20:00:07] health
  ok   ollama  6 model(s) available
  ok   disk    84.6% used, 35.9 GiB free
  ok   memory  44.1% used, 4.1 GiB available
  all nominal
[WARNING] disk degraded: 85.0% used, 34.9 GiB free, trending +0.50/hour, 30.0 h from 100
[20:53:00] health
  ok   ollama  6 model(s) available
  FAIL disk    85.0% used, 34.9 GiB free
  ok   memory  43.8% used, 4.2 GiB available
  degraded: disk
[CRITICAL] disk degraded: 88.1% used, 27.7 GiB free, trending +0.50/hour, 23.8 h from 100
[03:02:00] health
  ok   ollama  6 model(s) available
  FAIL disk    88.1% used, 27.7 GiB free
  ok   memory  41.6% used, 4.3 GiB available
  degraded: disk

Six hours between those two alerts, and both are checkable against the numbers in them: 15.0 points below the limit at half a point an hour is 30.0 hours, and 11.9 points is 23.8. Between them the monitor polled roughly 370 times and said nothing, because nothing it learned in those six hours changed what a person would do about it. That silence is the feature. The line that finally wakes someone arrives with a deadline attached, at an hour when there is still time to act on it.