Jetson Production Setup

# labs/jetson_production.py
import subprocess
from dataclasses import dataclass


@dataclass(frozen=True)
class Cmd:
    """One attempt to run a command, with all of its outcomes flattened into data."""
    argv: list[str]
    found: bool
    returncode: int | None
    stdout: str
    stderr: str

    @property
    def ok(self) -> bool:
        return self.found and self.returncode == 0

    def why(self) -> str:
        if not self.found:
            return f"{self.argv[0]}: not on PATH"
        if self.returncode != 0:
            msg = self.stderr.strip() or self.stdout.strip() or "no message"
            return f"{self.argv[0]}: exit {self.returncode}: {msg}"
        return f"{self.argv[0]}: ok"


def run(argv: list[str]) -> Cmd:
    """Never raises. A missing binary is an answer, not an accident."""
    try:
        proc = subprocess.run(argv, text=True, capture_output=True, check=False)
    except FileNotFoundError:
        return Cmd(argv, False, None, "", "")
    return Cmd(argv, True, proc.returncode, proc.stdout, proc.stderr)


if __name__ == "__main__":
    for argv in (["nvpmodel", "-q"], ["nvpmodel", "-m", "9"]):
        print(run(argv).why())

$ ssh glados-jetson python3 -m labs.jetson_production   # on the board
nvpmodel: ok
nvpmodel: exit 255: Error: Invalid power mode 9

$ uv run python -m labs.jetson_production   # the same file, on the laptop
nvpmodel: not on PATH
nvpmodel: not on PATH

Those two transcripts are the whole reason this wrapper exists. subprocess.run has two entirely separate ways of reporting trouble, and only one of them is an exit code. When the binary exists and complains, you get a CompletedProcess with a non-zero returncode, and check=False is what stops that from raising. When the binary is not on PATH at all, no process was ever created, so there is no exit code to check and subprocess raises FileNotFoundError before your code sees anything. Setting check=False does nothing about the second case, and a setup script that only handles the first dies on the laptop at the first Jetson-only command.

A third failure is coming later in the chapter and it does not involve subprocess at all: writing into /etc as an ordinary user raises PermissionError from the filesystem. Missing command, failing command, refusing filesystem. Those are the three, and each one wants its own handler, because a script that catches Exception around all of them can no longer tell you which of the three happened.

# labs/jetson_production.py — continued
import re
from pathlib import Path

NVPMODEL_CONF = Path("/etc/nvpmodel.conf")
NVPMODEL_STATUS = Path("/var/lib/nvpmodel/status")
WANTED_MODE = "MAXN_SUPER"

MODE_LINE = re.compile(r"<\s*POWER_MODEL\s+ID\s*=\s*(\d+)\s+NAME\s*=\s*(\S+)\s*>")


def mode_table(conf: Path = NVPMODEL_CONF) -> dict[str, int]:
    """Every power mode this board declares, by name, read off the board itself."""
    try:
        text = conf.read_text(errors="replace")
    except OSError:
        return {}
    return {name: int(ident) for ident, name in MODE_LINE.findall(text)}


def recorded_mode(status: Path = NVPMODEL_STATUS) -> int | None:
    """The mode nvpmodel wrote down, which is the one the next boot restores."""
    try:
        fields = dict(f.split(":", 1) for f in status.read_text().split())
    except (OSError, ValueError):
        return None
    return int(fields["pmode"]) if "pmode" in fields else None


def set_power_mode(name: str = WANTED_MODE) -> str:
    table = mode_table()
    if not table:
        return "no nvpmodel.conf here, so this is not a Jetson: skipped"
    if name not in table:
        return f"{name} is not a mode this board declares: {sorted(table)}"
    wanted = table[name]
    if recorded_mode() == wanted:
        return f"already {name} (mode {wanted}), recorded for the next boot"
    cmd = run(["nvpmodel", "-m", str(wanted)])
    if not cmd.ok:
        return cmd.why()
    return f"set to {name} (mode {wanted}); recorded mode is now {recorded_mode()}"


if __name__ == "__main__":
    print(f"modes on this board: {mode_table()}")
    print(f"power mode: {set_power_mode()}")

$ ssh glados-jetson sudo python3 -m labs.jetson_production   # first run
modes on this board: {'15W': 0, '7W': 1, 'MAXN_SUPER': 2}
power mode: set to MAXN_SUPER (mode 2); recorded mode is now 2

$ ssh glados-jetson sudo python3 -m labs.jetson_production   # again, after a reboot
modes on this board: {'15W': 0, '7W': 1, 'MAXN_SUPER': 2}
power mode: already MAXN_SUPER (mode 2), recorded for the next boot

The number in nvpmodel -m 2 is the part everybody copies from a forum post and the part that is not portable. Mode identifiers are assigned per board and per JetPack release, so the same 2 means MAXN_SUPER here, something else on an AGX, and nothing at all on a board whose configuration file lists three modes in a different order. The name is stable and the file that maps names to identifiers is already on the disk, so the script reads it. That is the same instinct as chapter 82's detect_platform: ask the machine what it is instead of telling it.

The second run is the one that matters for production. nvpmodel writes the chosen mode into /var/lib/nvpmodel/status, and a service that ships with JetPack reads that file during boot and re-applies it, so the mode survives a power cut without any help from you. That makes this step a one-time configuration and not a boot chore, and it also makes the mode something you can quietly break: one nvpmodel -m 1 typed during an experiment in June is still in effect in August. Reading the recorded value back is how the script tells the difference between a board set correctly and a board that merely was, once.

# labs/jetson_production.py — continued
THERMAL = Path("/sys/devices/virtual/thermal")


def temperatures() -> dict[str, float]:
    """Every thermal zone the kernel exposes, in degrees Celsius."""
    zones: dict[str, float] = {}
    for zone in sorted(THERMAL.glob("thermal_zone*")):
        try:
            name = (zone / "type").read_text().strip()
            milli = int((zone / "temp").read_text().strip())
        except (OSError, ValueError):
            continue
        zones[name] = milli / 1000
    return zones


def trip_c(zone_name: str) -> float | None:
    """The lowest trip point the kernel declares for a zone, in degrees Celsius."""
    for zone in THERMAL.glob("thermal_zone*"):
        try:
            if (zone / "type").read_text().strip() != zone_name:
                continue
            trips = [int(p.read_text()) / 1000 for p in zone.glob("trip_point_*_temp")]
        except (OSError, ValueError):
            continue
        return min(trips) if trips else None
    return None


def thermal_line() -> str:
    zones = temperatures()
    if not zones:
        return "no thermal zones here: skipped"
    name, temp = max(zones.items(), key=lambda kv: kv[1])
    trip = trip_c(name)
    if trip is None:
        return f"hottest {name} {temp:.1f} C, no trip point declared"
    return f"hottest {name} {temp:.1f} C, {trip - temp:.1f} C below the {trip:.1f} C trip point"


if __name__ == "__main__":
    for zone_name, celsius in temperatures().items():
        print(f"{zone_name:<13} {celsius:.1f} C")
    print(f"thermals: {thermal_line()}")

$ ssh glados-jetson python3 -m labs.jetson_production   # measured on the bench, yours will vary
cpu-thermal   46.5 C
gpu-thermal   45.2 C
cv0-thermal   44.9 C
soc2-thermal  47.8 C
tj-thermal    48.1 C
thermals: hottest tj-thermal 48.1 C, 43.9 C below the 92.0 C trip point

The power mode and the temperature answer two different questions and people run them together. MAXN_SUPER is a ceiling you asked for: it tells the board how much power it may draw and how high it may clock. The trip point is a ceiling the board imposes: cross it and the kernel drops clocks until the junction cools, with no log line in your journal and no exception in your Python. On the open bench with a fan over it, this board sat forty degrees clear of that limit. Inside a closed cabinet in a warm room, the margin is the number that changed, and the only symptom you would ever notice is that she started answering more slowly than the benchmark in chapter 80 said she should. Recording the margin the day you install the box gives you something to compare against on the day she feels sluggish.

# labs/jetson_production.py — continued
DECLARED_STORE = Path("/mnt/nvme/ollama/models")          # what the config says
OLLAMA_STORE = Path("/usr/share/ollama/.ollama/models")   # where Ollama really writes


def link_model_store(declared: Path = DECLARED_STORE,
                     store: Path = OLLAMA_STORE) -> str:
    """Make the path the config declares resolve to the directory Ollama uses."""
    if not store.is_dir():
        return f"{store} is missing: is Ollama installed?"
    if declared.is_symlink():
        if declared.resolve() == store.resolve():
            return f"{declared} already resolves to {store}"
        return f"{declared} points at {declared.resolve()} instead: left alone"
    if declared.exists():
        return f"{declared} exists and is not a link: left alone, move it yourself"
    declared.parent.mkdir(parents=True, exist_ok=True)
    declared.symlink_to(store)
    return f"{declared} -> {store}"


if __name__ == "__main__":
    print(f"model store: {link_model_store()}")

$ ssh glados-jetson sudo python3 -m labs.jetson_production   # first run, then immediately again
model store: /mnt/nvme/ollama/models -> /usr/share/ollama/.ollama/models
model store: /mnt/nvme/ollama/models already resolves to /usr/share/ollama/.ollama/models

Chapter 82's override table declares model_storage as /mnt/nvme/ollama/models for this board, and until now no such path existed. Ollama installs as a system user and keeps weights under /usr/share, which on this board is already the NVMe drive JetPack was flashed onto in chapter 76, so nothing here is about speed. It is about there being one path in the project that names where models live. The day a second drive goes into the carrier, one link moves and no Python changes.

Read the guards in order, because their order is the design. is_symlink comes before exists on purpose: exists follows a link and reports on the target, so a link that dangles answers False to exists while still being a link you must not silently overwrite. A real directory sitting at that path means somebody put files there, so the function refuses and says so instead of deleting anything. Running the script five times in a row leaves the machine in the same state as running it once, and no run of it can destroy data. That is the whole meaning of a setup script being safe to re-run.

# labs/jetson_production.py — continued
import socket
import sys

from labs.service_unit import PROJECT_ROOT

DROPIN_DIR = Path("/etc/systemd/system/glados.service.d")
DROPIN_PATH = DROPIN_DIR / "jetson.conf"
ASSERT_MODE = [sys.executable, "-m", "labs.jetson_production", "--assert-mode"]
ON_FAILURE = PROJECT_ROOT / "glados" / "on_failure.py"

DROPIN_TEMPLATE = """# Generated by labs/jetson_production.py on {host}. Do not edit by hand.
[Unit]
StartLimitIntervalSec=120
StartLimitBurst=10
OnFailure=glados-alert.service

[Service]
ExecStartPre={assert_mode}
RestartSec=2
WatchdogSec={watchdog_sec}
NotifyAccess=main
Environment=OLLAMA_MODELS={declared}
"""


def render_dropin(watchdog_sec: int = 90) -> str:
    return DROPIN_TEMPLATE.format(
        host=socket.gethostname(),
        assert_mode=" ".join(ASSERT_MODE),
        watchdog_sec=watchdog_sec,
        declared=DECLARED_STORE,
    )


def install_dropin() -> str:
    content = render_dropin()
    try:
        DROPIN_DIR.mkdir(parents=True, exist_ok=True)
        DROPIN_PATH.write_text(content)
    except PermissionError:
        print(f"\n--- write this to {DROPIN_PATH} yourself ---\n{content}---")
        return "no permission: re-run with sudo, or copy the block above"
    return f"wrote {DROPIN_PATH} ({len(content.splitlines())} lines)"

$ ssh glados-jetson cat /etc/systemd/system/glados.service.d/jetson.conf
# Generated by labs/jetson_production.py on glados-jetson. Do not edit by hand.
[Unit]
StartLimitIntervalSec=120
StartLimitBurst=10
OnFailure=glados-alert.service

[Service]
ExecStartPre=/home/glados/GladOS/.venv/bin/python -m labs.jetson_production --assert-mode
RestartSec=2
WatchdogSec=90
NotifyAccess=main
Environment=OLLAMA_MODELS=/mnt/nvme/ollama/models

Twelve lines, eight of them directives, and every directive is a claim that is only true on this board. The appended ExecStartPre runs after chapter 33's health_check and refuses to let her start on a board that came up in the wrong power mode, so a throttled machine announces itself at boot instead of hiding behind a slow reply. WatchdogSec=90 asks systemd to kill and restart the process if it stops sending heartbeats for a minute and a half. OnFailure names a unit to run when this one gives up, and StartLimitIntervalSec with StartLimitBurst is the pair the worked failure below is about. The PermissionError branch is the third failure kind from stage 1, handled the way a setup script should handle it: print the exact file and its exact contents, so a reader who forgot the sudo can finish by hand in ten seconds.

# glados/on_failure.py — systemd runs this when glados.service gives up
import socket

from labs.alerts import alert

alert("CRITICAL", f"glados.service failed on {socket.gethostname()}")

# labs/jetson_production.py — the entry point
def assert_power_mode(name: str = WANTED_MODE) -> int:
    table = mode_table()
    if not table:
        return 0                       # not a Jetson: nothing to assert
    recorded, wanted = recorded_mode(), table[name]
    if recorded == wanted:
        return 0
    print(f"power mode is {recorded}, expected {wanted} ({name})", file=sys.stderr)
    return 1


def main() -> int:
    if "--assert-mode" in sys.argv:
        return assert_power_mode()
    print("=" * 58)
    print("  Jetson production setup")
    print("=" * 58)
    print(f"  power mode : {set_power_mode()}")
    print(f"  thermals   : {thermal_line()}")
    print(f"  model store: {link_model_store()}")
    print(f"  drop-in    : {install_dropin()}")
    print("-" * 58)
    print("  sudo systemctl daemon-reload")
    print("  sudo systemctl enable --now glados")
    print("  journalctl -u glados -b -f")
    print("=" * 58)
    return 0


if __name__ == "__main__":
    sys.exit(main())

$ ssh glados-jetson sudo /home/glados/GladOS/.venv/bin/python -m labs.jetson_production
==========================================================
  Jetson production setup
==========================================================
  power mode : already MAXN_SUPER (mode 2), recorded for the next boot
  thermals   : hottest tj-thermal 48.1 C, 43.9 C below the 92.0 C trip point
  model store: /mnt/nvme/ollama/models already resolves to /usr/share/ollama/.ollama/models
  drop-in    : wrote /etc/systemd/system/glados.service.d/jetson.conf (12 lines)
----------------------------------------------------------
  sudo systemctl daemon-reload
  sudo systemctl enable --now glados
  journalctl -u glados -b -f
==========================================================

$ ssh glados-jetson sudo reboot; sleep 90; ssh glados-jetson 'systemctl is-enabled glados; systemctl is-active glados; uptime -p'
enabled
active
up 1 minute

The second transcript is the only proof this chapter accepts. Not that the script ran, not that enable printed a symlink, but that the power was interrupted and the box came back by itself with the service running. Do it with the monitor unplugged and the SSH session closed, then walk over and say her name.