GLaDOS Vol 8 · The Jetson Brain
ch 76 / 99
Chapter 76

Flashing JetPack

Flash successful, and nothing proven yet

The last dialog SDK Manager shows you is a green tick and the words Flash successful. Read that sentence for what it actually asserts. A program on your desktop pushed a few gigabytes down a USB cable, the bootloader on the far end acknowledged each transfer, and nothing errored. That is a claim about a conversation between two machines. It says nothing about which operating system the board comes up on, which CUDA is on its disk, or whether the compiler you need is installed at all.

The gap between those two things is not academic. A board can flash cleanly and still boot an older image off a microSD card somebody left in the slot. A flash can put the runtime libraries down and skip the toolkit, and you will not find out until an import three weeks from now fails with a message about a driver version that means nothing to you. By then the flash is ancient history and you are debugging the wrong layer.

Everything you need to know about a flash was written to a file on the board. The release banner, the CUDA version, the compiler binary: all of it sits at known paths in plain text, put there by the installer. So the rule for this chapter: a flash is finished when a script has read the board's own files back and printed each detected value beside the value it was supposed to be, and every row agrees. Not when the progress bar fills. When the report is clean.

◆ Note — the bench for this volume, and how to read its output

A Jetson Orin Nano Super developer kit, a 500 GB NVMe SSD in the M.2 slot, the barrel-jack supply that came in the box, and a desktop running Ubuntu 22.04 on an x86_64 processor to drive the flash. JetPack 6.2.2 throughout. Every command output below came off that one bench once. Version strings move between JetPack point releases, revision numbers differ, and your board will print its own; the prose says so wherever the exact digits matter.

What the host writes, and where it lands

The Jetson cannot flash itself. Rewriting the storage a machine is running from is a problem no running machine solves cleanly, so NVIDIA puts the job on a second computer. Bridge the pin marked FC REC to the ground pin beside it, apply power, and the module skips its normal boot and waits on the USB-C port for someone to tell it what to write. That is recovery mode. Two details trip people up on the first attempt: the jumper has to be in place before power arrives, and the USB-C port is a data port only, so power still comes from the barrel jack.

The host tool is SDK Manager, an NVIDIA application that runs on x86_64 Ubuntu and nowhere else. No Windows, no macOS, and no arm64. Before clicking anything in it, ask the host whether the board is even listening.

▣ Build · stage 1 — proving recovery mode from the host
# on the x86_64 Ubuntu host, with the jumper in place and the board powered on
lsusb | grep -i nvidia
$ lsusb | grep -i nvidia   # captured on the bench — your bus and device numbers will differ
Bus 001 Device 014: ID 0955:7523 NVIDIA Corp. APX

0955 is NVIDIA's USB vendor identifier and APX is the name the recovery bootloader answers to. The product identifier after the colon varies by module generation, so match on the vendor and the APX label and ignore the rest. An empty result means the board is booting normally, the jumper went on after power, or the cable is a charge-only one. Sorting that out now takes a minute; sorting it out after SDK Manager has sat at "waiting for target" for a quarter of an hour does not.

What happens next arrives in two phases, and keeping them separate in your head explains most of what can go wrong. Phase one is the flash proper. The host streams a bootloader chain into the QSPI-NOR memory soldered onto the module itself, and streams a root filesystem, a kernel and a handful of small partitions onto whatever external media you selected, the NVMe SSD in this build. Then the board reboots into the image it was just given.

Phase two happens over the network, to a board that is already running. SDK Manager logs in and installs the SDK components as ordinary Debian packages: CUDA, cuDNN, TensorRT, the developer tools. This is where a flash quietly comes up short. Phase one can succeed perfectly while phase two is skipped, interrupted, or run with half the components unticked, and the board still boots, still shows a desktop, and still looks finished.

One flash, two phases, three destinations The x86 host writes over USB in phase one to two places: the bootloader chain into QSPI-NOR memory on the module, and the root filesystem and kernel onto the NVMe SSD. In phase two the host installs CUDA, cuDNN and TensorRT over the network onto the board that is now running. Each destination is a separate chance to end up incomplete. PHASE 1 OVER USB · PHASE 2 OVER THE NETWORK x86 host SDK Manager USB QSPI-NOR bootloader chain NVMe SSD rootfs + kernel apt packages CUDA · cuDNN · TensorRT SSH, board already booted files at /etc/nv_tegra_release files under /usr/local/cuda each one a claim you can read back
Figure 76.1 — Three destinations, written by two different mechanisms at two different times. A single "flash successful" covers the first two at best, and the dashed path is the one that ends up half done.

Four claims, read off the board

Log in on the Jetson and the first file to open is the release banner the installer wrote. Read it before parsing it, because the parser you write next depends on its exact punctuation.

▣ Build · stage 2 — the banner, and a comparable version out of it
# labs/jetpack_verify.py
import re
from pathlib import Path

TEGRA_RELEASE = Path("/etc/nv_tegra_release")


def read_text(path: Path) -> str | None:
    """The file's contents, or None if this board has no such file."""
    try:
        return path.read_text().strip()
    except OSError:
        return None


def l4t_version(banner: str) -> str:
    """Turn the release banner into something comparable, like 'R36.4.3'."""
    match = re.search(r"R(\d+) \(release\), REVISION: (\d+\.\d+)", banner)
    if match is None:
        raise ValueError(f"unreadable release banner: {banner[:48]!r}")
    return f"R{match.group(1)}.{match.group(2)}"


if __name__ == "__main__":
    text = read_text(TEGRA_RELEASE)
    print(text)
    print("parsed:", l4t_version(text))
$ uv run python -m labs.jetpack_verify   # on the Jetson, after the flash — your GCID and date will differ
# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan  8 01:49:37 UTC 2025
parsed: R36.4.3

Two decisions in eleven lines. read_text catches OSError instead of FileNotFoundError, so a file that exists but cannot be opened (wrong permissions, a directory where a file should be) comes back as a missing value too, and a diagnostic script never dies on the first path it cannot read. The parser, on the other hand, raises when the banner is present but unrecognisable. Those are different situations and they deserve different answers: one is "the installer never wrote this", the other is "the installer wrote something I do not understand", and a report that collapses them into one row will send you looking in the wrong place. L4T R36.4.3 is what this bench's JetPack 6.2.2 media produced; the revision paired with your download is printed on its release page, so copy it from there instead of trusting mine.

▣ Build · stage 3 — a claim, its expectation, and a runner that survives it
# labs/jetpack_verify.py — continued
import platform
from collections.abc import Callable
from dataclasses import dataclass

CUDA_ROOT = Path("/usr/local/cuda")
CUDA_VERSION_TXT = CUDA_ROOT / "version.txt"

EXPECTED = {
    "l4t_release": "R36.4.3",
    "tegra_kernel": "-tegra",
    "cuda_runtime": "12.6",
}


@dataclass
class Check:
    name: str
    expected: str
    detected: str = "not read"
    ok: bool = False


def check_l4t_release() -> Check:
    check = Check("l4t_release", EXPECTED["l4t_release"])
    banner = read_text(TEGRA_RELEASE)
    if banner is None:
        check.detected = f"no file at {TEGRA_RELEASE}"
        return check
    check.detected = l4t_version(banner)
    check.ok = check.detected == check.expected
    return check


def check_tegra_kernel() -> Check:
    check = Check("tegra_kernel", EXPECTED["tegra_kernel"])
    check.detected = platform.release()
    check.ok = check.detected.endswith(EXPECTED["tegra_kernel"])
    return check


def check_cuda_runtime() -> Check:
    check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
    text = read_text(CUDA_VERSION_TXT)
    if text is None:
        check.detected = f"no file at {CUDA_VERSION_TXT}"
        return check
    check.detected = text.split()[-1]
    check.ok = check.detected.startswith(EXPECTED["cuda_runtime"])
    return check


def run_check(fn: Callable[[], Check]) -> Check:
    try:
        return fn()
    except Exception as exc:
        return Check(fn.__name__.removeprefix("check_"), "unknown",
                     f"{type(exc).__name__}: {exc}")


CHECKS: list[Callable[[], Check]] = [check_l4t_release, check_tegra_kernel, check_cuda_runtime]


def report(checks: list[Check]) -> None:
    width = max(len(c.name) for c in checks)
    for c in checks:
        mark = "  OK  " if c.ok else " FAIL "
        print(f"[{mark}] {c.name:<{width}}  expected {c.expected:<10}  detected {c.detected}")


if __name__ == "__main__":
    report([run_check(fn) for fn in CHECKS])
$ uv run python -m labs.jetpack_verify   # right after the flash, before any SDK components
[  OK  ] l4t_release   expected R36.4.3     detected R36.4.3
[  OK  ] tegra_kernel  expected -tegra      detected 5.15.148-tegra
[ FAIL ] cuda_runtime  expected 12.6        detected no file at /usr/local/cuda/version.txt

Two things are borrowed here and one is new. ok starts False and detected starts at "not read", so a check that returns early or dies halfway reports the truth about how far it got; Chapter 58's run_check does the rest, converting any exception into a row instead of an aborted run, which matters more here than it did there because a half-flashed board is exactly where a check hits something it never expected. The new part is the pairing. Every row carries the value the installer produced and the value JetPack 6.2.2 is documented to produce, side by side, and ok is nothing but the comparison between them. A detected version alone is a fact you then have to look up. A detected version beside its expectation is an answer.

The kernel row asks a question the release banner cannot. That banner is a text file, and a text file survives being copied onto a board running any kernel at all. platform.release() reports the kernel the machine actually booted, and the -tegra suffix is NVIDIA's build, carrying the drivers for the GPU and the camera and the rest of the silicon. Generic arm64 Ubuntu on this hardware boots, looks normal, and has no GPU.

▣ Build · stage 4 — the compiler is a separate question from the library
# labs/jetpack_verify.py — continued
def check_cuda_toolkit() -> Check:
    check = Check("cuda_toolkit", "nvcc on disk")
    nvcc = CUDA_ROOT / "bin" / "nvcc"
    if not nvcc.is_file():
        check.detected = f"no nvcc under {CUDA_ROOT}"
        return check
    check.detected = str(nvcc)
    check.ok = True
    return check


CHECKS: list[Callable[[], Check]] = [
    check_l4t_release, check_tegra_kernel, check_cuda_runtime, check_cuda_toolkit,
]
$ uv run python -m labs.jetpack_verify   # after SDK Manager finished installing the components
[  OK  ] l4t_release   expected R36.4.3     detected R36.4.3
[  OK  ] tegra_kernel  expected -tegra      detected 5.15.148-tegra
[ FAIL ] cuda_runtime  expected 12.6        detected no file at /usr/local/cuda/version.txt
[  OK  ] cuda_toolkit  expected nvcc on disk  detected /usr/local/cuda/bin/nvcc

CUDA arrives as two different kinds of thing and they install independently. The runtime is a set of shared libraries that a compiled program loads when it starts, and it is all a machine needs to run GPU code somebody else built. The toolkit adds the compiler nvcc and the headers, and it is what a machine needs to build GPU code, including the moment a Python package with CUDA sources decides to compile itself during installation. A board with the runtime and no toolkit runs prebuilt wheels happily and fails every source build, with an error message that blames the package. Asking two questions instead of one costs four lines and tells you which half is missing.

Now look at rows three and four together, because they contradict each other. The compiler is on disk, at a path inside the CUDA installation, and the check that says CUDA is absent is reading a file inside that same directory. Both cannot be right.

Why this works, and what it cost to find out

⚠ Worked failure — the version file that moved out from under the check

A check that reports a missing file is making a claim about the disk. Test it the cheapest way available, by looking at the directory yourself:

ls /usr/local/cuda/version*
cat /usr/local/cuda/version.json
$ ls /usr/local/cuda/version*
/usr/local/cuda/version.json
$ cat /usr/local/cuda/version.json
{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "12.6.11"
   },
   "cuda_nvcc" : {
      "name" : "cuda_nvcc",
      "version" : "12.6.11"
   }
}

CUDA is installed, complete, and the version is exactly the one expected. The file the check was reading has not existed since the 11.x series, when NVIDIA replaced a one-line version.txt with a structured version.json that lists every component separately. The path came from documentation old enough to predate the board, and it kept looking plausible because it was still inside a directory that exists.

This is the failure a verification script is uniquely able to commit, and it is worth naming precisely. The check did not lie about what it found. It found nothing at a path, and then reported that as "CUDA is missing", when the honest reading was "I looked at a path and it was not there". A missing file can mean the thing is absent or it can mean you are looking in the wrong place, and the check has no way to tell those apart on its own. What told them apart was the other row in the report: a compiler sitting inside the very installation the previous row called absent. Three checks would have caught nothing here. Four checks disagreed with each other, out loud, in the same printout.

# labs/jetpack_verify.py — the corrected runtime check
import json

CUDA_VERSION_JSON = CUDA_ROOT / "version.json"


def check_cuda_runtime() -> Check:
    check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
    text = read_text(CUDA_VERSION_JSON)
    if text is None:
        check.detected = f"no file at {CUDA_VERSION_JSON}"
        return check
    full = json.loads(text)["cuda"]["version"]      # "12.6.11"
    major_minor = ".".join(full.split(".")[:2])     # "12.6"
    check.detected = full
    check.ok = major_minor == check.expected
    return check

The comparison drops the patch number deliberately. That third digit moves between JetPack point releases while the CUDA that packages are built against is written as major and minor, so comparing all three would fail a board that is completely correct. The full string still gets printed, because when a row does fail you want the whole number in front of you.

Step back from the fix and the mechanism generalises past this board. An installer of any kind leaves evidence behind: files at known paths, a kernel with a particular name, a binary that either exists or does not. Verification is reading that evidence back and comparing it to a written expectation. The written part carries the weight. A script that prints versions and stops has handed you homework, because you still have to know what CUDA 12.6 was supposed to look like. A script that prints versions beside the numbers they were supposed to match has already done the thinking, and it keeps doing it in six months, when you no longer remember which JetPack this board was flashed with.

It follows that the expectations belong in the file, in one dictionary, at the top. Reflash to a different JetPack and you edit three strings and the report is honest again. Bury those numbers inside the checks and each one becomes a small assertion nobody can find, and the day the numbers change you will fix two of them.

▣ Build · stage 5 — a verdict, and an exit code to hang it on
# labs/jetpack_verify.py — the ending
import sys


def report(checks: list[Check]) -> bool:
    width = max(len(c.name) for c in checks)
    for c in checks:
        mark = "  OK  " if c.ok else " FAIL "
        print(f"[{mark}] {c.name:<{width}}  expected {c.expected:<13}  detected {c.detected}")
    agreed = sum(1 for c in checks if c.ok)
    print(f"{agreed}/{len(checks)} claims agree with JetPack 6.2.2.")
    return agreed == len(checks)


if __name__ == "__main__":
    sys.exit(report([run_check(fn) for fn in CHECKS]))
$ uv run python -m labs.jetpack_verify; echo "exit=$?"   # bench capture, your versions will vary
[  OK  ] l4t_release   expected R36.4.3        detected R36.4.3
[  OK  ] tegra_kernel  expected -tegra         detected 5.15.148-tegra
[  OK  ] cuda_runtime  expected 12.6           detected 12.6.11
[  OK  ] cuda_toolkit  expected nvcc on disk   detected /usr/local/cuda/bin/nvcc
4/4 claims agree with JetPack 6.2.2.
exit=1

Four green rows and an exit code of 1. Handing sys.exit anything other than None or an integer makes Python convert it, and int(True) is 1, which every shell on the planet reads as failure. The passing run just told your future automation that the flash was bad. Write the conversion out yourself, sys.exit(0 if report(...) else 1), and the same run ends at exit=0. Keep the exit code even while you run this by hand: a verdict another program can read is what lets this drop into a startup routine later, and it costs one line.

Checkpoint, and the machine you have not met yet

✓ Checkpoint — what you can now do
  • I can put the board in recovery mode and confirm from the host, before opening SDK Manager, that it is actually listening.
  • I can name the three destinations one flash writes to and say which of them phase two touches.
  • Handed a board that runs prebuilt CUDA wheels but fails every source build, I can say which half of CUDA is missing and which file settles it.
  • I can explain why a check that finds no file cannot, by itself, distinguish an absent component from a wrong path.
  • I know why the CUDA comparison drops the patch digit and keeps printing it.
  • I can add a fifth claim to the report without touching the runner or the printer.
⚡ Exercises — try first, then reveal
Exercise 1 — add cuDNN, whose version lives in a C header. JetPack 6.2.2 ships cuDNN 9.3. It publishes no JSON; the numbers are #define lines in /usr/include/cudnn_version.h. Write the check and add it to the list.

Three separate defines, so pull them with one pattern and a dictionary:

CUDNN_HEADER = Path("/usr/include/cudnn_version.h")
EXPECTED["cudnn"] = "9.3"


def check_cudnn() -> Check:
    check = Check("cudnn", EXPECTED["cudnn"])
    text = read_text(CUDNN_HEADER)
    if text is None:
        check.detected = f"no file at {CUDNN_HEADER}"
        return check
    found = dict(re.findall(r"#define CUDNN_(MAJOR|MINOR|PATCHLEVEL) (\d+)", text))
    if "MAJOR" not in found or "MINOR" not in found:
        check.detected = "header present but carries no version defines"
        return check
    check.detected = f"{found['MAJOR']}.{found['MINOR']}.{found.get('PATCHLEVEL', '?')}"
    check.ok = check.detected.startswith(EXPECTED["cudnn"])
    return check
$ uv run python -m labs.jetpack_verify   # bench capture
[  OK  ] cudnn         expected 9.3            detected 9.3.0

Note the middle branch. A header that exists but has no version defines is a third state, distinct from missing and from wrong, and it is what you get if the path ever points at some other project's header of the same name.

Exercise 2 — prove you are running from the SSD. A board flashed to NVMe with a microSD still in the slot can boot the card instead, and every check above passes on the wrong disk. Add a claim about which device the root filesystem is mounted from.

/proc/mounts lists the mounted filesystems, one per line, with the device first and the mount point second. Find the line whose mount point is /:

EXPECTED["root_device"] = "/dev/nvme"


def check_root_device() -> Check:
    check = Check("root_device", EXPECTED["root_device"])
    text = read_text(Path("/proc/mounts"))
    if text is None:
        check.detected = "no /proc/mounts"
        return check
    roots = [line.split()[0] for line in text.splitlines()
             if len(line.split()) > 1 and line.split()[1] == "/"]
    if not roots:
        check.detected = "no line in /proc/mounts mounts /"
        return check
    check.detected = roots[-1]
    check.ok = check.detected.startswith(EXPECTED["root_device"])
    return check
$ uv run python -m labs.jetpack_verify   # bench capture
[  OK  ] root_device   expected /dev/nvme      detected /dev/nvme0n1p1

Take the last matching line, not the first: a filesystem can be remounted over / during boot, and the final entry is the one in force. Pull the SSD and boot a card to watch this row turn into /dev/mmcblk1p1 while every other row stays green.

Exercise 3 — prove the checks can fail. Every row is green, which is also what a report full of broken checks looks like. Make the paths injectable and run the whole suite against a directory you know is empty.

Give each check an optional root and default it to the real one, then point the suite somewhere harmless:

def check_cuda_runtime(root: Path = CUDA_ROOT) -> Check:
    check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
    text = read_text(root / "version.json")
    ...


if __name__ == "__main__":
    import tempfile
    with tempfile.TemporaryDirectory() as empty:
        print(f"negative control against {empty}")
        report([run_check(lambda fn=fn: fn(Path(empty))) for fn in (check_cuda_runtime,
                                                                    check_cuda_toolkit)])
$ uv run python -m labs.jetpack_verify
negative control against /tmp/tmpq3k1w8ha
[ FAIL ] cuda_runtime  expected 12.6           detected no file at /tmp/tmpq3k1w8ha/version.json
[ FAIL ] cuda_toolkit  expected nvcc on disk   detected no nvcc under /tmp/tmpq3k1w8ha

The fn=fn default in the lambda binds the current function instead of the loop variable, a detail that bites everyone once. A check that has never been seen to fail is a check you are trusting on its word, and this run costs a second.

The board now has a report you can produce again at any time and hand to yourself as evidence. What it does not have is a way for you to reach it without a monitor, a keyboard and a spare desk, and that is the next thing to fix: the Ubuntu setup wizard on the HDMI output, then an address on your network and a shell you can open from the machine you were already sitting at.