Flashing JetPack
Flash successful, and nothing proven yet
The last dialog SDK Manager shows you is a green tick and the words Flash successful. Read that sentence for what it actually asserts. A program on your desktop pushed a few gigabytes down a USB cable, the bootloader on the far end acknowledged each transfer, and nothing errored. That is a claim about a conversation between two machines. It says nothing about which operating system the board comes up on, which CUDA is on its disk, or whether the compiler you need is installed at all.
The gap between those two things is not academic. A board can flash cleanly and still boot an older image off a microSD card somebody left in the slot. A flash can put the runtime libraries down and skip the toolkit, and you will not find out until an import three weeks from now fails with a message about a driver version that means nothing to you. By then the flash is ancient history and you are debugging the wrong layer.
Everything you need to know about a flash was written to a file on the board. The release banner, the CUDA version, the compiler binary: all of it sits at known paths in plain text, put there by the installer. So the rule for this chapter: a flash is finished when a script has read the board's own files back and printed each detected value beside the value it was supposed to be, and every row agrees. Not when the progress bar fills. When the report is clean.
A Jetson Orin Nano Super developer kit, a 500 GB NVMe SSD in the M.2 slot, the barrel-jack supply that came in the box, and a desktop running Ubuntu 22.04 on an x86_64 processor to drive the flash. JetPack 6.2.2 throughout. Every command output below came off that one bench once. Version strings move between JetPack point releases, revision numbers differ, and your board will print its own; the prose says so wherever the exact digits matter.
What the host writes, and where it lands
The Jetson cannot flash itself. Rewriting the storage a machine is running from is a problem no running machine solves cleanly, so NVIDIA puts the job on a second computer. Bridge the pin marked FC REC to the ground pin beside it, apply power, and the module skips its normal boot and waits on the USB-C port for someone to tell it what to write. That is recovery mode. Two details trip people up on the first attempt: the jumper has to be in place before power arrives, and the USB-C port is a data port only, so power still comes from the barrel jack.
The host tool is SDK Manager, an NVIDIA application that runs on x86_64 Ubuntu and nowhere else. No Windows, no macOS, and no arm64. Before clicking anything in it, ask the host whether the board is even listening.
# on the x86_64 Ubuntu host, with the jumper in place and the board powered on
lsusb | grep -i nvidia
$ lsusb | grep -i nvidia # captured on the bench — your bus and device numbers will differ
Bus 001 Device 014: ID 0955:7523 NVIDIA Corp. APX
0955 is NVIDIA's USB vendor identifier and APX is the name
the recovery bootloader answers to. The product identifier after the colon varies by
module generation, so match on the vendor and the APX label and ignore the rest. An
empty result means the board is booting normally, the jumper went on after power, or
the cable is a charge-only one. Sorting that out now takes a minute; sorting it out
after SDK Manager has sat at "waiting for target" for a quarter of an hour does not.
What happens next arrives in two phases, and keeping them separate in your head explains most of what can go wrong. Phase one is the flash proper. The host streams a bootloader chain into the QSPI-NOR memory soldered onto the module itself, and streams a root filesystem, a kernel and a handful of small partitions onto whatever external media you selected, the NVMe SSD in this build. Then the board reboots into the image it was just given.
Phase two happens over the network, to a board that is already running. SDK Manager logs in and installs the SDK components as ordinary Debian packages: CUDA, cuDNN, TensorRT, the developer tools. This is where a flash quietly comes up short. Phase one can succeed perfectly while phase two is skipped, interrupted, or run with half the components unticked, and the board still boots, still shows a desktop, and still looks finished.
Four claims, read off the board
Log in on the Jetson and the first file to open is the release banner the installer wrote. Read it before parsing it, because the parser you write next depends on its exact punctuation.
# labs/jetpack_verify.py
import re
from pathlib import Path
TEGRA_RELEASE = Path("/etc/nv_tegra_release")
def read_text(path: Path) -> str | None:
"""The file's contents, or None if this board has no such file."""
try:
return path.read_text().strip()
except OSError:
return None
def l4t_version(banner: str) -> str:
"""Turn the release banner into something comparable, like 'R36.4.3'."""
match = re.search(r"R(\d+) \(release\), REVISION: (\d+\.\d+)", banner)
if match is None:
raise ValueError(f"unreadable release banner: {banner[:48]!r}")
return f"R{match.group(1)}.{match.group(2)}"
if __name__ == "__main__":
text = read_text(TEGRA_RELEASE)
print(text)
print("parsed:", l4t_version(text))
$ uv run python -m labs.jetpack_verify # on the Jetson, after the flash — your GCID and date will differ
# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025
parsed: R36.4.3
Two decisions in eleven lines. read_text catches OSError
instead of FileNotFoundError, so a file that exists but cannot be opened
(wrong permissions, a directory where a file should be) comes back as a missing value
too, and a diagnostic script never dies on the first path it cannot read. The parser,
on the other hand, raises when the banner is present but unrecognisable. Those are
different situations and they deserve different answers: one is "the installer never
wrote this", the other is "the installer wrote something I do not understand", and a
report that collapses them into one row will send you looking in the wrong place.
L4T R36.4.3 is what this bench's JetPack 6.2.2 media produced; the revision paired with
your download is printed on its release page, so copy it from there instead of
trusting mine.
# labs/jetpack_verify.py — continued
import platform
from collections.abc import Callable
from dataclasses import dataclass
CUDA_ROOT = Path("/usr/local/cuda")
CUDA_VERSION_TXT = CUDA_ROOT / "version.txt"
EXPECTED = {
"l4t_release": "R36.4.3",
"tegra_kernel": "-tegra",
"cuda_runtime": "12.6",
}
@dataclass
class Check:
name: str
expected: str
detected: str = "not read"
ok: bool = False
def check_l4t_release() -> Check:
check = Check("l4t_release", EXPECTED["l4t_release"])
banner = read_text(TEGRA_RELEASE)
if banner is None:
check.detected = f"no file at {TEGRA_RELEASE}"
return check
check.detected = l4t_version(banner)
check.ok = check.detected == check.expected
return check
def check_tegra_kernel() -> Check:
check = Check("tegra_kernel", EXPECTED["tegra_kernel"])
check.detected = platform.release()
check.ok = check.detected.endswith(EXPECTED["tegra_kernel"])
return check
def check_cuda_runtime() -> Check:
check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
text = read_text(CUDA_VERSION_TXT)
if text is None:
check.detected = f"no file at {CUDA_VERSION_TXT}"
return check
check.detected = text.split()[-1]
check.ok = check.detected.startswith(EXPECTED["cuda_runtime"])
return check
def run_check(fn: Callable[[], Check]) -> Check:
try:
return fn()
except Exception as exc:
return Check(fn.__name__.removeprefix("check_"), "unknown",
f"{type(exc).__name__}: {exc}")
CHECKS: list[Callable[[], Check]] = [check_l4t_release, check_tegra_kernel, check_cuda_runtime]
def report(checks: list[Check]) -> None:
width = max(len(c.name) for c in checks)
for c in checks:
mark = " OK " if c.ok else " FAIL "
print(f"[{mark}] {c.name:<{width}} expected {c.expected:<10} detected {c.detected}")
if __name__ == "__main__":
report([run_check(fn) for fn in CHECKS])
$ uv run python -m labs.jetpack_verify # right after the flash, before any SDK components
[ OK ] l4t_release expected R36.4.3 detected R36.4.3
[ OK ] tegra_kernel expected -tegra detected 5.15.148-tegra
[ FAIL ] cuda_runtime expected 12.6 detected no file at /usr/local/cuda/version.txt
Two things are borrowed here and one is new. ok starts False
and detected starts at "not read", so a check that returns
early or dies halfway reports the truth about how far it got; Chapter 58's
run_check does the rest, converting any exception into a row instead of
an aborted run, which matters more here than it did there because a half-flashed board
is exactly where a check hits something it never expected. The new part is the pairing.
Every row carries the value the installer produced and the value JetPack 6.2.2 is
documented to produce, side by side, and ok is nothing but the comparison
between them. A detected version alone is a fact you then have to look up. A detected
version beside its expectation is an answer.
The kernel row asks a question the release banner cannot. That banner is a text file,
and a text file survives being copied onto a board running any kernel at all.
platform.release() reports the kernel the machine actually booted, and
the -tegra suffix is NVIDIA's build, carrying the drivers for the GPU and
the camera and the rest of the silicon. Generic arm64 Ubuntu on this hardware boots,
looks normal, and has no GPU.
# labs/jetpack_verify.py — continued
def check_cuda_toolkit() -> Check:
check = Check("cuda_toolkit", "nvcc on disk")
nvcc = CUDA_ROOT / "bin" / "nvcc"
if not nvcc.is_file():
check.detected = f"no nvcc under {CUDA_ROOT}"
return check
check.detected = str(nvcc)
check.ok = True
return check
CHECKS: list[Callable[[], Check]] = [
check_l4t_release, check_tegra_kernel, check_cuda_runtime, check_cuda_toolkit,
]
$ uv run python -m labs.jetpack_verify # after SDK Manager finished installing the components
[ OK ] l4t_release expected R36.4.3 detected R36.4.3
[ OK ] tegra_kernel expected -tegra detected 5.15.148-tegra
[ FAIL ] cuda_runtime expected 12.6 detected no file at /usr/local/cuda/version.txt
[ OK ] cuda_toolkit expected nvcc on disk detected /usr/local/cuda/bin/nvcc
CUDA arrives as two different kinds of thing and they install independently. The
runtime is a set of shared libraries that a compiled program loads when it starts, and
it is all a machine needs to run GPU code somebody else built. The toolkit
adds the compiler nvcc and the headers, and it is what a machine needs to
build GPU code, including the moment a Python package with CUDA sources
decides to compile itself during installation. A board with the runtime and no toolkit
runs prebuilt wheels happily and fails every source build, with an error message that
blames the package. Asking two questions instead of one costs four lines and tells you
which half is missing.
Now look at rows three and four together, because they contradict each other. The compiler is on disk, at a path inside the CUDA installation, and the check that says CUDA is absent is reading a file inside that same directory. Both cannot be right.
Why this works, and what it cost to find out
A check that reports a missing file is making a claim about the disk. Test it the cheapest way available, by looking at the directory yourself:
ls /usr/local/cuda/version*
cat /usr/local/cuda/version.json
$ ls /usr/local/cuda/version* /usr/local/cuda/version.json $ cat /usr/local/cuda/version.json { "cuda" : { "name" : "CUDA SDK", "version" : "12.6.11" }, "cuda_nvcc" : { "name" : "cuda_nvcc", "version" : "12.6.11" } }
CUDA is installed, complete, and the version is exactly the one expected. The file the
check was reading has not existed since the 11.x series, when NVIDIA replaced a
one-line version.txt with a structured version.json that
lists every component separately. The path came from documentation old enough to
predate the board, and it kept looking plausible because it was still inside a
directory that exists.
This is the failure a verification script is uniquely able to commit, and it is worth naming precisely. The check did not lie about what it found. It found nothing at a path, and then reported that as "CUDA is missing", when the honest reading was "I looked at a path and it was not there". A missing file can mean the thing is absent or it can mean you are looking in the wrong place, and the check has no way to tell those apart on its own. What told them apart was the other row in the report: a compiler sitting inside the very installation the previous row called absent. Three checks would have caught nothing here. Four checks disagreed with each other, out loud, in the same printout.
# labs/jetpack_verify.py — the corrected runtime check
import json
CUDA_VERSION_JSON = CUDA_ROOT / "version.json"
def check_cuda_runtime() -> Check:
check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
text = read_text(CUDA_VERSION_JSON)
if text is None:
check.detected = f"no file at {CUDA_VERSION_JSON}"
return check
full = json.loads(text)["cuda"]["version"] # "12.6.11"
major_minor = ".".join(full.split(".")[:2]) # "12.6"
check.detected = full
check.ok = major_minor == check.expected
return check
The comparison drops the patch number deliberately. That third digit moves between JetPack point releases while the CUDA that packages are built against is written as major and minor, so comparing all three would fail a board that is completely correct. The full string still gets printed, because when a row does fail you want the whole number in front of you.
Step back from the fix and the mechanism generalises past this board. An installer of any kind leaves evidence behind: files at known paths, a kernel with a particular name, a binary that either exists or does not. Verification is reading that evidence back and comparing it to a written expectation. The written part carries the weight. A script that prints versions and stops has handed you homework, because you still have to know what CUDA 12.6 was supposed to look like. A script that prints versions beside the numbers they were supposed to match has already done the thinking, and it keeps doing it in six months, when you no longer remember which JetPack this board was flashed with.
It follows that the expectations belong in the file, in one dictionary, at the top. Reflash to a different JetPack and you edit three strings and the report is honest again. Bury those numbers inside the checks and each one becomes a small assertion nobody can find, and the day the numbers change you will fix two of them.
# labs/jetpack_verify.py — the ending
import sys
def report(checks: list[Check]) -> bool:
width = max(len(c.name) for c in checks)
for c in checks:
mark = " OK " if c.ok else " FAIL "
print(f"[{mark}] {c.name:<{width}} expected {c.expected:<13} detected {c.detected}")
agreed = sum(1 for c in checks if c.ok)
print(f"{agreed}/{len(checks)} claims agree with JetPack 6.2.2.")
return agreed == len(checks)
if __name__ == "__main__":
sys.exit(report([run_check(fn) for fn in CHECKS]))
$ uv run python -m labs.jetpack_verify; echo "exit=$?" # bench capture, your versions will vary
[ OK ] l4t_release expected R36.4.3 detected R36.4.3
[ OK ] tegra_kernel expected -tegra detected 5.15.148-tegra
[ OK ] cuda_runtime expected 12.6 detected 12.6.11
[ OK ] cuda_toolkit expected nvcc on disk detected /usr/local/cuda/bin/nvcc
4/4 claims agree with JetPack 6.2.2.
exit=1
Four green rows and an exit code of 1. Handing sys.exit anything other
than None or an integer makes Python convert it, and int(True)
is 1, which every shell on the planet reads as failure. The passing run just told your
future automation that the flash was bad. Write the conversion out yourself,
sys.exit(0 if report(...) else 1), and the same run ends at
exit=0. Keep the exit code even while you run this by hand: a verdict
another program can read is what lets this drop into a startup routine later, and it
costs one line.
Checkpoint, and the machine you have not met yet
- I can put the board in recovery mode and confirm from the host, before opening SDK Manager, that it is actually listening.
- I can name the three destinations one flash writes to and say which of them phase two touches.
- Handed a board that runs prebuilt CUDA wheels but fails every source build, I can say which half of CUDA is missing and which file settles it.
- I can explain why a check that finds no file cannot, by itself, distinguish an absent component from a wrong path.
- I know why the CUDA comparison drops the patch digit and keeps printing it.
- I can add a fifth claim to the report without touching the runner or the printer.
Exercise 1 — add cuDNN, whose version lives in a C header.
JetPack 6.2.2 ships cuDNN 9.3. It publishes no JSON; the numbers are
#define lines in /usr/include/cudnn_version.h. Write the
check and add it to the list.
Three separate defines, so pull them with one pattern and a dictionary:
CUDNN_HEADER = Path("/usr/include/cudnn_version.h")
EXPECTED["cudnn"] = "9.3"
def check_cudnn() -> Check:
check = Check("cudnn", EXPECTED["cudnn"])
text = read_text(CUDNN_HEADER)
if text is None:
check.detected = f"no file at {CUDNN_HEADER}"
return check
found = dict(re.findall(r"#define CUDNN_(MAJOR|MINOR|PATCHLEVEL) (\d+)", text))
if "MAJOR" not in found or "MINOR" not in found:
check.detected = "header present but carries no version defines"
return check
check.detected = f"{found['MAJOR']}.{found['MINOR']}.{found.get('PATCHLEVEL', '?')}"
check.ok = check.detected.startswith(EXPECTED["cudnn"])
return check
$ uv run python -m labs.jetpack_verify # bench capture
[ OK ] cudnn expected 9.3 detected 9.3.0
Note the middle branch. A header that exists but has no version defines is a third state, distinct from missing and from wrong, and it is what you get if the path ever points at some other project's header of the same name.
Exercise 2 — prove you are running from the SSD. A board flashed to NVMe with a microSD still in the slot can boot the card instead, and every check above passes on the wrong disk. Add a claim about which device the root filesystem is mounted from.
/proc/mounts lists the mounted filesystems, one per line, with the
device first and the mount point second. Find the line whose mount point is
/:
EXPECTED["root_device"] = "/dev/nvme"
def check_root_device() -> Check:
check = Check("root_device", EXPECTED["root_device"])
text = read_text(Path("/proc/mounts"))
if text is None:
check.detected = "no /proc/mounts"
return check
roots = [line.split()[0] for line in text.splitlines()
if len(line.split()) > 1 and line.split()[1] == "/"]
if not roots:
check.detected = "no line in /proc/mounts mounts /"
return check
check.detected = roots[-1]
check.ok = check.detected.startswith(EXPECTED["root_device"])
return check
$ uv run python -m labs.jetpack_verify # bench capture
[ OK ] root_device expected /dev/nvme detected /dev/nvme0n1p1
Take the last matching line, not the first: a filesystem can be remounted over
/ during boot, and the final entry is the one in force. Pull the SSD
and boot a card to watch this row turn into /dev/mmcblk1p1 while every
other row stays green.
Exercise 3 — prove the checks can fail. Every row is green, which is also what a report full of broken checks looks like. Make the paths injectable and run the whole suite against a directory you know is empty.
Give each check an optional root and default it to the real one, then point the suite somewhere harmless:
def check_cuda_runtime(root: Path = CUDA_ROOT) -> Check:
check = Check("cuda_runtime", EXPECTED["cuda_runtime"])
text = read_text(root / "version.json")
...
if __name__ == "__main__":
import tempfile
with tempfile.TemporaryDirectory() as empty:
print(f"negative control against {empty}")
report([run_check(lambda fn=fn: fn(Path(empty))) for fn in (check_cuda_runtime,
check_cuda_toolkit)])
$ uv run python -m labs.jetpack_verify
negative control against /tmp/tmpq3k1w8ha
[ FAIL ] cuda_runtime expected 12.6 detected no file at /tmp/tmpq3k1w8ha/version.json
[ FAIL ] cuda_toolkit expected nvcc on disk detected no nvcc under /tmp/tmpq3k1w8ha
The fn=fn default in the lambda binds the current function instead of
the loop variable, a detail that bites everyone once. A check that has never been
seen to fail is a check you are trusting on its word, and this run costs a second.
The board now has a report you can produce again at any time and hand to yourself as evidence. What it does not have is a way for you to reach it without a monitor, a keyboard and a spare desk, and that is the next thing to fix: the Ubuntu setup wizard on the HDMI output, then an address on your network and a shell you can open from the machine you were already sitting at.