GLaDOS Vol 8 · The Jetson Brain
ch 77 / 99
Chapter 77

First Boot and Remote Access

The one morning she needs a monitor

Every board in this build so far joined the network before you ever looked at it. The Raspberry Pi imager writes a wireless password, a hostname and a public key onto the card while the card is still in your laptop, so the Pi's first boot already has an account, an address and an open SSH port. The first thing you ever type at that board is ssh. Nothing about the Jetson works that way.

What lands on the Jetson's disk is an operating system with no account inside it. Somebody has to answer for that account, and the questions arrive about a minute after the first power-up: a language, a keyboard layout, a licence page, a timezone, a username and password, a name for the computer, how many gigabytes to give the APP partition, and which power mode the board should run in. They arrive on the HDMI output. They wait there until a human with a keyboard answers them, and there is no file you can drop beside the image to answer them ahead of time.

So this chapter has a step you cannot script. A monitor, a USB keyboard, a power supply, and about ten minutes at whatever surface has a screen on it. That is the honest cost, it happens once per board, and pretending otherwise only means finding out about it while the board is already bolted into a chassis.

The wizard is not the trap. The trap is the twenty minutes after it, when the monitor is sitting right there and working, so the rest of the setup happens at the board: packages installed at that keyboard, files fetched in that browser, an address nobody wrote down. Then the board goes into the robot and you have a machine you can no longer talk to. The developer kit ships with the M.2 Key E slot empty, so there is no wireless card either. Skip the Ethernet cable and the only way back is to carry the monitor over again.

The finish line is not "the desktop appeared". It is this: the setup is done when a script you start from your desk can tell you where the board lives, what it is running, and how much room is left on it. Until that script passes, the monitor is still load-bearing. That script is what this chapter builds: jetson_setup_check.py, four checks and one verdict line, run over SSH from the same workstation you have written every other chapter's code on.

◆ Note — the bench for this volume, and how to read its output

One Jetson Orin Nano Super developer kit with an NVMe drive in the M.2 slot, an Ethernet cable into a home switch on the 192.168.1.0/24 network, and a workstation on that same network. The board runs the JetPack 6.2 image, which is Ubuntu 22.04 with NVIDIA's kernel and drivers on top. Every address, lease time, disk figure and kernel string printed below came off that one bench on one afternoon; yours will differ in all of them, and the commands are what carry over. The hostname and username used here are glados-jetson and glados, chosen at the wizard, so that the paths in later chapters match what you type.

The ritual, and the last time you perform it

▣ Build · stage 1 — finish the wizard, then read the address it borrowed
# at the monitor, in a terminal on the board, once
ip -4 addr show eth0
systemctl is-active ssh
$ ip -4 addr show eth0   # measured on the bench — yours will vary
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.1.137/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 84903sec preferred_lft 84903sec
$ systemctl is-active ssh
active

Two words in that inet line decide the rest of the chapter. 192.168.1.137/24 is the address and the size of the network it belongs to: 24 bits of network, leaving the last 8 bits for hosts. dynamic means the address was lent by the router's DHCP server, and valid_lft 84903sec is the lease counting down, a little under a day. Reboot the board on a busy network, or leave it powered off over a weekend, and the number you carefully wrote on a sticky note points at somebody's laptop. The JetPack image already runs an SSH server, which is what active reports. If yours prints inactive, install it with sudo apt install -y openssh-server and enable it with sudo systemctl enable --now ssh.

▣ Build · stage 2 — the first connection from your desk, and the last password
# on your workstation, not on the board
ssh-keygen -t ed25519 -C "glados jetson"    # skip if you already have a key
ssh glados@192.168.1.137                    # the password you chose at the wizard
exit
ssh-copy-id glados@192.168.1.137
ssh glados@192.168.1.137 uname -a
$ ssh-copy-id glados@192.168.1.137
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
glados@192.168.1.137's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'glados@192.168.1.137'"
and check to make sure that only the key you wanted was added.
$ ssh glados@192.168.1.137 uname -a   # measured on the bench — yours will vary
Linux glados-jetson 5.15.148-tegra #1 SMP PREEMPT Tue Jan  7 17:14:38 PST 2025 aarch64 aarch64 aarch64 GNU/Linux

ssh-copy-id does something small and exact: it appends one line, your public key, to ~/.ssh/authorized_keys on the board, creating the directory with the permissions the SSH server insists on. From then on the server proves you are you by asking your client to sign a challenge with the matching private key, and no password crosses the network. The reason to bother is not security alone. A password prompt is interactive, and every command in the rest of this volume is going to run as ssh glados-jetson something from a script, in a loop, or out of a Makefile. Interactive prompts have no place there.

Read the tail of that uname line before moving on. aarch64, three times, is the board telling you it is 64-bit ARM. Every wheel, container image and compiled binary that arrives here has to exist for that architecture, and a fair number of things you are used to installing on an x86 laptop simply do not. 5.15.148-tegra is NVIDIA's kernel, not a stock Ubuntu one, so board support arrives on NVIDIA's release schedule and not Ubuntu's.

▣ Build · stage 3 — an address that stops moving, and a name for it
# on the board, over ssh now
nmcli -t -f NAME,DEVICE connection show
sudo nmcli connection modify "Wired connection 1" \
    ipv4.method manual \
    ipv4.addresses 192.168.1.100/24 \
    ipv4.gateway 192.168.1.1 \
    ipv4.dns 192.168.1.1
sudo nmcli connection up "Wired connection 1"
$ nmcli -t -f NAME,DEVICE connection show
Wired connection 1:eth0
$ sudo nmcli connection up "Wired connection 1"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
client_loop: send disconnect: Broken pipe
# on your workstation: ~/.ssh/config
Host glados-jetson
    HostName 192.168.1.100
    User glados
    IdentityFile ~/.ssh/id_ed25519

That Broken pipe is not a bug and it is not optional. The last command tears down and rebuilds the connection on eth0, and your SSH session is running over eth0, so the session dies in the middle of the command that killed it. The board is fine. Reconnect at the new address, which after the ~/.ssh/config entry above is just ssh glados-jetson. Expect a warning about a changed host key only if some other machine used to answer at 192.168.1.100.

There are two honest ways to pin an address and they disagree about where the truth lives. A DHCP reservation on the router keeps every address on your network in one table, and the board keeps asking for a lease as normal. The static configuration above puts the answer on the board, where it survives replacing the router and where it will happily collide with a lease the router hands somebody else. Pick a number outside the router's DHCP pool, pick one method, and write down which one you chose, because the failure mode of forgetting is a board that answers at an address you cannot explain.

◆ Note — the flashing cable is also a network

The USB-C port you flashed the board through comes up as a network device once the system is running. The Jetson holds 192.168.55.1 on it and hands your workstation 192.168.55.100, so ssh glados@192.168.55.1 works over the same cable with no switch involved. It cannot rescue the first boot, because the account you would log in as does not exist until the wizard creates it. Afterwards it is the way back in when the Ethernet cable is out, the static address was wrong, or the board has been carried to a desk with no switch on it.

A script that answers from the other side of the wire

The board is reachable. What is missing is a way to ask it, in one command and without remembering four of them, whether it is still the machine you think it is. Back in volume 1 the microphone work began by listing every capture device the machine would admit to, because the right device is only obvious once you can see the whole list. Network interfaces reward exactly the same habit, and for the same reason: this board has three of them, and two of them are traps if you assume there is only one.

▣ Build · stage 4 — what the operating system says it is
# labs/jetson_setup_check.py — runs on the board, with the interpreter it shipped with
import socket


def read_os_release(path: str = "/etc/os-release") -> dict[str, str]:
    """Parse the distro's KEY=value file into a dict."""
    info: dict[str, str] = {}
    try:
        with open(path) as handle:
            for line in handle:
                line = line.strip()
                if not line or line.startswith("#") or "=" not in line:
                    continue
                key, value = line.split("=", 1)
                info[key] = value.strip('"')
    except FileNotFoundError:
        pass
    return info


if __name__ == "__main__":
    release = read_os_release()
    print("Hostname:", socket.gethostname())
    print("OS      :", release.get("PRETTY_NAME", "unknown"))
    print("Version :", release.get("VERSION_ID", "unknown"))
$ python3 labs/jetson_setup_check.py   # measured on the bench — yours will vary
Hostname: glados-jetson
OS      : Ubuntu 22.04.5 LTS
Version : 22.04

/etc/os-release is a file every mainstream Linux distribution promises to write, in a format simple enough to read from a shell script or from twelve lines of Python. Parsing it beats running lsb_release and picking words out of its output, because a file with a published format does not change its wording between releases. The 1 in split("=", 1) caps the split at the first equals sign, and it is not decoration: the file carries a HOME_URL whose query string contains more of them, and without the cap that line unpacks into four values and raises ValueError. Note this script runs with the system python3, the 3.10 that Ubuntu 22.04 ships. The project's pinned 3.11 and its uv workspace are not on this board yet, and the audit has to work before they are.

▣ Build · stage 5 — asking the kernel which addresses it holds
import fcntl
import struct

SIOCGIFADDR = 0x8915          # the kernel's request number for "this interface's address"


def interface_ipv4(name: str) -> str | None:
    """Ask the kernel for the IPv4 address held by one interface."""
    probe = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        request = struct.pack("256s", name[:15].encode("utf-8"))
        reply = fcntl.ioctl(probe.fileno(), SIOCGIFADDR, request)
        return socket.inet_ntoa(reply[20:24])
    except OSError:
        return None
    finally:
        probe.close()


def interface_addresses() -> dict[str, str]:
    """Every interface currently holding an IPv4 address, in kernel order."""
    found: dict[str, str] = {}
    for _index, name in socket.if_nameindex():
        address = interface_ipv4(name)
        if address is not None:
            found[name] = address
    return found


if __name__ == "__main__":
    for name, address in interface_addresses().items():
        print(f"{name:<10} {address}")
$ python3 labs/jetson_setup_check.py   # measured on the bench — yours will vary
lo         127.0.0.1
eth0       192.168.1.100
l4tbr0     192.168.55.1

socket.if_nameindex() is the kernel's own list of network interfaces, the same list ip link prints, handed over as pairs of index and name. For each name, fcntl.ioctl passes a request straight to the kernel: here is a buffer with an interface name in it, fill in its address. The reply comes back the same length as the request, laid out as the C structure the kernel uses. Sixteen bytes of interface name, then an address structure whose first four bytes are the family and the port, so the four bytes of IPv4 address begin at offset 20 and socket.inet_ntoa turns them into the dotted string. An OSError means that interface exists but holds no IPv4 address right now, which is ordinary for an interface that is down.

Three interfaces, three different meanings. lo is the loopback: it exists on every Linux machine, it never leaves the box, and its presence tells you nothing about your network. eth0 is the Ethernet port, holding the static address from stage 3, and it is the only one your desk can reach. l4tbr0 is the bridge behind the USB device-mode network, reachable only by whatever is on the far end of that cable. An audit that reported one address would have to choose, and any choice it made would be wrong somewhere.

Three interfaces on the board and who can reach each one The workstation reaches the board's eth0 interface at 192.168.1.100 through the LAN switch. A dashed path from the workstation reaches l4tbr0 at 192.168.55.1 directly over the USB-C cable, with no switch involved. The loopback interface at 127.0.0.1 has no path from outside; only programs running on the board itself can reach it. ONE BOARD · THREE ADDRESSES · THREE DIFFERENT ANSWERS workstation LAN switch glados-jetson eth0 192.168.1.100 l4tbr0 192.168.55.1 lo 127.0.0.1 192.168.1.0/24 USB-C cable, no switch involved no path from outside the board
Figure 77.1 — The audit prints all three, because all three are real. Only the top one is the address you type at your desk, and the bottom one is the one a program on the board uses to check whether its own SSH server is listening.
▣ Build · stage 6 — four questions, one verdict
import shutil
from pathlib import Path

MIN_FREE_BYTES = 20 * 1024**3   # a floor for the models that arrive later in this volume


def format_bytes(count: float) -> str:
    """Render a byte count the way df -h does, in steps of 1024."""
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if count < 1024:
            return f"{count:.1f} {unit}"
        count /= 1024
    return f"{count:.1f} PB"


def port_is_open(port: int = 22, host: str = "127.0.0.1") -> bool:
    """True when something on this machine is listening on that port."""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:
        probe.settimeout(1.0)
        return probe.connect_ex((host, port)) == 0


def count_authorized_keys() -> int:
    path = Path.home() / ".ssh" / "authorized_keys"
    if not path.exists():
        return 0
    return sum(1 for line in path.read_text().splitlines()
               if line.strip() and not line.startswith("#"))


def run_audit() -> dict[str, tuple[bool, str]]:
    """Four questions whose answers decide whether the monitor can come off."""
    lan = {n: a for n, a in interface_addresses().items() if not a.startswith("127.")}
    listening = port_is_open()
    keys = count_authorized_keys()
    free = shutil.disk_usage("/").free
    return {
        "reachable address": (bool(lan),
                              ", ".join(f"{n} {a}" for n, a in lan.items()) or "loopback only"),
        "ssh listening": (listening,
                          "127.0.0.1:22 accepted a connection" if listening else "nothing on port 22"),
        "key login ready": (keys > 0, f"{keys} key(s) in ~/.ssh/authorized_keys"),
        "room for models": (free >= MIN_FREE_BYTES,
                            f"{format_bytes(free)} free (floor {format_bytes(MIN_FREE_BYTES)})"),
    }


def main() -> None:
    release = read_os_release()
    usage = shutil.disk_usage("/")
    results = run_audit()
    passed = sum(1 for ok, _ in results.values() if ok)

    print("=" * 58)
    print("  Jetson headless audit")
    print("=" * 58)
    print(f"  Hostname   : {socket.gethostname()}")
    print(f"  OS         : {release.get('PRETTY_NAME', 'unknown')}")
    print(f"  Interfaces : {', '.join(f'{n} {a}' for n, a in interface_addresses().items())}")
    print(f"  Disk       : {format_bytes(usage.free)} free of {format_bytes(usage.total)}")
    print("-" * 58)
    for name, (ok, detail) in results.items():
        print(f"  {'PASS' if ok else 'FAIL':<4}  {name:<18} {detail}")
    print("=" * 58)
    verdict = "the monitor can come off" if passed == len(results) else "keep the monitor attached"
    print(f"  {passed}/{len(results)} checks passed: {verdict}.")
    print("=" * 58)


if __name__ == "__main__":
    main()
$ scp labs/jetson_setup_check.py glados-jetson:~/
jetson_setup_check.py                             100% 3241     1.4MB/s   00:00
$ ssh glados-jetson python3 jetson_setup_check.py   # measured on the bench — yours will vary
==========================================================
  Jetson headless audit
==========================================================
  Hostname   : glados-jetson
  OS         : Ubuntu 22.04.5 LTS
  Interfaces : lo 127.0.0.1, eth0 192.168.1.100, l4tbr0 192.168.55.1
  Disk       : 414.4 GB free of 457.5 GB
----------------------------------------------------------
  PASS  reachable address  eth0 192.168.1.100, l4tbr0 192.168.55.1
  PASS  ssh listening      127.0.0.1:22 accepted a connection
  PASS  key login ready    1 key(s) in ~/.ssh/authorized_keys
  PASS  room for models    414.4 GB free (floor 20.0 GB)
==========================================================
  4/4 checks passed: the monitor can come off.
==========================================================

Each check is a question you would otherwise ask by hand and forget to ask again. port_is_open is the one worth looking at twice: connect_ex attempts a real TCP connection to the board's own port 22 and returns an error number instead of raising, so a refused connection is a value to test rather than an exception to catch. It runs on the board, over lo, and that is the reason the loopback interface earns a line on the report even though nothing outside can use it. The disk floor of 20 GB is the only number here with an opinion in it; the models this volume moves onto the board are several gigabytes each, and finding that out with 4 GB free is a bad afternoon.

Notice that run_audit returns data and prints nothing, while main prints and decides nothing. Keeping them apart is what makes the last exercise in this chapter a five-line change instead of a rewrite: a dictionary of named results can be rendered as a table, written as JSON, or compared against yesterday's copy, and the code that gathered it does not care which.

Why this works: two questions that sound identical

Nothing in the audit guesses. /etc/os-release is a file the distribution maintains for programs to read. if_nameindex and the ioctl are the kernel answering about its own hardware. shutil.disk_usage is a thin wrapper over the system call that df uses. Each returns plain data, a dict or a tuple of integers, so every one of these functions can be called from something else later without dragging a report format along with it.

The part that generalises past this chapter is the difference between two questions. "What is this machine called, and what does that name resolve to" is a question for the name resolver, and it travels through /etc/hosts, DNS, and whatever else the system was told to consult. "What addresses do this machine's interfaces currently hold" is a question for the kernel, and it has nothing to do with names at all. On most desktop Linux boxes the two questions happen to produce similar answers, which is how a program that asks the wrong one survives long enough to be shipped. On this board they produce completely different answers, and the failure that follows is what that looks like from the inside.

⚠ Worked failure — the board reports no address, over a connection to that address

The obvious way to find a machine's IP address, and the one nearly every tutorial prints, is to look up its own hostname:

# the version that looks right and is not
def get_ip_addresses() -> list[str]:
    addrs: list[str] = []
    for entry in socket.getaddrinfo(socket.gethostname(), None):
        ip = entry[4][0]
        if not ip.startswith("127.") and ":" not in ip and ip not in addrs:
            addrs.append(ip)
    return addrs
$ ssh glados-jetson python3 jetson_setup_check.py
  Hostname   : glados-jetson
  OS         : Ubuntu 22.04.5 LTS
  IP address : None found
  SSH: ssh glados@<ip-address>

Read that carefully. The report claims the board has no network address, and it arrived on your screen through an SSH connection to 192.168.1.100. Both statements cannot be true, so the program is wrong about something, and the fastest way to find out what is to print the thing being filtered instead of the thing that survived the filter:

$ ssh glados-jetson python3 -c "import socket; print(socket.getaddrinfo(socket.gethostname(), None))"
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.1.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.1.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('127.0.1.1', 0))]
$ ssh glados-jetson grep glados-jetson /etc/hosts
127.0.1.1	glados-jetson

There it is, in the third line of a file the installer wrote. Debian and Ubuntu map a machine's own hostname to 127.0.1.1 so that any program resolving that name gets an answer even when the network is down, and the resolver dutifully reports it. The filter then discards it for starting with 127., correctly, and the list comes back empty. Every step behaved exactly as documented. The bug is one level up: the code asked what the machine's name means, when what it needed to know was what its interfaces hold. Swapping getaddrinfo for if_nameindex and the ioctl changes the question, and the answer stops depending on a text file that somebody else's installer wrote.

Checkpoint, and the hardware still unproven

✓ Checkpoint — what you can now do
  • I can say what the Raspberry Pi imager does before first boot that flashing a Jetson cannot, and why that leaves exactly one attended step in this volume.
  • Given inet 192.168.1.137/24 ... dynamic valid_lft 84903sec, I can point at the word that means the address may not be there tomorrow.
  • I know why nmcli connection up kills the SSH session that issued it, and that the board is fine afterwards.
  • I can explain what socket.if_nameindex() supplies, what the ioctl supplies, and why the address sits at bytes 20 through 24 of the reply.
  • Handed a report claiming "no IP address" from a board I am connected to, I can name 127.0.1.1 and the file that put it there.
  • I can say which of the four checks fails first on a board whose Ethernet cable fell out, and which one still passes.
⚡ Exercises — try first, then reveal
Exercise 1 — find out what the root filesystem is really on. JetPack will boot happily from a microSD card, and everything afterwards will be slower for it. Add a check that reports the device behind / and passes only when it is NVMe.

/proc/mounts lists every mounted filesystem as device, mount point, type, and options. Find the line whose mount point is exactly /:

def root_device() -> str:
    """The device backing the root filesystem, per /proc/mounts."""
    with open("/proc/mounts") as handle:
        for line in handle:
            device, mount_point, *_rest = line.split()
            if mount_point == "/":
                return device
    return "unknown"


if __name__ == "__main__":
    device = root_device()
    print(f"root filesystem on {device}: "
          f"{'nvme' if device.startswith('/dev/nvme') else 'not nvme'}")
$ ssh glados-jetson python3 root_device.py   # measured on the bench — yours will vary
root filesystem on /dev/nvme0n1p1: nvme

Add it to run_audit as a fifth entry and the verdict line counts to five on its own, because it counts the dictionary instead of a hard-coded number. A board booted from microSD prints /dev/mmcblk0p1 and fails the check, which is the moment to move the install while nothing depends on it yet.

Exercise 2 — make the audit comparable to itself. Write each run to a timestamped JSON file, so that a check which used to pass and now fails is visible without remembering what yesterday looked like.
import json
import time


def save_audit(results: dict[str, tuple[bool, str]], directory: str = ".") -> Path:
    stamp = time.strftime("%Y%m%dT%H%M%S")
    path = Path(directory) / f"audit-{stamp}.json"
    snapshot = {
        "captured": stamp,
        "host": socket.gethostname(),
        "checks": {name: {"ok": ok, "detail": detail} for name, (ok, detail) in results.items()},
    }
    path.write_text(json.dumps(snapshot, indent=2))
    return path
$ ssh glados-jetson python3 jetson_setup_check.py --save   # measured on the bench
saved audit-20260812T094118.json
  "reachable address": {"ok": true, "detail": "eth0 192.168.1.100, l4tbr0 192.168.55.1"}

The dictionary run_audit already returns converts to JSON with no reshaping, which is the payoff for keeping gathering separate from printing. Diffing two of these files after a JetPack update or a power-supply swap turns "something feels off" into a list of exactly which checks changed.

Exercise 3 (stretch) — prove the cable route works before you need it. Unplug the Ethernet cable, connect the board's USB-C port to your workstation, and run the audit over that link.

From the workstation, ssh glados@192.168.55.1 python3 jetson_setup_check.py. The report comes back with eth0 gone from the interface list, the reachable-address check passing on l4tbr0 alone, and the other three checks unchanged. Do this once while the board is still on your desk and the monitor is still within reach, because the day you need this route is the day the board is inside a chassis and the switch it used to talk to is in another room. Note the address is the same 192.168.55.1 on every Jetson developer kit, so plugging in two of them at once gives your workstation two routes to one address and neither works.

The board has a fixed address, answers to a name your workstation knows, accepts a key instead of a password, and can prove all of that in one command run from your desk. The monitor and keyboard can go back in the drawer. What none of those four checks touched is the reason this board replaced the Pi at all: the GPU. Next comes the awkward gap between a driver that reports a healthy GPU and a compiler the shell cannot find, which is one of the few failures where both halves of the system are telling the truth.