Ollama on Jetson

# on the board, over ssh
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
ollama run llama3.2:3b "Say exactly: facility online" > /dev/null
ollama ps

$ ollama ps   # measured on the bench — yours will vary
NAME            ID              SIZE      PROCESSOR    UNTIL
llama3.2:3b     a80c4f17acd5    3.5 GB    100% GPU     4 minutes from now

ollama ps lists the models currently held in memory, and the PROCESSOR column is the first thing anyone points at. Read it precisely: it reports how Ollama placed the model when it loaded it. A model too large for the memory Ollama believes the GPU can have gets split, and this column then says something like 47%/53% CPU/GPU, with the layers that did not fit running on the processor. That is real information and it is still the accused describing its own conduct. The UNTIL column matters later: five minutes after the last request, Ollama unloads the model and the next request pays the load time again.

# terminal one, on the board
sudo tegrastats --interval 1000

# terminal two, on the board
ollama run llama3.2:3b "Explain what a semiconductor is, in three sentences."

$ sudo tegrastats --interval 1000   # measured on the bench — yours will vary
RAM 3894/7620MB SWAP 0/3810MB CPU [6%@1728,4%@1728,3%@1728,5%@1728,2%@1728,3%@1728] GR3D_FREQ 0% cpu@50.5C soc0@49.1C tj@51.2C VDD_IN 4821mW
RAM 6142/7620MB SWAP 0/3810MB CPU [22%@1728,18%@1728,9%@1728,11%@1728,8%@1728,7%@1728] GR3D_FREQ 61% cpu@52.0C soc0@50.4C tj@52.9C VDD_IN 9942mW
RAM 6148/7620MB SWAP 0/3810MB CPU [9%@1728,7%@1728,5%@1728,6%@1728,4%@1728,5%@1728] GR3D_FREQ 99% cpu@53.4C soc0@51.8C tj@54.1C VDD_IN 13508mW
RAM 6147/7620MB SWAP 0/3810MB CPU [8%@1728,6%@1728,6%@1728,5%@1728,4%@1728,4%@1728] GR3D_FREQ 98% cpu@54.1C soc0@52.2C tj@54.6C VDD_IN 13704mW

tegrastats is the Jetson's own instrument, and on this board it answers a question nvidia-smi will not. The GPU here is on the same die as the CPU and reads the same memory pool, so nvidia-smi has no separate video memory to report and usually prints an empty process table even while inference is running. GR3D_FREQ is the field that counts: the percentage of the last interval the graphics engine spent busy. Idle at 0, climbing to 99 as the tokens come out, with RAM up by about 2.3 GB and VDD_IN, total board power, nearly tripled. Something is loading the GPU.

Something. A percentage sampled once a second tells you the engine was busy; it does not tell you how much of the answer it produced, and it cannot be compared against last month's reading or against the Pi. Neither witness gives you a number with tokens in it. That is the next stage.

# labs/ollama_gpu_check.py — runs on the board, on the python3 JetPack shipped
import json
import time
import urllib.request

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
MODEL = "llama3.2:3b"
PROMPT = "In two sentences, what is a GPU good at that a CPU is not?"


def generate(prompt: str, num_gpu: int | None = None, timeout: float = 180.0) -> dict:
    """One non-streamed generation. num_gpu=0 forces every layer onto the CPU."""
    body: dict = {"model": MODEL, "prompt": prompt, "stream": False}
    if num_gpu is not None:
        body["options"] = {"num_gpu": num_gpu}
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(body).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as reply:
        return json.loads(reply.read().decode("utf-8"))


if __name__ == "__main__":
    started = time.monotonic()
    reply = generate(PROMPT)
    wall = time.monotonic() - started
    for key in ("total_duration", "load_duration", "prompt_eval_count",
                "prompt_eval_duration", "eval_count", "eval_duration"):
        print(f"{key:<21}{reply.get(key)}")
    print(f"{'wall clock (s)':<21}{wall:.2f}")

$ ssh glados-jetson python3 ollama_gpu_check.py   # measured on the bench — yours will vary
total_duration       7810722000
load_duration        4912000000
prompt_eval_count    21
prompt_eval_duration 73000000
eval_count           54
eval_duration        2814722000
wall clock (s)       7.83

"stream": False is the reason those counters exist in one place. A streamed reply arrives as one JSON object per token and only the last object, the one with done: true, carries the totals; with streaming off you get a single object with the finished sums in it. The ten-digit numbers are nanoseconds, which is the unit Go's duration type prints and Ollama is written in Go. Three of them describe different work and the fourth contains the other three: 4.9 seconds reading 3.5 GB of weights off the NVMe drive, 0.073 seconds reading the prompt, 2.8 seconds writing the answer, and 7.81 seconds of total_duration around all of it. The wall clock agrees to within the time urllib spent parsing.

NS_PER_S = 1_000_000_000


def rate(count: int, duration_ns: int) -> float:
    """Tokens per second, guarding a zero or missing duration."""
    if not count or not duration_ns:
        return 0.0
    return count / (duration_ns / NS_PER_S)


def phases(reply: dict) -> dict[str, float]:
    """The four numbers that describe one generation."""
    return {
        "load (s)": reply.get("load_duration", 0) / NS_PER_S,
        "prefill (t/s)": rate(reply.get("prompt_eval_count", 0),
                              reply.get("prompt_eval_duration", 0)),
        "decode (t/s)": rate(reply.get("eval_count", 0),
                             reply.get("eval_duration", 0)),
        "total (s)": reply.get("total_duration", 0) / NS_PER_S,
    }


if __name__ == "__main__":
    for name, value in phases(generate(PROMPT)).items():
        print(f"{name:<15}{value:>8.2f}")

$ ssh glados-jetson python3 ollama_gpu_check.py   # measured on the bench — yours will vary
load (s)           4.91
prefill (t/s)    287.67
decode (t/s)      19.18
total (s)          7.81

Two rates, and they differ by a factor of fifteen on the same hardware in the same second. Prefill is the model reading your prompt: all 21 tokens are known in advance, so the arithmetic for all of them runs at once, and a GPU eats that for breakfast. Decode is the model writing its answer, one token at a time, each one an input to the next, and no amount of parallel hardware makes token 30 available before token 29 exists. Decode is the rate a person waiting for a sentence actually feels, and it is the only one of the two you should ever quote as "tokens per second".

if __name__ == "__main__":
    accelerated = phases(generate(PROMPT))["decode (t/s)"]
    forced_cpu = phases(generate(PROMPT, num_gpu=0))["decode (t/s)"]
    print(f"decode, as installed : {accelerated:6.2f} t/s")
    print(f"decode, num_gpu=0    : {forced_cpu:6.2f} t/s")
    print(f"the GPU is worth     : {accelerated / forced_cpu:6.2f}x")

$ ssh glados-jetson python3 ollama_gpu_check.py   # measured on the bench — yours will vary
decode, as installed :  19.18 t/s
decode, num_gpu=0    :   4.58 t/s
the GPU is worth     :   4.19x

num_gpu is the number of model layers Ollama offloads to the GPU, and zero means none of them. This is the whole chapter in one call: the same board, the same model, the same prompt, run once as installed and once with the GPU forbidden. If the two numbers had come back equal, the installed configuration was already running on the CPU and the install log was lying by omission. They came back a factor of four apart, so the GPU is carrying the generation. Expect both calls to be slow the first time you run them, because switching num_gpu evicts the resident model and reloads it onto the other processor.

CPU_BASELINE_TPS = 4.58    # measured on this board with num_gpu=0
ACCEL_FLOOR_TPS = 12.0     # two and a half times the baseline, rounded down


def verdict(decode_tps: float) -> str:
    """Where one decode rate sits between the CPU number and the GPU number."""
    if decode_tps >= ACCEL_FLOOR_TPS:
        return "GPU carried the generation"
    if decode_tps >= CPU_BASELINE_TPS * 1.5:
        return "partial: some layers are running on the CPU"
    return "CPU only: the GPU contributed nothing"


def main() -> None:
    started = time.monotonic()
    reply = generate(PROMPT)
    wall = time.monotonic() - started
    p = phases(reply)

    print("=" * 58)
    print(f"  Ollama throughput check: {MODEL}")
    print("=" * 58)
    print(f"  Reply      : {reply.get('response', '').strip()[:60]}...")
    print(f"  Load       : {p['load (s)']:6.2f} s   weights read into memory")
    print(f"  Prefill    : {p['prefill (t/s)']:6.2f} t/s over "
          f"{reply.get('prompt_eval_count', 0)} prompt tokens")
    print(f"  Decode     : {p['decode (t/s)']:6.2f} t/s over "
          f"{reply.get('eval_count', 0)} generated tokens")
    print(f"  Wall clock : {wall:6.2f} s")
    print("-" * 58)
    print(f"  {verdict(p['decode (t/s)'])}")
    print(f"  floor {ACCEL_FLOOR_TPS:.1f} t/s, CPU baseline {CPU_BASELINE_TPS:.2f} t/s")
    print("=" * 58)


if __name__ == "__main__":
    main()

$ ssh glados-jetson python3 ollama_gpu_check.py   # measured on the bench — yours will vary
==========================================================
  Ollama throughput check: llama3.2:3b
==========================================================
  Reply      : A GPU runs thousands of small arithmetic operations at the s...
  Load       :   4.91 s   weights read into memory
  Prefill    : 287.67 t/s over 21 prompt tokens
  Decode     :  19.18 t/s over 54 generated tokens
  Wall clock :   7.83 s
----------------------------------------------------------
  GPU carried the generation
  floor 12.0 t/s, CPU baseline 4.58 t/s
==========================================================

The two constants at the top are the only opinions in the file, and both are traceable to something measured rather than picked. CPU_BASELINE_TPS is what stage 5 recorded with the GPU forbidden. The floor sits well above it and well below the 19 this board actually reaches, so a thermally throttled run or a slightly larger model still passes while a fallback cannot. The middle band exists because placement is not a coin flip: a model that half fits runs half on each processor, and a rate of 8 t/s is neither a healthy GPU nor a pure CPU. Write your own numbers into those constants from your own board, and put the date beside them in a comment, because a floor derived from someone else's silicon is a guess wearing a decimal point.