A Frame Becomes a Sentence

# glados/vision.py
import base64
import json
import time
import urllib.request

import cv2
import numpy as np

CAMERA_INDEX = 0
WARMUP_FRAMES = 5

def capture_frame(index: int = CAMERA_INDEX, warmup: int = WARMUP_FRAMES) -> np.ndarray:
    cap = cv2.VideoCapture(index)
    if not cap.isOpened():
        raise RuntimeError(f"camera {index} would not open")
    try:
        for _ in range(warmup):
            cap.read()
        ok, frame = cap.read()
    finally:
        cap.release()
    if not ok:
        raise RuntimeError(f"camera {index} opened but returned no frame")
    return frame

if __name__ == "__main__":
    frame = capture_frame()
    height, width, channels = frame.shape
    print(f"{width}x{height}, {channels} channels, dtype {frame.dtype}")
    print(f"corner pixel (B, G, R): {frame[0, 0]}")

$ uv run python glados/vision.py   # your camera and its resolution will differ
640x480, 3 channels, dtype uint8
corner pixel (B, G, R): [ 41  38  34]

Two independent failures, two checks. isOpened() is false when the device could not be acquired at all: wrong index, no permission, another process already holding it. ok is false when the device opened and a particular read came back empty, which is what an unplugged cable looks like halfway through a session. Testing one and not the other leaves a real failure with no message attached to it.

The finally matters more than it looks. A camera is an exclusive operating-system resource: while your process holds it, nothing else on the machine can open it, including the next run of this same script. Releasing inside finally hands the device back within milliseconds even when the read raises, and it happens before the compression and the network call, which are the slow parts. Note the printed pixel too: OpenCV stores blue, green, red in that order, and the dimensions run height before width, because NumPy counts rows before columns.

DARK_MEAN = 12.0     # 0-255 brightness; below this there is nothing to describe
FLAT_SPREAD = 8.0    # below this the frame is one flat colour

def frame_stats(frame: np.ndarray) -> tuple[float, float]:
    """Mean brightness and its standard deviation, on a 0-255 scale."""
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    return float(gray.mean()), float(gray.std())

def frame_verdict(mean: float, spread: float) -> str:
    if mean < DARK_MEAN:
        return "too dark"
    if spread < FLAT_SPREAD:
        return "no contrast"
    return "usable"

if __name__ == "__main__":
    cap = cv2.VideoCapture(CAMERA_INDEX)
    for i in range(6):
        ok, frame = cap.read()
        mean, spread = frame_stats(frame)
        print(f"frame {i}: mean {mean:6.1f}  spread {spread:6.1f}  {frame_verdict(mean, spread)}")
    cap.release()

$ uv run python glados/vision.py   # your camera and your room will differ
frame 0: mean    4.6  spread    2.1  too dark
frame 1: mean   21.9  spread   14.7  usable
frame 2: mean   63.4  spread   35.0  usable
frame 3: mean   99.8  spread   49.6  usable
frame 4: mean  116.2  spread   53.8  usable
frame 5: mean  118.7  spread   54.2  usable

That ramp is the reason capture_frame throws away five frames before keeping one. The sensor wakes with its exposure closed and opens it over the first fraction of a second, so frame 0 is nearly black on a lit desk while frame 5 is the room. It is the pixel version of the bug chapter 4 caught in audio, where saving before the buffer was full wrote five seconds of zeros: in both cases the device answered before it had anything to say, and in both cases the file that came out was perfectly valid and completely empty.

Mean brightness alone would not be enough, since a lens cap and a dark room both read low. The spread is the sturdier signal, because it measures how much the pixels differ from each other: a real room has edges, highlights, a lamp, and its pixels scatter, while a covered lens is sensor noise around one value and lands near zero however the exposure is set. Neither number can tell you which of the two happened, and neither needs to. Both answers mean there is nothing here for a model to look at.

JPEG_QUALITY = 85

def encode_jpeg(frame: np.ndarray, quality: int = JPEG_QUALITY) -> bytes:
    ok, buf = cv2.imencode(".jpg", frame, [int(cv2.IMWRITE_JPEG_QUALITY), quality])
    if not ok:
        raise RuntimeError("JPEG encode failed")
    jpeg = buf.tobytes()
    if jpeg[:2] != b"\xff\xd8" or jpeg[-2:] != b"\xff\xd9":
        raise RuntimeError(f"incomplete JPEG: {len(jpeg)} bytes")
    return jpeg

def to_base64(jpeg: bytes) -> str:
    return base64.b64encode(jpeg).decode("ascii")

if __name__ == "__main__":
    frame = capture_frame()
    jpeg = encode_jpeg(frame)
    text = to_base64(jpeg)
    height, width, channels = frame.shape
    raw = height * width * channels
    print(f"raw pixels: {raw} bytes")
    print(f"jpeg:       {len(jpeg)} bytes  ({raw / len(jpeg):.1f}x smaller)")
    print(f"base64:     {len(text)} chars  ({len(text) / len(jpeg):.3f}x the jpeg)")
    print(f"first 14:   {text[:14]}")
    print(f"type:       {type(text).__name__}")

$ uv run python glados/vision.py   # byte counts depend on what the camera saw
raw pixels: 921600 bytes
jpeg:       41682 bytes  (22.1x smaller)
base64:     55576 chars  (1.333x the jpeg)
first 14:   /9j/4AAQSkZJRg
type:       str

Three of those numbers are arithmetic you can check. 480 rows times 640 columns times 3 colour channels is 921,600 bytes of raw pixels. Base64 turns every 3 bytes into exactly 4 characters, so 41,682 divides by 3 to give 13,894 groups and the string is exactly 55,576 characters long, four thirds of the input, every time. And /9j/ is not a coincidence: every JPEG in existence starts with the bytes ff d8 ff, and those three bytes always encode to those four characters. If your string starts with anything else, what you compressed was not an image.

The order of the two encodes is the whole design. Base64 the raw pixel array and you would be posting 1,228,800 characters over HTTP. Compress to JPEG first and you post 55,576, for a picture the model cannot tell apart from the original, because it resizes whatever you send down to its own input size of a few hundred pixels a side before it looks at anything. Shrink, then make it text. Doing it the other way round works and costs twenty-two times the bytes.

OLLAMA_URL = "http://127.0.0.1:11434/api/generate"
VISION_MODEL = "llava:7b"
VISION_PROMPT = (
    "You are GLaDOS from Portal. In one or two sentences, describe what you see. "
    "Be clinical and unimpressed. Do not describe anything you cannot see."
)

def build_payload(b64_image: str, prompt: str = VISION_PROMPT) -> bytes:
    return json.dumps({
        "model": VISION_MODEL,
        "prompt": prompt,
        "images": [b64_image],
        "stream": False,
    }).encode("utf-8")

if __name__ == "__main__":
    payload = build_payload(to_base64(encode_jpeg(capture_frame())))
    parsed = json.loads(payload)
    print(f"payload:  {len(payload)} bytes")
    print(f"keys:     {list(parsed.keys())}")
    print(f"images:   {len(parsed['images'])} item, type {type(parsed['images'][0]).__name__}")
    print(f"prompt:   {len(parsed['prompt'])} characters")

$ uv run python glados/vision.py
payload:  55789 bytes
keys:     ['model', 'prompt', 'images', 'stream']
images:   1 item, type str
prompt:   145 characters

Parsing the payload back and printing it costs one line and settles a whole category of argument. The request is 213 bytes of everything else wrapped around 55,576 bytes of picture, the image field really is a list holding one string, and no model was needed to establish either. When a call later comes back strange, you already know the request was well formed and can go looking elsewhere.

images is a list because the endpoint accepts several pictures in one request, which the last exercise in this chapter puts to use. stream: False asks for the finished reply in a single JSON object instead of one object per token, and that single object is also where the timing counters live, which is what the next stage reads.

def describe(b64_image: str, timeout: float = 180.0) -> dict:
    """One non-streamed vision generation. Returns Ollama's whole reply."""
    request = urllib.request.Request(
        OLLAMA_URL,
        data=build_payload(b64_image),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=timeout) as reply:
        return json.loads(reply.read().decode("utf-8"))

if __name__ == "__main__":
    frame = capture_frame()
    mean, spread = frame_stats(frame)
    print(f"frame: mean {mean:.1f} spread {spread:.1f} {frame_verdict(mean, spread)}")
    started = time.monotonic()
    reply = describe(to_base64(encode_jpeg(frame)))
    print(f"load {reply['load_duration'] / 1e9:.1f} s   "
          f"total {time.monotonic() - started:.1f} s")
    print(f"GLaDOS: {reply['response'].strip()}")

$ uv run python glados/vision.py   # her sentence will differ every time
frame: mean 118.7 spread 54.2 usable
load 11.6 s   total 19.4 s
GLaDOS: A desk, four cables that appear to lead nowhere, and a mug containing
        something that stopped being coffee several hours ago.

$ uv run python glados/vision.py   # run again within the keep-alive window
frame: mean 121.4 spread 55.0 usable
load 0.0 s   total 7.8 s
GLaDOS: The same desk. The mug has not moved. Neither, I notice, have you.

The three-minute timeout comes straight out of that first run. Eleven and a half of those nineteen seconds are load_duration, the weights coming off the drive into memory, and urllib's default would have given up in the middle of it. Run it again while the model is still resident and the load cost is zero. Anything that opens a model for the first time needs a timeout sized for the load, not for the work.

There is a second thing hiding in that load time, and ollama ps names it:

$ ollama ps
NAME        ID              SIZE      PROCESSOR    UNTIL
llava:7b    8dd30f6b0cb1    6.0 GB    100% GPU     4 minutes from now

Her chat model is gone from that list. The Jetson has one memory pool that the CPU and the GPU share, chapter 83 sized her chat model against it, and a six-gigabyte vision model does not fit beside it, so Ollama evicted one to make room for the other. The next thing you say to her pays a reload of the model that answers you. Either accept the swap because looking is rare, or pick a lighter vision model such as gemma3:4b that co-resides with her chat model. The measurement is what turns this from a surprise into a decision.

# labs/she_sees.py
from glados.core import GladOSCore
from glados.vision import (capture_frame, describe, encode_jpeg, frame_stats,
                           frame_verdict, to_base64)
from labs.system_config import build_default_config
from labs.wire_core import build_core

BLIND_REPLY = "I am looking at nothing at all. Riveting."

def look(core: GladOSCore) -> str:
    frame = capture_frame()
    mean, spread = frame_stats(frame)
    if frame_verdict(mean, spread) != "usable":
        core.speak(BLIND_REPLY)
        return BLIND_REPLY
    text = describe(to_base64(encode_jpeg(frame)))["response"].strip()
    core.speak(text)
    return text

if __name__ == "__main__":
    print(look(build_core(build_default_config())))

$ uv run python labs/she_sees.py   # lens covered on purpose
Loading whisper base on cpu...
Loading F5-TTS...
I am looking at nothing at all. Riveting.

The guard from stage 2 is doing more here than catching a bug. A vision model always answers, so an unusable frame costs nineteen seconds and buys a fluent description of a room the camera never saw. Two cheap statistics turn that into an instant honest sentence in her own voice, which is the cost cascade chapter 13 built for audio: a free check standing in front of an expensive one.