Asking the Chronicle a Question

$ go run ./cmd/beside -mode plan
beside: the schema this build holds, and not one statement of it run

  n    file                       sha256, first 16     bytes  stmts  what it makes
  0001 0001_world.sql             55b759d43aa929d4       641      1  world
  0002 0002_chronicle.sql         58878101d9384245       735      2  chronicle
  0003 0003_creature.sql          5f95f200d36d9d2e      1740      5  creature
  0004 0004_death.sql             43845ea86c947233       696      2  death
  0005 0005_species.sql           54029ed637c5b65d       858      1  species
  0006 0006_snapshot.sql          e80d9a5289d0f66c      1183      2  snapshot, snapshot_part
  0007 0007_chronicle_vector.sql  b3d9bebcf37a8003      2313      2  the vector extension, chronicle_vector

  7 files, 8166 bytes, 8 tables, applied in the order of their numbers and no other
  the seventh is the only one that installs anything, and the only one whose
  table can be dropped without this world losing a fact it does not still hold

  the eighth table, and where its width is written down
    declared in     0007_chronicle_vector.sql
    held in Go as   store.Dim
    both of them    768

The first six checksums are the six this schema has always had, unchanged, because a migration that has been applied anywhere is finished and a change goes in a new file. The seventh is two statements and a little over two kilobytes, most of which is the argument for four columns. What it makes is on the right of its row: an extension, and one table.

The last block is the one to watch. A vector column is declared at a width and the program filling it has to know the same width, so one fact now lives in two places. There is a test further down this page whose only job is to hold the two against each other.

-- The extension has to exist before the type below does. It is the
-- only statement in this schema that installs anything, and it is in
-- the same file as the one table that needs it, because "this server
-- can hold a vector" and "this world holds these vectors" are one
-- change and a migration is one change.
CREATE EXTENSION IF NOT EXISTS vector;

IF NOT EXISTS, in a book whose migration chapter spent a worked failure on the difference between a statement that is harmless to repeat and a system that is idempotent. It is the right tool here for the reason it was the right tool on the ledger table, and the reason needs stating precisely. The ledger records what this world's migrations did to this database. An extension belongs to the database and not to a migration: an operator may have installed it before this world ever ran, and the ledger has no row that could say so. Every other statement in this schema is protected by the ledger. This one has nothing to look itself up in, so it protects itself.

-- What a model turned one line of the chronicle into: one row a line,
-- a row a model.
--
-- It is a table beside the chronicle and not a column on it. The
-- chronicle is appended to and never rewritten, and a column filled
-- in afterwards would be an UPDATE of the one table this world has
-- promised never to take one. So the numbers go in a row of their
-- own, keyed by the entry they are about.
--
-- digest is the point of the table. A row of numbers is welded to the
-- weights that made it: two models' rows are not comparable in any
-- way at all, and a table that does not say which model filled it is
-- a table nobody can safely fill again. So it is not a note beside
-- the vector, it is half the key. model is the tag a person reads and
-- digest is what actually decided the numbers, because a tag can be
-- re-pointed at other weights and a digest cannot.
--
-- There is no column saying how wide a row is: the type says so, and
-- one fact in two places is two places that can disagree. There is no
-- wall clock either. Everything here can be worked out again from the
-- chronicle and the model, so when it was worked out answers nothing.
CREATE TABLE chronicle_vector (
    entry  bigint      NOT NULL,
    digest text        NOT NULL,
    model  text        NOT NULL,
    v      vector(768) NOT NULL,

    CONSTRAINT chronicle_vector_key PRIMARY KEY (entry, digest),

    -- Which of the two tables is the derived one, written down where
    -- the server keeps it: there is no vector for a line that was
    -- never written, and taking a line away takes its vectors.
    CONSTRAINT chronicle_vector_of FOREIGN KEY (entry)
        REFERENCES chronicle (entry) ON DELETE CASCADE,

    -- A digest is a sha256 written out in lower case, and a pin that
    -- is not one pins nothing.
    CONSTRAINT chronicle_vector_is_pinned CHECK (digest ~ '^[0-9a-f]{64}$'),
    CONSTRAINT chronicle_vector_names_its_model CHECK (model <> '')
);

Four columns, and the argument for each is the same argument the creature table's seven got: a column exists because something filters, joins, groups or climbs on it. entry is what this row is about and the only way back to the words. v is the thing being stored. The middle two are the chapter.

digest is half the primary key, and putting it there is the difference between a table that can be filled again and one that cannot. Suppose it were a note beside the vector, with entry alone as the key. The day somebody changes the embedding model, every row is a row of numbers about other weights, and the job asks “which lines have no vector yet” and is told: none of them. With the digest in the key the question becomes “which lines have no vector from these weights”, the answer is all of them, and old rows and new can sit in the table together while the change is going on. Both runs are further down this page.

model is the tag a person reads, and it is a separate column because the two answer different questions. A tag is a name somebody chose and can re-point at other weights in an afternoon; a digest is what actually decided the numbers. Storing only the tag puts this table back where it started; storing only the digest leaves sixty-four characters of hex where a person needs a name. Both are kept, and the key is built on the one that cannot lie.

Two columns are deliberately absent. Nothing says how wide a row is, because the type says so. And there is no timestamp: the snapshot table carries the only wall clock in this schema, and here everything can be worked out again from the chronicle and the model, so when it was worked out answers nothing anybody will ask.

The foreign key with ON DELETE CASCADE has nothing to do with deleting chronicle lines, since nothing deletes one. It writes down, where the server will enforce it, which of the two tables is the derived one: no vector exists for a line that was never written. The two checks cost one evaluation an insert and never again. digest ~ '^[0-9a-f]{64}' is a pattern match, because a pin that is not a sha256 in lower case pins nothing and the earliest complaint about that is the useful one.

And there is no second index. The backfill climbs one column, and that column is already the leading half of the primary key, so an ordered structure for it exists and came free. A spare index on rows three kilobytes wide would be cheap, and cheap is not a reason: nothing in this world asks a question a second index would answer.

$ podman kill world-db && podman start world-db
world-db
world-db
$ podman exec -w /bench world-go go run ./cmd/beside -mode up -twice
beside: bringing a database up to the schema this build holds

  the first run
    0001 0001_world.sql             applied
    0002 0002_chronicle.sql         applied
    0003 0003_creature.sql          applied
    0004 0004_death.sql             applied
    0005 0005_species.sql           applied
    0006 0006_snapshot.sql          applied
    0007 0007_chronicle_vector.sql  applied
    7 applied, 0 already there, and this run changed the database

  the second run
    0001 0001_world.sql             already applied, and its bytes still agree
    0002 0002_chronicle.sql         already applied, and its bytes still agree
    0003 0003_creature.sql          already applied, and its bytes still agree
    0004 0004_death.sql             already applied, and its bytes still agree
    0005 0005_species.sql           already applied, and its bytes still agree
    0006 0006_snapshot.sql          already applied, and its bytes still agree
    0007 0007_chronicle_vector.sql  already applied, and its bytes still agree
    0 applied, 7 already there, and this run changed nothing at all

  the vector extension, which 0007 asked for   0.8.6
  tables in this database, the ledger aside    8

The database is killed and started first so that it is properly empty: its data directory is a tmpfs, so a restart is a fresh server with nothing in it. Seven applied; then seven skipped, none applied, and the run says the bytes still agree, which is the checksum being checked and not merely stored. The second run sent no statements at all. It read seven rows and decided.

The extension's version on the last-but-one line is compared for the reason the server's own version string is: the image it arrives in is pinned by digest, so a version that moved means the pin moved, and finding that out is the point of pinning.

$ podman exec -w /bench world-go go run ./cmd/beside -mode show
beside: the world, as the server describes it back

  table              column       type                        null  indexes
  chronicle          entry        bigint                        no        2
                     tick         bigint                        no        1
                     kind         text                          no        0
                     text         text                          no        0
  chronicle_vector   entry        bigint                        no        1
                     digest       text                          no        1
                     model        text                          no        0
                     v            vector(768)                   no        0
  creature           id           bigint                        no        1
                     row          integer                       no        0
                     species      integer                       no        1
                     tick         bigint                        no        1
                     parent       bigint                       yes        1
                     mate         bigint                       yes        1
                     generation   integer                       no        0
                     genome       jsonb                         no        0
  death              id           bigint                        no        1
                     tick         bigint                        no        1
                     cause        text                          no        0
  snapshot           world        text                          no        1
                     tick         bigint                        no        1
                     taken_at     timestamp with time zone      no        0
                     parts        integer                       no        0
  snapshot_part      world        text                          no        1
                     tick         bigint                        no        1
                     part         text                          no        1
                     bytes        bytea                         no        0
  species            number       integer                       no        1
                     opened       bigint                        no        0
                     closed       bigint                       yes        0
                     holder       bigint                       yes        0
  world              world        text                          no        1
                     seed         bigint                        no        0
                     opened       bigint                        no        0
                     genesis      jsonb                         no        0
                     one          boolean                       no        1

  8 tables, 36 columns, and one of the columns is 768 numbers

  rows in each of them
    chronicle                 0
    chronicle_vector          0
    creature                  0
    death                     0
    snapshot                  0
    snapshot_part             0
    species                   0
    world                     0

The same schema read out of the server's catalogue by the program that made it, with the ledger left out and a column added for how many indexes name each column. The new table's entry and digest are named by one index between them, and model and v by none. Then the counts, all eight of them nought: a schema is made before there is anything to put in it.

$ podman exec -w /bench world-go go run ./cmd/beside -mode found
beside: the committed chronicle, put back into a database that has none

  the dump           chronicle.jsonl, 91 rows
  the document       11 keys, written at tick 0
  founded            11 lines at tick 0, in one transaction
  appended           80 lines, one a year, in the order they were written

  the table against the dump, row for row, ORDER BY entry
    rows in the dump                 91
    rows in the chronicle            91
    identical, entry tick kind text  91

  entry     tick   kind      text
  1         0      genesis   This is The World. I built it and I run it, a...
  11        0      genesis   The thing worth going after is the Wellspring...
  12        3150   count     year 1: 11 hobbs standing in The Hollow, 45 b...
  91        287550 count     year 80: 238 hobbs standing in The Hollow, 19...

  91 rows, 7032 bytes of text between them, and not one of them will ever
  be rewritten: that is the law the next table on this page is built around

The founding document goes in as eleven rows in one transaction, the way it always has, and the eighty yearly counts are appended one at a time in the order they were written. Then the table is read back and held against the committed dump, row for row, on all four columns. Ninety-one identical, which is what lets everything below be a number instead of a claim.

Seven thousand and thirty-two bytes of text. Hold on to that; the vectors about to be written for the same lines are considerably larger than the lines.

// internal/store/vector.go
// Text is how a vector is written down for the server: the numbers in
// square brackets, separated by commas, which is the literal the type
// parses.
//
// Each number is written in the shortest form that reads back as the
// same float32, so nothing is lost between the row a model produced
// and the row the table holds. The alternative is a fixed number of
// decimal places, which is either longer than it needs to be or
// quietly rounds, and there is no way to tell which by looking.
func Text(v []float32) string {
	var b strings.Builder
	b.WriteByte('[')
	for i, x := range v {
		if i > 0 {
			b.WriteByte(',')
		}
		b.Write(strconv.AppendFloat(nil, float64(x), 'g', -1, 32))
	}
	b.WriteByte(']')
	return b.String()
}

'g' with a precision of −1 asks for the shortest string that reads back as the same float32, and that is the part that matters. A fixed number of decimal places is either longer than it needs to be or quietly rounds, with no way to tell which by looking. This way the row the server holds is the row the model produced, bit for bit, and a test on this page proves it over numbers chosen to be awkward. The cost is that a vector on the wire is much larger than a vector in a column, and the next run says by how much.

// internal/store/vector.go
// Meant writes rows into the eighth table, all of them in one COPY.
//
// The creature table was loaded with the driver's own bulk copy,
// which builds the server's binary format. That road is shut here:
// the binary format needs the driver to know the type, and this one
// does not. So the copy is written out in the text format the server
// has always accepted, one row a line, columns separated by tabs, and
// the vector arrives as the literal Text produced.
//
// It is still one statement and one pass over the socket, which is
// the whole of what bulk loading buys.
func (d *DB) Meant(ctx context.Context, rows []Vector) (int64, error) {
	if len(rows) == 0 {
		return 0, nil
	}
	var b strings.Builder
	for _, r := range rows {
		if err := check(r); err != nil {
			return 0, err
		}
		fmt.Fprintf(&b, "%d\t%s\t%s\t%s\n", r.Entry, r.Digest, r.Model, Text(r.V))
	}
	conn, err := d.pool.Acquire(ctx)
	if err != nil {
		return 0, fmt.Errorf("store: filling the eighth table: %w", err)
	}
	defer conn.Release()
	const q = `COPY chronicle_vector (entry, digest, model, v) FROM STDIN`
	tag, err := conn.Conn().PgConn().CopyFrom(ctx, strings.NewReader(b.String()), q)
	if err != nil {
		return 0, fmt.Errorf("store: filling the eighth table: %w", err)
	}
	return tag.RowsAffected(), nil
}

The creature table was loaded with the driver's own bulk copy, which builds the binary format the server speaks natively. That road is shut here, because the binary format needs the driver to know the type. What is left is the text format: one row a line, tab-separated columns, still one statement and one pass over the socket, which is what a bulk load buys over a thousand inserts.

$ podman exec -w /bench world-go go run ./cmd/beside -mode load
beside: the committed rows, copied into the table beside the chronicle

  the fixture        vectors.json, 16 rows and 4 questions, 768 numbers each
  the model          nomic-embed-text:v1.5
  pinned at          970aa74c0a90ef74

  one COPY, in the text format the server has always taken
    rows before                           0
    rows the copy reported               16
    rows after                           16
    the literals it sent             136230 bytes
    one row, as the server holds it     3076 bytes
    all of them                       49216 bytes

  what the table holds, GROUP BY model, ORDER BY digest
    model                        pinned at            rows
    nomic-embed-text:v1.5        970aa74c0a90ef74       16

  16 of the chronicle's 91 lines now have a row from this model, and 75 do not

Sixteen rows in one statement. The two byte counts are the pair to look at: 136,230 bytes of literals went over the socket and 49,216 bytes came to rest, so writing the numbers out as text costs a factor of nearly three on the wire and nothing at all in the table. One stored row is 3,076 bytes, which is 768 four-byte numbers and four of the server's own bookkeeping, read back off the server with pg_column_size.

The last two blocks are the questions this table exists to answer: what is in it, grouped by the model that filled it, and how many lines that model has not seen. Seventy-five, because the fixture covers sixteen. Those seventy-five are the job's work and it does them two sections from here; the sixteen are what the next section puts questions to.

// internal/store/vector.go
// search is the statement itself, kept where a plan can be asked for
// it as well as an answer.
const search = `
SELECT v.entry, c.text, 1 - (v.v <=> $1::vector) AS cos
  FROM chronicle_vector v
  JOIN chronicle c ON c.entry = v.entry
 WHERE v.digest = $2
 ORDER BY v.v <=> $1::vector, v.entry
 LIMIT $3`

<=> is the extension's cosine distance: nought for two rows pointing the same way, one for square on, two for opposite. Subtracting it from one gives the cosine, which is what the rest of this book calls the measurement, so the two halves of this chapter can be printed in the same column and compared digit for digit.

The join is the reason any of this is in a database. The eighth table holds an entry and some numbers; the words are one column away, in the table the entry points at, so a search that returns lines of history instead of row identifiers costs one join clause and no memory at all.

WHERE v.digest = $2 does more than keep unwanted rows out: it is what makes the answer mean anything. In a table designed to hold two models' rows during a change, a search that did not name a model would be sorting numbers with no relation to each other, and would return a ranking, and would look fine.

The ORDER BY is on the distance ascending, which is the form the extension's own index types can use, with the entry on the end so that two rows scoring alike come back the same way round on every machine. Every query in this book carries one.

$ podman exec -w /bench world-go go run ./cmd/beside -mode ask -q 1
beside: one question, asked of the table and asked of the same rows in Go

  the question       Where does a corpse rot?
  the model          nomic-embed-text:v1.5
  pinned at          970aa74c0a90ef74
  scored by          1 - (v <=> the question), and by lang.Cos

  rank  entry  the server    lang.Cos         apart  the line
  1        10    0.602859    0.602859      6.32e-08  What is finished with goes to the Mid...
  2         3    0.533825    0.533825      2.15e-08  The animals are hobbs. One of them is...
  3         2    0.516247    0.516247      5.18e-08  The land is The Hollow. It is one val...
  4         7    0.474781    0.474781      5.49e-09  The doors between one part of the gro...
  5         1    0.451704    0.451704      4.92e-09  This is The World. I built it and I r...
  6        91    0.449923    0.449923      8.44e-10  year 80: 238 hobbs standing in The Ho...

  every one of the 16 rows scored both ways, not just the 6 printed
  the widest gap between the two is 6.32e-08, at entry 10
  which is agreement to 7 decimal places

Two columns of the same six numbers. The server computed the left one from the rows in the table, the Go computed the right one from the same rows read out of the committed fixture, and the third column is the distance between them. The widest gap anywhere in the sixteen is 6.32 in the eighth decimal place, so the two agree to seven, and the run says so instead of leaving it to be counted off the page.

Seven places is neither a disappointment nor luck. Both sides add up 768 products of numbers stored as float32, and they do not add them in the same order: one is a Go loop running from the first number to the last, the other is C the compiler is free to turn into several partial sums added at the end. Floating-point addition is not associative, so two correct implementations of one sum disagree in the last bits, and the third column is the size of that disagreement. A gap in the third decimal place would be the alarming case, because it would mean the two were computing different things.

The ranking itself is the one the last chapter got, on the same rows, out of a different program: the line about the Midden first, with no word in common with the question. Nothing about the search changed by moving into the database. What changed is that the rows no longer have to be anywhere in particular for it to run.

$ podman exec -w /bench world-go go run ./cmd/beside -mode ask -q 2 -top 4
beside: one question, asked of the table and asked of the same rows in Go

  the question       How does a newcomer get inside?
  the model          nomic-embed-text:v1.5
  pinned at          970aa74c0a90ef74
  scored by          1 - (v <=> the question), and by lang.Cos

  rank  entry  the server    lang.Cos         apart  the line
  1         6    0.636975    0.636975      4.55e-09  The way in is The Commons. Anybody wh...
  2         7    0.531150    0.531150       6.4e-09  The doors between one part of the gro...
  3         5    0.467180    0.467180      5.75e-08  People gather at Firstlight. Nobody i...
  4         9    0.447553    0.447553      3.58e-08  Past the mapped ground are the Unwrit...

  every one of the 16 rows scored both ways, not just the 4 printed
  the widest gap between the two is 5.78e-08, at entry 1
  which is agreement to 7 decimal places

The second question is the same result from a different direction, and its widest gap is 5.78 in the eighth place, at a row the printed four do not include. The run is being careful on purpose: it scores all sixteen both ways and reports the worst disagreement anywhere, not the worst among the rows that happened to be shown.

$ podman exec -w /bench world-go go run ./cmd/beside -mode cost -q 1
beside: what one question costs, counted in rows the server read

  the question       Where does a corpse rot?
  the table          16 rows, 768 numbers each
  asked for          EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)

  the plan the server made, with the costs switched off
  (two constants are cut here: the question's numbers, and the digest)
    Limit
      ->  Sort
            Sort Key: ((v.v <=> '[768 numbers]'::vector)), v.entry
            ->  Hash Join
                  Hash Cond: (c.entry = v.entry)
                  ->  Seq Scan on chronicle c
                  ->  Hash
                        ->  Seq Scan on chronicle_vector v
                              Filter: (digest = '970aa74c0a90ef74...'::text)

  LIMIT          returned    vector rows chronicle rows
  1                     1             16             91
  4                     4             16             91
  16                   16             16             91

  the LIMIT changes what comes back and not what was read: every row this
  model filled has to be scored before the closest one is known

  one question, in multiplications
    rows scored                    16
    numbers in a row               768
    multiply-adds                  12288

The plan carries two constants this book does not print: the question, which is 768 numbers a live model produced, and a sha256. Both are cut and the run says they were, which is the only honest way to publish a listing you have edited.

Read it from the bottom. The server reads the eighth table straight through, keeping the rows whose digest matches; builds a hash of them; joins the chronicle for the words; sorts by distance; takes the first few. There is not a number in any of those lines, so there is nothing in them to drift.

Then the counts, and the column that matters is the last but one. Asking for one row, four rows or all sixteen reads sixteen rows every time. The LIMIT decides what comes back and nothing about what was done, because a sort cannot know which row is closest until it has scored every row there is. That one fact decides how this table behaves as it grows, and seeing it measured before doing any arithmetic with it is what keeps the arithmetic honest.

// internal/store/vector.go
// Waiting is the backfill's whole question: which lines of the
// chronicle have no row from this model yet, oldest first.
//
// The model is named by its digest and not by its tag, because the
// question is about the weights. A world whose tag was re-pointed at
// other weights overnight asks this and is told, correctly, that
// every line is waiting.
func (d *DB) Waiting(ctx context.Context, digest string, limit int) ([]Line, error) {
	if limit < 1 {
		return nil, fmt.Errorf("store: a backfill of %d rows at a time is not a backfill", limit)
	}
	const q = `
SELECT c.entry, c.tick, c.kind, c.text
  FROM chronicle c
 WHERE NOT EXISTS (SELECT 1 FROM chronicle_vector v
                    WHERE v.entry = c.entry AND v.digest = $1)
 ORDER BY c.entry
 LIMIT $2`
	rows, err := d.pool.Query(ctx, q, digest, limit)
	if err != nil {
		return nil, fmt.Errorf("store: reading what is waiting to be embedded: %w", err)
	}
	defer rows.Close()
	var out []Line
	for rows.Next() {
		var l Line
		if err := rows.Scan(&l.Entry, &l.Tick, &l.Kind, &l.Text); err != nil {
			return nil, fmt.Errorf("store: reading what is waiting to be embedded: %w", err)
		}
		out = append(out, l)
	}
	return out, rows.Err()
}

NOT EXISTS against the eighth table, matching on the entry and the digest. That second condition is the reason the digest is in the key: without it the query cannot be written, because there is nothing to match on. ORDER BY on the entry does the oldest waiting line first, which matters not at all for correctness and quite a lot for a job that gets killed halfway through: what it finished is the beginning of the history, and what it starts with next time is where it stopped.

LIMIT is there for a second reason, and it is the difference between a job whose memory is a fixed batch and one whose memory grows with a history that only grows. It costs an extra round trip a batch to have.

// cmd/beside/fill.go
	cl := lang.Dial(srv)
	passes, asked, written := 0, 0, int64(0)
	for {
		batch, err := db.Waiting(ctx, m.Digest, c.batch)
		if err != nil {
			return err
		}
		if len(batch) == 0 {
			break
		}
		passes++
		var rows []store.Vector
		for _, l := range batch {
			v, err := cl.Embed(ctx, m.Name, l.Text)
			if err != nil {
				return err
			}
			asked++
			rows = append(rows, store.Vector{Entry: l.Entry, Digest: m.Digest, Model: m.Name, V: v})
		}
		n, err := db.Meant(ctx, rows)
		if err != nil {
			return err
		}
		written += n
	}

Read a batch, ask for a row apiece, write them back, go round again until nothing is waiting. There is no cleverness in it, and the absence is a design choice. What it leaves out matters more, because every one of these is something a job like this grows: it holds no transaction open across a model call, keeps nothing in memory between passes, has no notion of priority or importance, and decides nothing about which lines deserve to be embedded. Every line gets one row, oldest first.

It also does not run inside a tick. The pool it works through is the one this world has always used, four connections at most and a lifetime measured in minutes, sized that way precisely because the tick loop is not a caller. A tick is a hundred milliseconds. A round trip to a model server is measured in seconds: a tick that waited on one would not be late, it would have stopped being a tick.

$ podman exec -w /bench world-go go run ./cmd/beside -mode fill -batch 32
beside: the job that fills the eighth table, run once by hand

  the server         world-lm:11434, waiting up to 10m0s for an answer
  the model          nomic-embed-text:v1.5
  pinned at          970aa74c0a90ef74
  a pass takes       32 waiting lines at a time

  before
    lines in the chronicle             91
    rows in the eighth table           16
    lines with no row from this model  75

  after
    passes over what was waiting       3
    lines sent to the model            75
    rows the copies reported           75
    rows in the eighth table           91
    lines with no row from this model  0

  what the table holds, GROUP BY model, ORDER BY digest
    model                        pinned at            rows
    nomic-embed-text:v1.5        970aa74c0a90ef74       91

  not one number this run received is on this page, and yours would differ anyway
$ podman exec -w /bench world-go go run ./cmd/beside -mode fill -batch 32
beside: the job that fills the eighth table, run once by hand

  the server         world-lm:11434, waiting up to 10m0s for an answer
  the model          nomic-embed-text:v1.5
  pinned at          970aa74c0a90ef74
  a pass takes       32 waiting lines at a time

  before
    lines in the chronicle             91
    rows in the eighth table           91
    lines with no row from this model  0

  after
    passes over what was waiting       0
    lines sent to the model            0
    rows the copies reported           0
    rows in the eighth table           91
    lines with no row from this model  0

  what the table holds, GROUP BY model, ORDER BY digest
    model                        pinned at            rows
    nomic-embed-text:v1.5        970aa74c0a90ef74       91

  not one number this run received is on this page, and yours would differ anyway

Seventy-five lines in three passes of thirty-two, and then the same command again, on the same database, doing nothing whatsoever. The job is idempotent for the reason the migration runner is: it consults a record of what has been done before it does anything, and the record is a row in a table and not a flag somebody remembered to set. The table now holds ninety-one rows from one model, and not one of the numbers in it is on this page.

$ podman exec -w /bench world-go go run ./cmd/beside -mode cost -q 1
beside: what one question costs, counted in rows the server read

  the question       Where does a corpse rot?
  the table          91 rows, 768 numbers each
  asked for          EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF)

  the plan the server made, with the costs switched off
  (two constants are cut here: the question's numbers, and the digest)
    Limit
      ->  Sort
            Sort Key: ((v.v <=> '[768 numbers]'::vector)), v.entry
            ->  Hash Join
                  Hash Cond: (c.entry = v.entry)
                  ->  Seq Scan on chronicle c
                  ->  Hash
                        ->  Seq Scan on chronicle_vector v
                              Filter: (digest = '970aa74c0a90ef74...'::text)

  LIMIT          returned    vector rows chronicle rows
  1                     1             91             91
  4                     4             91             91
  91                   91             91             91

  the LIMIT changes what comes back and not what was read: every row this
  model filled has to be scored before the closest one is known

  one question, in multiplications
    rows scored                    91
    numbers in a row               768
    multiply-adds                  69888

The same command against the same database after the job has run. Ninety-one rows now, and the answer to all three questions is ninety-one. The plan is the same plan, the multiply-adds have gone from 12,288 to 69,888, and the rows read are exactly the rows there are, which is the relationship the arithmetic below is built on.

One caution about that block, and it is this volume's standing rule. Seventy-five of those rows were produced on this machine this afternoon by a live model, and no score computed from them appears here. What appears is a count, which is the same count whatever the numbers in them are.

$ go test ./internal/store/ -run 'TestTheMigrationsAreNumberedFromOneWithNoGaps|TestTheSchemaIsTheseTablesAndNoOthers|TestTheSchemaAndThisPackageAgreeOnTheWidth|TestAVectorIsWrittenDownWithoutLosingANumber|TestARowOfTheWrongWidthNeverReachesTheServer|TestAVectorWithNoModelBehindItIsRefused' -v
=== RUN   TestTheMigrationsAreNumberedFromOneWithNoGaps
    migrate_test.go:43: 7 migrations, numbered 1 to 7, every one of them read
--- PASS: TestTheMigrationsAreNumberedFromOneWithNoGaps (0.00s)
=== RUN   TestTheSchemaIsTheseTablesAndNoOthers
    migrate_test.go:79: 8 tables, in 7 files, and not one of them is the ledger
--- PASS: TestTheSchemaIsTheseTablesAndNoOthers (0.00s)
=== RUN   TestTheSchemaAndThisPackageAgreeOnTheWidth
    vector_test.go:37: 1 vector column declared in the schema, 768 wide, and store.Dim is 768
--- PASS: TestTheSchemaAndThisPackageAgreeOnTheWidth (0.00s)
=== RUN   TestAVectorIsWrittenDownWithoutLosingANumber
    vector_test.go:64: 8 numbers written and read back, every one of them the same bits, in 60 characters
--- PASS: TestAVectorIsWrittenDownWithoutLosingANumber (0.00s)
=== RUN   TestARowOfTheWrongWidthNeverReachesTheServer
    vector_test.go:77: refused: store: entry 1: a row of 767 numbers for a column of 768
--- PASS: TestARowOfTheWrongWidthNeverReachesTheServer (0.00s)
=== RUN   TestAVectorWithNoModelBehindItIsRefused
    vector_test.go:90: a row of the right width and no model behind it is still not a row this table takes
--- PASS: TestAVectorWithNoModelBehindItIsRefused (0.00s)
PASS
ok  	theworld/internal/store	0.005s

The second test is the one that moved. It pulls every CREATE TABLE out of the embedded SQL and fails unless the set is exactly the list this world keeps in Go. That list was seven names and is now eight:

// internal/store/migrate_test.go
var world = []string{
	"world", "chronicle", "creature", "death", "species", "snapshot", "snapshot_part",
	"chronicle_vector",
}

A test like that does not exist to stop a schema growing. It exists so that growing one is a decision somebody makes on purpose, in the same change that adds the file, which is what happened here.

The third keeps the width honest: it pulls vector(768) out of the migration with a pattern and holds the number against the constant the Go side uses, so the two places this fact lives cannot drift apart without something going red. The fourth writes eight awkward numbers out and reads them back, including a third, a value too small for any decimal place to reach, and the smallest float32 there is. Every one comes back as the same bits.

The last two are refusals and both happen before a socket is wanted. A row of the wrong width is caught with both widths in the message, which is what makes the error actionable and not merely correct. A row of the right width with no model behind it is refused too: a vector with no digest on it is the row this table exists to make impossible, and letting one in through a helper function would be a strange way to enforce a primary key.

$ podman exec -w /bench world-go go run ./cmd/beside -mode reembed
beside: what a backfill finds left to do after somebody changes the model

  the table          the eighth table, which records which model filled each row
  filled by          nomic-embed-text:v1.5, pinned at 970aa74c0a90ef74
  then configured    qwen2.5:0.5b-instruct-q4_K_M, pinned at c5396e06af294bd1
  which is           another of the three pins this world holds, and is asked nothing

  the chronicle holds 91 lines, and 16 of them were filled

  what the job finds waiting
    under the model that filled the table   75
    under the model now configured          91

  the second number is every line in the chronicle, and that is the right answer:
  a change of weights makes every row in the table a row about other weights.
  Nothing was rewritten for the job to be able to say so, and nothing had to be.
$ podman exec -w /bench world-go go run ./cmd/beside -mode reembed -naive
beside: what a backfill finds left to do after somebody changes the model

  the table          a vector table with no model on it, keyed by the chronicle line alone
  filled by          nomic-embed-text:v1.5, pinned at 970aa74c0a90ef74
  then configured    qwen2.5:0.5b-instruct-q4_K_M, pinned at c5396e06af294bd1
  which is           another of the three pins this world holds, and is asked nothing

  the chronicle holds 91 lines, and 16 of them were filled

  what the job finds waiting
    under the model that filled the table   75
    under the model now configured          75

  the same number twice, because the table cannot be asked the question. A row
  is there or it is not, and nothing in it says whose numbers those are, so the
  16 lines already filled are done for good, by whatever it was that filled them.

Read the middle block of each. The first run says 75 and 91: seventy-five lines still waiting under the model that filled the table, and ninety-one under the model configured now, which is every line there is. That second number is the right answer and getting it rewrote nothing. The sixteen rows already there stay where they are, correct about the weights that made them, while the job works through the whole chronicle again. Halfway through, the table holds both models' rows at once and the search names which set it wants.

The second run says 75 and 75, not because the answer is the same but because the question cannot be asked. The table has no column that could tell one model from another, so NOT EXISTS can only match on the entry, and a row is either there or it is not. The sixteen lines already done are done for good, by whatever it was that did them. Nothing errors, and every search against that table from then on sorts sixteen rows of one arithmetic against seventy-five of another and returns a ranking with no complaint at all.

That is the failure the two columns prevent; notice what kind it is. There is no error message in it anywhere. The naive table is a perfectly good table that answers every query it is given. It has lost the one fact needed to put it right, and it lost that at the moment it was designed.

The second pin in both runs is another of the three this world already holds, chosen for being a different sixty-four characters. Nothing is asked of it: a backfill's question is about a digest and not about what the digest names.