The Archive Becomes a Table

$ go run ./cmd/climb -mode grow -dir /tmp/climb-78
climb: two centuries of the valley, kept as a file and nothing else

  a 16x12 valley, 21 browsers off stream 12 and 8 hunters off stream 22
  every birth and every death appended to climb-78.jsonl, and nothing in the valley reads it

   year   brwsrs  hunters      born      gone       bytes
     25      238        0      1734      1496    15860215
     50      237        0      1872      1635    17125399
     75      238        0      1991      1753    18216308
    100      238        0      1999      1761    18289556
    125      236        0      2001      1765    18308094
    150      235        0      2010      1775    18390624
    175      238        0      2024      1786    18518631
    200      237        0      2025      1788    18527773

  the archive at the end of 200 years
    born records                                    2025
    died records                                    1788
    records in all                                  3813
    bytes                                       18527773
    of those, the born records                  18439280
    and the died records                           88493
    sha256, first 16                    e04e8d6b021b6972
    still walking when it stopped                    237

  /tmp/climb-78/climb-78.jsonl

  719550 ticks in 5m32.469s, 2164 ticks a second (measured here; yours will differ)

Two thousand and twenty-five births and seventeen hundred and eighty-eight deaths, in eighteen and a half megabytes, and the timing on the last line is an eight-core Ryzen 7 3700X talking to itself. Everything else in the block is arithmetic and comes out the same anywhere.

The column of birth counts is the part to sit with, because it is not what a reader expects two centuries to look like. Seventeen hundred and thirty-four of those two thousand births happened in the first twenty-five years. The next hundred and seventy-five years produced two hundred and ninety-one. The population settles at about two hundred and thirty-seven browsers and then very nearly stops changing: this valley sets a child down on an empty cell, so once the ground is crowded a creature with a full store asking for a child is told there is nowhere to put it. A full valley writes almost no history. That is a fact about the valley and it is also the reason to keep an archive: the interesting years are over long before anybody thinks to ask about them.

$ podman exec -w /bench world-go go run ./cmd/climb -mode file -dir /tmp/climb-78 | tail -32 | head -16
  /tmp/climb-78/climb-78.jsonl
    records                                         3813
    of those, born                                  2025
    and died                                        1788
    bytes                                       18527773
    creatures still walking                          237

  the deepest line of record in it
    creature                                        1555
    steps up to the founder at the top of it          35
    and that founder is                               21

  the oldest animal still walking and the youngest
           426  born on tick   25887, generation   17, row 0
          2025  born on tick  688536, generation   21, row 0

Three facts and every one of them cost the same thing. The deepest line of record in this history runs thirty-five parents back from creature 1555 to creature 21, which was one of the twenty-nine animals this valley was founded with and has no ancestor above it. The oldest creature still walking is 426, born on tick 25,887, which is inside the eighth year. The youngest is 2025, born on tick 688,536, which is inside the hundred and ninety-second. Those two are alive at the same moment and their births are a hundred and eighty-four years apart.

$ podman exec -w /bench world-go go run ./cmd/climb -mode file -dir /tmp/climb-78 | tail -15
    the most recent ancestor both of them have       421
    born on tick                                   25761
    steps up from the older                            1
    steps up from the younger                          5
    ancestors the two of them share                   37

  and what any of that cost
    records read before the first answer            3813
    bytes parsed doing it                       18527773
    questions answered afterwards, for nothing        5

  a file has one reading order and it is the whole of it. Every question
  above came out of 3813 records held in memory at once, the cheapest of
  them cost the same 3813 as the dearest, and a process that wants one
  answer pays for all of them.

The two of them meet at creature 421, which is 426's own parent and stands five steps above 2025. Thirty-seven animals are ancestors of both. That is a real answer to a real question and the file gave it up without complaint, and the last block is the bill. Three thousand eight hundred and thirteen records parsed and eighteen and a half megabytes read, before any of the five questions above could be answered at all. All five were free afterwards, which sounds like a bargain until you notice what it costs: the only way to make one question cheap is to hold the entire history in memory, and the history is the thing that grows forever.

// internal/store/kin.go
// Rows turns an archive into creature rows, in the file's own order and
// in no other.
//
// The order is not a preference. creature.parent and creature.mate are
// foreign keys into creature itself, so a row naming a parent the table
// has not got is refused, and the archive is written in birth order,
// which means a parent is always above its child in the file. Sorting
// these by anything at all is how a load stops working.
//
// Generation is the one column that is not in the file. It is the
// length of the walk from a creature to the founder at the top of its
// line of record, and it is worked out here in one pass rather than by
// walking: the file is in birth order, so a parent's generation is
// already known by the time its child is read, and a child is one more
// than the parent that paid for it.
func Rows(recs []gene.Record) ([]Row, error) {
	gen := map[sim.EntityID]int{}
	var out []Row
	for _, rec := range recs {
		if rec.R != gene.Born {
			continue
		}
		// ... and a refusal for a creature born twice in one archive ...
		g := 0
		if rec.Of != 0 {
			up, ok := gen[rec.Of]
			if !ok {
				return nil, fmt.Errorf(
					"store: creature %d names parent %d, which is not above it in the file: "+
						"an archive out of birth order is an archive nothing can load", rec.ID, rec.Of)
			}
			g = up + 1
		}
		gen[rec.ID] = g

		blob, err := BlobOf(rec)
		if err != nil {
			return nil, err
		}
		raw, err := blob.JSON()
		if err != nil {
			return nil, err
		}
		out = append(out, Row{
			ID: rec.ID, Row: rec.Row, Species: rec.Kind, Tick: rec.Tick,
			Parent: ref(rec.Of), Mate: ref(rec.With), Generation: g, Genome: raw,
		})
	}
	return out, nil
}

Two decisions are in that function and both were made on the schema page. The genome goes in through BlobOf, which is the born record's own blocks under their own names with the row riding along, so the column holds a written form that was already checked every time an archive was opened, and no second description of an animal comes into existence. And ref turns a parent of nought into a null pointer, which the driver sends as SQL's null, because nought is a creature number and a founder has no parent at all.

generation is the column that is not in the file, and the loop is where it gets cheap. The archive is in birth order, so by the time a child is read its parent's generation is already in the map, and a child is one more than that. One pass, no climbing, and a lookup that fails is not a missing number: it is a file with a child above its parent, which is a file no database will take. That is checked here, where the file is still the obvious suspect, and not at the server, where the answer comes back as a constraint name.

$ go test ./internal/store/ -run 'GenerationIsCountedInOnePass|AnArchiveOutOfBirthOrder|TheGenomeColumnIsTheRecordsOwnBlocks|TheDeathsAreOnlyTheDiedRecords|TheRowsComeOutInTheFilesOwnOrder' -v
=== RUN   TestGenerationIsCountedInOnePassOverTheFile
    kin_test.go:60: 5 rows, generations counted without a single walk up a pedigree
--- PASS: TestGenerationIsCountedInOnePassOverTheFile (0.00s)
=== RUN   TestAnArchiveOutOfBirthOrderIsRefused
    kin_test.go:73: store: creature 3 names parent 1, which is not above it in the file: an archive out of birth order is an archive nothing can load
--- PASS: TestAnArchiveOutOfBirthOrderIsRefused (0.00s)
=== RUN   TestTheGenomeColumnIsTheRecordsOwnBlocks
    kin_test.go:96: every block of a born record in the column, and a genome back out of it
--- PASS: TestTheGenomeColumnIsTheRecordsOwnBlocks (0.00s)
=== RUN   TestTheDeathsAreOnlyTheDiedRecords
    kin_test.go:107: 2 deaths, in the order the file wrote them, every one naming a creature that is above it
--- PASS: TestTheDeathsAreOnlyTheDiedRecords (0.00s)
=== RUN   TestTheRowsComeOutInTheFilesOwnOrder
    kin_test.go:127: 5 rows in birth order, and every parent above the child that names it
--- PASS: TestTheRowsComeOutInTheFilesOwnOrder (0.00s)
PASS
ok  	theworld/internal/store	0.00s

Five tests on a pedigree of five creatures, and not one of them opens a socket. The second one is the load's whole safety argument written as something that fails: swap two records so that a child sits above its parent and the rows are refused before anything is sent. The last one is the rule easiest to miss, which is why it is there: the rows come out in the file's order and nothing anywhere is allowed to sort them.

// internal/store/kin.go
// Fill sends every row in one COPY.
//
// One statement is the whole design. A self-referencing foreign key is
// an after-row trigger, and an after-row trigger fires when the
// statement that queued it ends, not when the row lands: so inside one
// COPY every parent is already in the table by the time anything is
// checked. Break the load into two statements and that stops being
// true, which is why Chunks exists and why it is a different function
// with a different rule attached to it.
func (d *DB) Fill(ctx context.Context, rows []Row) (int64, error) {
	n, err := d.pool.CopyFrom(ctx, pgx.Identifier{"creature"}, creatureCols, copyRows(rows))
	if err != nil {
		return n, fmt.Errorf("store: loading %d creatures: %w", len(rows), err)
	}
	return n, nil
}

// Chunks sends the same rows as a run of COPYs of at most size rows
// each, which is what a load of a history too big to hold in memory has
// to become.
//
// Every chunk is its own statement and therefore its own check, so the
// file's order stops being a detail and becomes the thing that makes
// the load possible at all: a parent has to be in a chunk that has
// already gone. It hands back how many chunks it sent and how many rows
// landed, so a caller can say what a database holds after a load that
// stopped in the middle.
func (d *DB) Chunks(ctx context.Context, rows []Row, size int) (int, int64, error) {
	if size < 1 {
		return 0, 0, fmt.Errorf("store: a chunk of %d rows is not a chunk", size)
	}
	sent, done := 0, int64(0)
	for at := 0; at < len(rows); at += size {
		end := min(at+size, len(rows))
		n, err := d.pool.CopyFrom(ctx, pgx.Identifier{"creature"}, creatureCols, copyRows(rows[at:end]))
		done += n
		sent++
		if err != nil {
			return sent, done, fmt.Errorf("store: chunk %d, rows %d to %d: %w", sent, at+1, end, err)
		}
	}
	return sent, done, nil
}

COPY is not an insert with a faster wrapper round it. It is a different protocol: the client opens the copy, streams rows down the connection with no statement parsing and no plan and no round trip a row, and the server takes them until the stream ends. Eighteen megabytes of genome goes over in one exchange. It is also one statement, which turns out to matter more than the speed.

$ podman exec -w /bench world-go go run ./cmd/climb -mode load -dir /tmp/climb-78
climb: two centuries of births, out of a file and into a table, in one COPY

  what the load is about to send
    creature rows                                   2025
    death rows                                      1788
    bytes of genome                             18313213
    the file they came out of                   18527773
    the first row to go is creature 1, which names parent null
    the last  row to go is creature 2025, which names parent 1465

  before it
    creature          0 rows
    death             0 rows

  one COPY, and one after it for the deaths
    creature       2025 rows sent
    death          1788 rows sent
    creature       2025 rows now
    death          1788 rows now

  and what the server put on disk for it
    relation                  bytes   what it holds
    creature                 188416   the heap: the seven integer columns
    creature_genomes        8298496   the genomes, out of line and compressed
    creature_by_mate          57344   an index on mate
    creature_by_parent        65536   an index on parent
    creature_by_species       32768   an index on species
    creature_by_tick          65536   an index on tick
    creature_key              65536   an index on id
                            8773632

    the heap a question is answered out of        188416
    the genomes no question mentions             8298496
    the five indexes kept beside them             286720
    the indexes, as a share of the whole table      3.38%
    the heap,    as a share of the whole table      2.22%

Two statements for a history: one for the births and one for the deaths, in that order because death.id is a foreign key into creature and a death row for an animal the table has never heard of is refused. Within each of the two, the order is the file's.

The lower block is the schema chapter's argument arriving as bytes on a disk, and it came out better than the argument claimed. The genomes went in as eighteen million bytes of JSON and are sitting in eight point three million: Postgres does not keep a large value in the row it belongs to. Anything over about two kilobytes is compressed and moved into a store of its own, and the row keeps a pointer where the value was. So the creature table's heap, the thing a query walks when it is looking for a parent, is a hundred and eighty-eight thousand bytes for two thousand and twenty-five births, which is ninety-three bytes an animal. Two and two-tenths percent of what this history occupies is the part any question has ever mentioned.

The five indexes come to two hundred and eighty-six thousand bytes between them, which is three and four-tenths percent of the table. Four of those five were argued for on the schema page against the row they point into, and here is the row they point into, measured. An index is cheap next to a blob and expensive next to a pair of integers, and the same eight bytes of key is both.

WITH RECURSIVE up (id, parent, mate, tick, species, row, generation, step) AS (
      SELECT c.id, c.parent, c.mate, c.tick, c.species, c.row, c.generation, 0
        FROM creature c
       WHERE c.id = $1
    UNION ALL
      SELECT c.id, c.parent, c.mate, c.tick, c.species, c.row, c.generation, up.step + 1
        FROM up
        JOIN creature c ON c.id = up.parent
)
SELECT step, id, parent, mate, tick, species, row, generation FROM up ORDER BY step

The first SELECT is the anchor and it runs once: one creature, by its number, at step nought. The second runs over and over. Each time, up inside it means the rows the previous round produced and nothing else, so a round joins one row to the creature table and produces one row, which becomes the next round's up. When a round produces nothing the recursion stops, and a round produces nothing exactly when the creature it was handed has a null parent, because a join finds no match for null. That is the whole loop condition and no special case wrote it.

UNION ALL and not UNION is deliberate here. UNION would throw away a row it had produced before, which costs a comparison against everything so far on every round and buys nothing: a chain up parents of record cannot visit anybody twice, because every parent was born before its child and the ticks only go one way. The next query on this page is the case where that is not true.

ORDER BY step is the last line for the reason every query in this book has one. A recursive query produces its rounds in order and then hands them to whatever comes next, and what comes next is under no obligation to keep them in that order. The ordering is written down or it is luck.

$ podman exec -w /bench world-go go run ./cmd/climb -mode climb -dir /tmp/climb-78 | tail -31 | head -17
  the deepest line of record in this history, from creature 1555

   step   creature     parent       mate      tick  species   row
      0       1555       1548       1266     55212        1     0
      1       1548       1541       1451     54827        1     0
      2       1541       1451       1378     54322        1     0
      3       1451       1249        557     49264        1     0
    ... 29 rows between, one a generation
     33         93         44       null      5724        1     0
     34         44         21         11       902        1     0
     35         21       null       null       901        1     0

    36 rows, and the last of them is the founder: parent null
    steps up to it                                    35
    the walk over the file says                       35
    the generation column on the first row says       35
    three ways of counting one climb, and one answer

Thirty-six rows, with the run printing the two ends of the climb and counting what it left out. The bottom is the interesting part: creature 44 was born on tick 902, one tick after the founders stood up, to parents 21 and 11, and creature 93 has a null mate, which is a birth paid for by one parent with no second animal in reach over the threshold. Every row of the climb sits on species 1 and row 0. This line never crossed a species boundary in thirty-five generations, and it never changed rows, because two animals of different rows are refused each other before any distance between them is computed.

The three counts underneath are the check that makes the rest of this page worth reading. Thirty-five is how many steps the query climbed; thirty-five is what a walk up the file in memory says; thirty-five is what the generation column has held since the load computed it in one pass. The column is the one that could have drifted, because it is derived data written down once, and it agrees.

Climbing up parents of record is a chain. Real ancestry is not: every creature has two parents, so the set of animals above one creature widens as you climb and then narrows again, because a valley founded with twenty-nine animals runs out of ancestors and the same names start appearing on both sides. Two creatures share an ancestor when that ancestor is in both sets, and the useful one is the most recent, because in a history this old nearly everybody shares a founder and being told so says nothing.

WITH RECURSIVE
  mine (id, step) AS (
        SELECT c.id, 0 FROM creature c WHERE c.id = $1
      UNION
        SELECT u.up, m.step + 1
          FROM mine m
          JOIN creature c ON c.id = m.id
          CROSS JOIN LATERAL (VALUES (c.parent), (c.mate)) AS u (up)
         WHERE u.up IS NOT NULL
  ),
  theirs (id, step) AS (
        SELECT c.id, 0 FROM creature c WHERE c.id = $2
      UNION
        SELECT u.up, t.step + 1
          FROM theirs t
          JOIN creature c ON c.id = t.id
          CROSS JOIN LATERAL (VALUES (c.parent), (c.mate)) AS u (up)
         WHERE u.up IS NOT NULL
  ),
  ours  AS (SELECT id, min(step) AS step FROM mine   GROUP BY id),
  yours AS (SELECT id, min(step) AS step FROM theirs GROUP BY id)
SELECT c.id, c.tick, ours.step, yours.step, count(*) OVER ()
  FROM ours
  JOIN yours ON yours.id = ours.id
  JOIN creature c ON c.id = ours.id
 ORDER BY c.tick DESC, c.id DESC
 LIMIT 1

Four pieces. Two recursions, one a creature, each walking both parents instead of one: the VALUES pair turns a row into two candidate ancestors and the WHERE drops the nulls, so a founder contributes nothing and the branch ends. Two little queries that take the shortest number of steps to each ancestor, since a mesh reaches the same animal by several paths and the useful distance is the short one. Then a join, an ordering and a limit.

UNION without ALL is what makes the recursion safe, and it is the opposite decision from the climb for the opposite reason. Adding every path up would count the same ancestor once for every way of reaching it, and in a pedigree thirty-five deep the ways of reaching an animal are counted in thousands. Dropping a pair of numbers already produced bounds the work by how many creatures there are and how deep they stand, which is a number, instead of by how many routes exist, which is not.

count(*) OVER () is the last column and it is free. A window function with an empty OVER runs across the whole result before the LIMIT takes one row from it, so the count of shared ancestors comes back beside the most recent one instead of being a second question with a second walk behind it.

$ podman exec -w /bench world-go go run ./cmd/climb -mode meet -dir /tmp/climb-78
climb: where two living lines come back together

  creatures with a birth row and no death row        237
    the oldest of them       426, born on tick   25887, generation   17
    the youngest            2025, born on tick  688536, generation   21

  one statement, two recursions and a join
    the most recent ancestor both of them have       421
    born on tick                                   25761
    steps up from the older                            1
    steps up from the younger                          5
    ancestors the two of them share                   37

  the same five numbers, walked out of the file in memory
           421       25761           1           5          37
    the file and the table are answering with the same numbers

Two hundred and thirty-seven creatures have a birth row and no death row, which is a left join and a null test and is the whole of what putting deaths in a table of their own bought. The oldest and the youngest of them meet at creature 421, one step above the first and five above the second, with thirty-seven ancestors in common. Five numbers, and the same five came out of a walk over the file in memory, which is the only reason to believe either.

That comparison is the habit to keep. A new way of computing an old answer is worth exactly as much as the old way it agrees with, and the moment the file stops being consulted is the moment a wrong query becomes the truth.

// internal/store/kin.go
// How says which of the two ways of finding a row the server is allowed
// to use while it answers, which is how this book asks what an index is
// worth: the same question, the same rows and the same answer, reached
// once each way and counted both times.
//
// Chosen leaves the decision where it belongs. Descend forbids reading
// a table straight through, so anything with an index on it is reached
// through the index. Sweep forbids the three ways of reaching a row
// through an index, so the only road left is reading everything.
type How int

const (
	Chosen How = iota
	Descend
	Sweep
)

var forced = map[How][]string{
	Chosen:  nil,
	Descend: {"SET LOCAL enable_seqscan = off"},
	Sweep: {"SET LOCAL enable_indexscan = off",
		"SET LOCAL enable_indexonlyscan = off",
		"SET LOCAL enable_bitmapscan = off"},
}

// Steady is what every plan in this book is asked for: no costs, no
// timings, no summary, and no parallel workers, so that what comes back
// is the same on two machines and on two afternoons.
const steady = "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF, FORMAT JSON) "

The three settings are SET LOCAL, so they last until the transaction ends and change nothing about the database. They do not make the server wrong; they take a road away and make it answer the same question by another one. That is the only way to ask what a road is worth without a second database to compare against.

$ podman exec -w /bench world-go go run ./cmd/climb -mode cost -dir /tmp/climb-78 | tail -30 | head -10
  climbing creature 1555, 35 steps up to a founder, over 2025 rows

                                                  rows touched
    walking the file, records parsed first                3813
    the query, as the server chose                       70876
    the query, made to use the index                        37
    the query, made to read everything                   70876

    the file over the query on the index                 103.1
    the sweep over the query on the index               1915.6

Thirty-seven against seventy thousand eight hundred and seventy-six. The index turns a climb that reads the whole table once a generation into one that reads about a row a generation, and against the file it is a hundred and three to one on records touched. Those two numbers are the answer this chapter was asked for.

The second line is the one that changes the argument. Left to itself, the server read everything. The plan it chose and the plan it is forced into when the index is taken away are the same plan, to the row, and the plan itself says so if you ask for it with no numbers in it at all.

$ podman exec -w /bench world-go go run ./cmd/climb -mode climb -dir /tmp/climb-78 | tail -13
  the plan the server made, with the costs switched off
    Sort
      Sort Key: up.step
      CTE up
        ->  Recursive Union
              ->  Index Scan using creature_key on creature c
                    Index Cond: (id = '1555'::bigint)
              ->  Hash Join
                    Hash Cond: (c_1.id = up_1.parent)
                    ->  Seq Scan on creature c_1
                    ->  Hash
                          ->  WorkTable Scan on up up_1
      ->  CTE Scan on up

That is the whole of a plan with COSTS OFF on it: node types, what each one reads, and the conditions it reads them under. There is not a number in it to drift. The anchor is an Index Scan using creature_key because the anchor asks for one row by its number and there is nothing to weigh up. Underneath it, the recursive term is a Hash Join whose outer side is a Seq Scan on creature: every round, the server builds a hash of the one row the last round produced and then reads the entire table past it looking for a match. The WorkTable Scan is that one row. This is the plan the counts below are counting.

$ podman exec -w /bench world-go go run ./cmd/climb -mode cost -dir /tmp/climb-78 | tail -19 | head -10
  what each plan actually read, node by node
    as the server chose                        rows  loops  rows touched
      Index Scan using creature_key               1      1             1
      Seq Scan on creature                     2025     35         70875
    made to use the index                      rows  loops  rows touched
      Index Scan using creature_key               1      1             1
      Index Scan using creature_key               1     36            36
    made to read everything                    rows  loops  rows touched
      Seq Scan on creature                        1      1             1
      Seq Scan on creature                     2025     35         70875

Two nodes a plan, because a recursive query has two halves and each half reads the table once. In the plan the server picked, the anchor uses the index and the recursive term does not: thirty-five sweeps of two thousand and twenty-five rows to find thirty-five parents. Made to descend, both halves use creature_key and the recursive term runs thirty-six times, once a round including the empty round at the end. A plan reports rows as an average over the runs of a node, so the last descent, which found no row, is rounded up into the thirty-six; the honest reading of that line is thirty-six lookups that between them fetched thirty-five rows.

Now the part that makes this a measurement and not a demonstration. Two of the three numbers cannot move. Sweeping a table of two thousand and twenty-five rows thirty-five times is seventy thousand eight hundred and seventy-five rows on any machine that ever runs it, and descending an index thirty-six times is thirty-six lookups. The first line is different in kind: it is a decision the server made, and it made it because reading this table straight through is not expensive. The heap is a hundred and eighty-eight thousand bytes, which is twenty-three pages, and a server reads pages that lie next to each other far more cheaply than pages it has to jump between. Twenty-three pages read in order against thirty-five jumps into an index and thirty-five more into the heap is not the landslide the row counts make it look like.

So the honest statement of the result has two halves. The index reduces the work of this question by a factor of nineteen hundred, measured. And at this table's size the server does not take it, and is right not to. What makes that reproducible is that everything in the block except the first line is arithmetic over a fixed number of rows: run it on a bigger machine, a slower disk, a different afternoon, and thirty-six descents are still thirty-six descents.

$ podman exec -w /bench world-go go run ./cmd/climb -mode cost -dir /tmp/climb-78 | tail -8
  and the meeting of creatures 426 and 2025, the same three ways
    walking the file, records parsed first                3813
    the statement, as the server chose                  131627
    the statement, made to use the index                  3412
    the statement, made to read everything              131627

  the file is 18527773 bytes and the whole of it is parsed before the first
  answer; the table holds 2025 rows and the climb reads 37 of them.

The meeting statement is a harder question and the numbers say so. Two recursions over a widening mesh touch three thousand four hundred and twelve rows on the index, against a hundred and thirty-one thousand reading everything, and against three thousand eight hundred and thirteen records for the file. That is the one place on this page where the file and the index are within a factor of two of each other, and the reason is plain: the mesh above two creatures in this valley is a large fraction of the valley, so a query that visits all of it is doing most of the work a full pass does. The file wins nothing by it, because the file has to do the same work again for the next pair and the table does not.

Two things underneath all of that are checkable without a server anywhere near them, and both of them are the sort of thing that would be embarrassing to have wrong. Rows touched is rows times runs, added up over the nodes that read one table and no others. And the rule this whole database was built under still holds on the day a package in it learned to read a pedigree.

$ go test ./internal/store/ -run 'RowsTouchedIsRowsTimesLoops|NoSimulationPackageImportsTheStore' -v
=== RUN   TestRowsTouchedIsRowsTimesLoops
    kin_test.go:151: 34 rows out of the creature table across 34 index descents, and no other node counted
--- PASS: TestRowsTouchedIsRowsTimesLoops (0.00s)
=== RUN   TestNoSimulationPackageImportsTheStore
    rule_test.go:61: every file in every sealed package read, and not one of them names it
--- PASS: TestNoSimulationPackageImportsTheStore (0.00s)
PASS
ok  	theworld/internal/store	0.00s

The first is arithmetic over a plan somebody wrote down by hand, so a node that reads a different table is not counted and a node with no table under it is not counted either. The second is the rule from the chapter that opened this database, run again here because this chapter is where the temptation lives: it would have been so easy to let internal/gene ask the creature table who a genome's parent was. It does not, nothing else does, and a valley's tick still cannot open a transaction.