Rows of Neurons, Run in Order

// internal/mind/net.go
package mind

// Layer runs one row of neurons across one row of inputs. w holds the
// weight runs end to end, b holds one bias per neuron, and out takes one
// number per neuron. The outer loop picks the neuron; the inner loop is
// that neuron's own arithmetic and nothing else.
func Layer(in, w, b, out []float64) {
	for j := range out {
		row := w[j*len(in) : (j+1)*len(in)]
		sum := b[j]
		for i, x := range in {
			sum += row[i] * x
		}
		out[j] = sum
	}
}

// cmd/layers/main.go — the bench: no valley, no creature, one layer

// hand is one layer small enough to check on paper: four inputs, three
// neurons, fifteen numbers in the slice.
func hand() {
	in := []float64{0.5, -0.2, 1, 0}
	w := []float64{
		0.1, 0.2, -0.3, 0.4,
		-0.5, 1.0, 0.25, -0.1,
		2.0, 0.0, 0.5, 1.0,
	}
	b := []float64{0.05, 0, -1}
	out := make([]float64, len(b))

	fmt.Printf("one layer: %d inputs, %d neurons, %d numbers in the slice\n",
		len(in), len(b), len(w)+len(b))
	print1("inputs", in)
	fmt.Println()
	for j := range out {
		fmt.Printf("  neuron %d reads W[%2d..%2d], bias W[%2d]\n",
			j, j*len(in), (j+1)*len(in)-1, len(w)+j)
		fmt.Printf("    ")
		sum := b[j]
		for i, x := range in {
			fmt.Printf("%+.2f*%+.2f ", w[j*len(in)+i], x)
			sum += w[j*len(in)+i] * x
		}
		fmt.Printf("%+.2f = %+.4f\n", b[j], sum)
	}
	fmt.Println()
	mind.Layer(in, w, b, out)
	print1("Layer says", out)

	// the same three neurons again, each built as its own value over a
	// run of the flat slice instead of read out of it by the loop
	one := make([]float64, len(b))
	for j := range one {
		one[j] = mind.Neuron{W: w[j*len(in) : (j+1)*len(in)], B: b[j]}.Sum(in)
	}
	print1("one by one", one)
}

$ go run ./cmd/layers -mode hand
one layer: 4 inputs, 3 neurons, 15 numbers in the slice
  inputs       0.5000  -0.2000   1.0000   0.0000

  neuron 0 reads W[ 0.. 3], bias W[12]
    +0.10*+0.50 +0.20*-0.20 -0.30*+1.00 +0.40*+0.00 +0.05 = -0.2400
  neuron 1 reads W[ 4.. 7], bias W[13]
    -0.50*+0.50 +1.00*-0.20 +0.25*+1.00 -0.10*+0.00 +0.00 = -0.2000
  neuron 2 reads W[ 8..11], bias W[14]
    +2.00*+0.50 +0.00*-0.20 +0.50*+1.00 +1.00*+0.00 -1.00 = +0.5000

  Layer says  -0.2400  -0.2000   0.5000
  one by one  -0.2400  -0.2000   0.5000

The two bottom lines are the pencil answer, and the three above them are the pencil working. The bench prints the products because the products are the only thing here anybody can get wrong, and printing them turns a disagreement between you and the program into a hunt for one term instead of an argument about the whole layer.

The last line is the point of the exercise. A mind.Neuron holds its own weights and its own bias and answers with Sum; the layer holds nothing and reads a run of weights straight out of a slice somebody else owns. Build the three neurons one at a time, each over the run of four numbers the loop would have used, and they answer −0.2400, −0.2000 and 0.5000: the same three numbers to the last bit. The arithmetic did not change and was never going to. What changed is who owns the weights, and that turns out to decide everything about what a controller can be handed around as.

The inner loop of Layer is the neuron and nothing but: multiply, add, multiply, add, having started the running total at the bias. One line above it carries the whole design: row := w[j*len(in) : (j+1)*len(in)]. Neuron j's weights are the run of len(in) numbers starting at j*len(in), so neuron 0 owns W[0..3], neuron 1 owns W[4..7], neuron 2 owns W[8..11], and the slice knows nothing about any of that. The bookkeeping is one multiplication and one addition, and it is the same bookkeeping the frame buffer does when it turns a row and a column into one offset into a flat array of pixels.

// internal/mind/net.go

// Net is one controller: the size of its three rows, and every weight it
// owns in one flat slice.
//
// W holds four blocks, in this order: the hidden row's weights, one run
// of In per neuron; the hidden row's biases, one per neuron; the output
// row's weights, one run of Hid per output; the output row's biases.
// A neuron's weights sit end to end the way a row of pixels sits in the
// frame buffer, so hidden neuron h's weight on input i is W[h*In+i].
type Net struct {
	In  int // numbers the network is handed
	Hid int // neurons in the middle row
	Out int // numbers it hands back
	W   []float64
}

// Weights is how many numbers a network of these three sizes owns.
func Weights(in, hid, out int) int {
	return in*hid + hid + hid*out + out
}

// New is a network of these sizes with every weight left at zero.
func New(in, hid, out int) Net {
	return Net{In: in, Hid: hid, Out: out,
		W: make([]float64, Weights(in, hid, out))}
}

// HidW, HidB, OutW and OutB are where each of the four blocks starts in
// W. Nothing else anywhere works out an offset into the slice.
func (n Net) HidW() int { return 0 }
func (n Net) HidB() int { return n.In * n.Hid }
func (n Net) OutW() int { return n.HidB() + n.Hid }
func (n Net) OutB() int { return n.OutW() + n.Hid*n.Out }

The bench reads every boundary back out of those four methods rather than restating them, so an offset written down wrongly in the package would come out as a wrong column on the page.

// cmd/layers/main.go

// slice prints where each of the four blocks starts and ends, and what
// one pass through the network costs in multiplies.
func slice(i, h, o int) {
	n := mind.New(i, h, o)
	fmt.Printf("a %d-%d-%d network owns %d numbers\n", i, h, o, len(n.W))
	fmt.Printf("  %-16s %6s %6s %8s  %s\n", "block", "from", "to", "numbers", "what one entry is")
	fmt.Printf("  %-16s %6d %6d %8d  %s\n", "hidden weights", n.HidW(), n.HidB()-1,
		n.HidB()-n.HidW(), fmt.Sprintf("%d runs of %d", h, i))
	fmt.Printf("  %-16s %6d %6d %8d  %s\n", "hidden biases", n.HidB(), n.OutW()-1,
		n.OutW()-n.HidB(), "one per hidden neuron")
	fmt.Printf("  %-16s %6d %6d %8d  %s\n", "output weights", n.OutW(), n.OutB()-1,
		n.OutB()-n.OutW(), fmt.Sprintf("%d runs of %d", o, h))
	fmt.Printf("  %-16s %6d %6d %8d  %s\n", "output biases", n.OutB(), len(n.W)-1,
		len(n.W)-n.OutB(), "one per output")
	fmt.Printf("\n  hidden neuron h's weight on input i is W[h*%d+i]\n", i)
	for _, p := range [][2]int{{0, 0}, {0, i - 1}, {1, 0}, {h - 1, i - 1}} {
		fmt.Printf("    neuron %2d, input %2d  ->  W[%d]\n", p[0], p[1], p[0]*i+p[1])
	}
	fmt.Printf("\n  one pass multiplies %d times: %d*%d into the middle row, %d*%d out of it\n",
		i*h+h*o, i, h, h, o)
}

$ go run ./cmd/layers -mode slice
a 24-12-6 network owns 378 numbers
  block              from     to  numbers  what one entry is
  hidden weights        0    287      288  12 runs of 24
  hidden biases       288    299       12  one per hidden neuron
  output weights      300    371       72  6 runs of 12
  output biases       372    377        6  one per output

  hidden neuron h's weight on input i is W[h*24+i]
    neuron  0, input  0  ->  W[0]
    neuron  0, input 23  ->  W[23]
    neuron  1, input  0  ->  W[24]
    neuron 11, input 23  ->  W[287]

  one pass multiplies 360 times: 24*12 into the middle row, 12*6 out of it

Four methods and not one of them takes an argument, because each is a sum of sizes the network already knows. Writing them down once is the point. A layout like this is usually re-derived at every call site, and every re-derivation is a chance to be off by twelve; here there is exactly one place that says the hidden biases start at In*Hid, and everything that reads them goes through it.

The last line of the run is the cost of one thought: 360 multiplications and 360 additions, for one creature, on one tick. Multiply by however many creatures the valley is carrying and the total is still not frightening, which is a pleasant thing to establish early and a claim this chapter is not yet in a position to prove.

// internal/mind/net.go

// Forward runs in through both rows and writes the answer into out. hid
// is the caller's scratch for the middle row. Nothing here allocates:
// whoever owns the network makes its two buffers once and keeps them.
func (n Net) Forward(in, hid, out []float64) {
	Layer(in, n.W[n.HidW():n.HidB()], n.W[n.HidB():n.OutW()], hid)
	Layer(hid, n.W[n.OutW():n.OutB()], n.W[n.OutB():], out)
}

// Buffers is the scratch one run of this network needs: the middle row
// and the row it answers with.
func (n Net) Buffers() (hid, out []float64) {
	return make([]float64, n.Hid), make([]float64, n.Out)
}

Two things about those two lines. The first is that they run in order and have to: the second Layer call reads hid, so the first call must have finished writing all twelve of them before it starts. Layers within a row are independent and layers between rows are not, and that is the only ordering rule in the whole package. The second is that in and hid must be different slices. Hand the same one twice and the first call would overwrite the inputs it is still reading, halfway through, and Go would not say a word about it: they are both []float64 and the compiler has no opinion on whether they point at the same array.

Forward takes the buffers instead of making them, and that is a deliberate choice with a cost attached. It is uglier at the call site: three slices go in where one would do. What it buys is that a creature makes twelve floats and six floats once, when it is born, and then thinks for the rest of its life without asking the runtime for memory. A version that returned a fresh slice would be one line shorter and would hand the garbage collector two small allocations per creature per tick, forever.

Now it needs weights, and weights have to come from somewhere. This page takes them off a ruler: a repeating rule, stated once, that anybody can check by counting on their fingers. Where a controller's numbers ought to come from is a real question and a separate one, and a rule like this answers none of it. It only makes every figure below reproducible without a random stream anywhere in the chapter. What runs them is three lines of setup and one of reading: size the network off the ruler, take its two buffers, run the pass, then say which of the six came out highest.

// cmd/layers/main.go

// ruler fills a slice of n weights off a repeating rule instead of a
// random stream, so every number this bench prints can be checked by
// hand: weight k is k modulo 13, minus 6, over ten.
func ruler(n int) []float64 {
	w := make([]float64, n)
	for k := range w {
		w[k] = (float64(k%13) - 6) / 10
	}
	return w
}

// row is a stand-in for a sensor row: input i is i modulo 5, minus 2,
// over two, so the values run -1.0, -0.5, 0.0, 0.5, 1.0 and repeat.
func row(n int) []float64 {
	x := make([]float64, n)
	for i := range x {
		x[i] = (float64(i%5) - 2) / 2
	}
	return x
}

// run puts a row of numbers through the network and prints both rows it
// produces.
func run(i, h, o int) {
	n := mind.Net{In: i, Hid: h, Out: o, W: ruler(mind.Weights(i, h, o))}
	x := row(i)
	hid, out := n.Buffers()
	n.Forward(x, hid, out)

	fmt.Printf("a %d-%d-%d network on %d weights off the ruler\n", i, h, o, len(n.W))
	for k := 0; k < i; k += 12 {
		print1(fmt.Sprintf("in[%d..]", k), x[k:min(k+12, i)])
	}
	for k := 0; k < h; k += 12 {
		print1(fmt.Sprintf("hid[%d..]", k), hid[k:min(k+12, h)])
	}
	print1("out", out)
	best := 0
	for j, v := range out {
		if v > out[best] {
			best = j
		}
	}
	fmt.Printf("  the largest of the %d is out[%d] at %+.4f\n", o, best, out[best])
}

$ go run ./cmd/layers -mode run
a 24-12-6 network on 378 weights off the ruler
  in[0..]     -1.0000  -0.5000   0.0000   0.5000   1.0000  -1.0000  -0.5000   0.0000   0.5000   1.0000  -1.0000  -0.5000
  in[12..]     0.0000   0.5000   1.0000  -1.0000  -0.5000   0.0000   0.5000   1.0000  -1.0000  -0.5000   0.0000   0.5000
  hid[0..]    -0.3500  -0.0500  -1.0500  -0.1000  -0.4500   1.1500   0.1500  -0.2000  -0.5500   1.0500   1.3500  -0.9500
  out          1.1100   1.2100   0.8550   0.8900  -0.3750  -1.7050
  the largest of the 6 is out[1] at +1.2100

Check the first middle number by hand if you want the layout confirmed rather than believed. Neuron 0's weights are W[0..23], which the ruler makes −0.6, −0.5, and so on up to 0.6 at W[12] before starting over. Multiply each by the matching input, add the twenty-four products, and you get 0.05. Neuron 0's bias is W[288], and 288 divided by 13 leaves a remainder of 2, so the bias is (2 − 6) ÷ 10 = −0.40. 0.05 − 0.40 is −0.35, which is what the hid row prints in its first column.

The out row is six numbers on no particular scale, and the last line picks the largest. That is the whole interface a controller offers whatever ends up reading it: here are six figures, compare them yourself. None of them is a probability and none is bounded, so out[5] at −1.7050 reads as a low bid and never as a refusal.

// cmd/layers/main.go

// flatten multiplies a network's two rows of weights into the single row
// that does the same job: m[o*In+i] is what output o ends up doing with
// input i, and c[o] is what it adds whatever the inputs are.
func flatten(n mind.Net) (m, c []float64) {
	m = make([]float64, n.Out*n.In)
	c = make([]float64, n.Out)
	hw, hb := n.W[n.HidW():n.HidB()], n.W[n.HidB():n.OutW()]
	ow, ob := n.W[n.OutW():n.OutB()], n.W[n.OutB():]
	for o := 0; o < n.Out; o++ {
		c[o] = ob[o]
		for h := 0; h < n.Hid; h++ {
			v := ow[o*n.Hid+h]
			c[o] += v * hb[h]
			for i := 0; i < n.In; i++ {
				m[o*n.In+i] += v * hw[h*n.In+i]
			}
		}
	}
	return m, c
}

// collapse multiplies the two rows of weights together into the one row
// that does the same job, and checks it against the network.
func collapse(i, h, o, show int) {
	fmt.Println("two neurons in the middle, one output, worked on paper")
	small := mind.Net{In: 2, Hid: 2, Out: 1, W: []float64{
		1.0, 2.0,
		-0.5, 0.4,
		0.1, -0.2,
		3.0, 1.0,
		0.05,
	}}
	sx := []float64{0.5, -0.2}
	shid, sout := small.Buffers()
	small.Forward(sx, shid, sout)
	print1("inputs", sx)
	print1("middle", shid)
	print1("output", sout)
	sm, sc := flatten(small)
	print1("one layer", sm)
	print1("constant", sc)
	fmt.Printf("  and %+.2f*%+.2f %+.2f*%+.2f %+.2f = %+.4f\n",
		sm[0], sx[0], sm[1], sx[1], sc[0], sm[0]*sx[0]+sm[1]*sx[1]+sc[0])

	n := mind.Net{In: i, Hid: h, Out: o, W: ruler(mind.Weights(i, h, o))}
	x := row(i)
	hid, out := n.Buffers()
	n.Forward(x, hid, out)
	m, c := flatten(n)
	flat := make([]float64, o)
	for j := range flat {
		flat[j] = c[j]
		for k := range x {
			flat[j] += m[j*i+k] * x[k]
		}
	}
	fmt.Printf("\nthe same thing done to the %d-%d-%d network\n", i, h, o)
	fmt.Printf("  %d weights in two rows against %d in one\n", len(n.W), len(m)+len(c))
	print1("two rows", out)
	print1("one row", flat)
	worst := 0.0
	for j := range out {
		if d := math.Abs(out[j] - flat[j]); d > worst {
			worst = d
		}
	}
	fmt.Printf("  largest difference between them: %.3g\n", worst)
	fmt.Printf("\n  the collapsed row, first %d of its %d columns\n", show, i)
	for j := 0; j < o; j++ {
		print1(fmt.Sprintf("out %d", j), append(append([]float64{}, m[j*i:j*i+show]...), c[j]))
	}
	fmt.Printf("  (the last column is the constant, not a weight)\n")
}

$ go run ./cmd/layers -mode collapse
two neurons in the middle, one output, worked on paper
  inputs       0.5000  -0.2000
  middle       0.2000  -0.5300
  output       0.1200
  one layer    2.5000   6.4000
  constant     0.1500
  and +2.50*+0.50 +6.40*-0.20 +0.15 = +0.1200

the same thing done to the 24-12-6 network
  378 weights in two rows against 150 in one
  two rows     1.1100   1.2100   0.8550   0.8900  -0.3750  -1.7050
  one row      1.1100   1.2100   0.8550   0.8900  -0.3750  -1.7050
  largest difference between them: 7.22e-16

  the collapsed row, first 6 of its 24 columns
  out 0        0.1500  -0.0500   0.5300   0.2000   0.6500   0.1900   0.9400
  out 1        0.1100  -0.0800   0.5100   0.1900   0.6500   0.2000   0.9900
  out 2       -0.7100  -0.7600  -0.0300  -0.2100   0.3900   0.0800   0.5200
  out 3       -0.1000  -0.0100  -0.8300  -0.8700  -0.1300  -0.3000   0.1800
  out 4        0.2500   0.4800  -0.2000  -0.1000  -0.9100  -0.9400  -0.0300
  out 5        0.3400   0.7100   0.1700   0.4100  -0.2600  -0.1500  -1.4100
  (the last column is the constant, not a weight)

The top block is the paper working, printed: 0.2000 and −0.5300 in the middle, 0.1200 out, and the same 0.1200 from 2.5000, 6.4000 and a constant of 0.1500. The bottom block is the same argument at full size. 378 weights in two rows produce 1.1100, 1.2100, 0.8550, 0.8900, −0.3750, −1.7050; 150 weights in one row produce the same six numbers, and the largest disagreement anywhere among them is 7.22e-16.

That last figure is a fact about float64 and not a softening of the argument. Two arithmetically identical routes to the same answer round differently at the last bit, because the two-row route sums 24 products and then 12, while the one-row route sums 24 products once. 7.22e-16 is about three ulps at this magnitude, which is the noise floor of doing the same sum in a different order, and it is the reason a replayed simulation has to fix the order of its arithmetic and not only its inputs.