A decoder-only language model in PyTorch whose residual can forget. Each block writes x + beta * k * (v - <k, x>), so a layer overwrites one direction instead of only adding to it: for a unit k, the linear part has eigenvalue 1 - beta along k and 1 everywhere orthogonal to it. Everything around the update is conventional on purpose, so the update is the only variable.
Read the update as an instruction. The block picks a direction k, measures how much of the current state already lies along it, and writes the difference between that and a target v, scaled by a learned gate beta. At beta one the old component along k is gone; at zero nothing changes. One direction is rewritten and the rest of the representation is untouched — the thing an additive residual structurally cannot do.
The surrounding model is deliberately standard: Hugging Face model and config integration, causal scaled dot-product attention, rotary embeddings, QK RMSNorm, SwiGLU, a tied embedding and output head, bounded per-token beta gates with scalar write targets, and an optional causal one-token key-value shift.
The default path uses a fused multiply-add instead of materialising the scaled direction as an intermediate tensor, and a reference implementation of the same update stays in the tree with output and gradient parity under test. A benchmark script measures both paths and, on CUDA, their peak memory. The acceptance bar: equal outputs, equal gradients, no increase in peak allocation.
The constraints are enforced, not documented. Hidden size must equal heads times head dimension. An input longer than the block size fails explicitly instead of being truncated into a wrong answer. Diagnostics — per-layer beta range and mean, gain along the written axis, projection error before and after the write, update norm — are opt-in, because extracting scalars forces an accelerator sync.