× N

skip

skip

THE MATH, BY HAND

The same numbers as the Pipeline, expanded to every scalar operation.

Pick a query, key, token and activation below; every derivation recomputes live from your input. These are the exact values the matrices in the other tabs are built from: nothing here is collapsed into a matmul.

1 One attention score · scalar by scalar

Query "{{ mathQword }}" (q{{ mQi }}) scored against key "{{ mathKword }}" (k{{ mKj }}) in head {{ mHeadi }}: a dot product over the head's 2 dimensions, scaled by √dₖ.

2 Softmax over query {{ mQi }}'s scores

Exponentiate each visible score (max subtracted for numerical stability) and normalize so the row sums to 1. The result is the attention-weight row.

k{{ r.j }}·{{ r.word }} {{ r.s }} {{ r.ez }} {{ r.w }}

{{ softInfo }}

3 Head output = weighted sum of values

Each output dimension is the attention-weighted average of that dimension across all visible value vectors: out[i] = Σⱼ wᵢⱼ · vⱼ.

4 LayerNorm on token "{{ mathTokWord }}"

Standardize the residual vector: subtract the row mean, divide by the row standard deviation. (γ=1, β=0 here, so no rescale.)

5 GELU on FFN unit {{ mFFi }} of "{{ mathTokWord }}"

The feed-forward non-linearity: a smooth gate that lets small negatives leak through and passes positives almost unchanged.

6 Temperature reshapes the final softmax

The same eight next-token logits z, divided by T before the exponent: pⱼ = e^(zⱼ/T) / Σₖ e^(zₖ/T). Small T stretches the gaps between logits apart (sharper), large T shrinks them (flatter): the ranking never changes, only the confidence. This is the knob on the Generate tab.

{{ tempInfo }}

x⁰

read

⊕

read

⊕

↻ real models stack N identical blocks · the stream flows straight through, each block reading & writing it

out

AUTOREGRESSIVE GENERATION

Predict one token, append it, feed the whole thing back in. repeat.

A single forward pass predicts one token. Generation is the outer loop: sample from that distribution, stick the token on the end of the sequence, and run the entire pass again on the longer input. Everything you saw in the other tabs runs once per generated token.

sequence→ forward pass→ sample next→ append↺

TEMP {{ genTempVal }}

{{ genHint }}

Context cap reached (this toy runs to 14 tokens). Reset to start over.

ROLL-OUT TRACE

COMPUTE · WHAT GENERATION COSTS

You've now seen both phases of inference. Prefill: all prompt positions computed in parallel, one pass, like the all-position pass on the Training tab. Decode: this loop, strictly one token at a time, because each token's input includes the last output. This toy recomputes every position on every step; real models keep a KV-cache (the K and V rows you saw in attention, stored per layer) so each new token only computes its own row.

{{ c.k }}

{{ c.v }}

{{ c.sub }}

Decode speed is memory-bound, not FLOP-bound: every single token streams all N weights through the chip once, so bandwidth sets the single-stream ceiling and the arithmetic units mostly idle. Serving stacks many users into one batch to share each weight-read: that's why inference is priced per token.

reference numbers for an 8-B-param model in fp16 on one H100-class GPU (~3.4 TB/s memory) · this toy: ~2 kFLOP per token, no cache · your browser tab is a supercomputer for it

TRAINING PIPELINE

Everything you've seen is inference. Training is the loop that sets the weights.

The forward pass in the other tabs runs a fixed set of weights. Training is the outer loop that produced them: run the forward pass, measure how wrong the prediction was, push every weight a little in the direction that reduces the error, and repeat. Crucially, the operations never change. Attention, LayerNorm, the FFN shapes are all fixed. Training only nudges the ≈188 numbers inside them.

The training signal, on your actual sentence. At each position the model predicts the next token; we compare that to the real next token and score it with cross-entropy loss = −ln p(correct). This is exactly what backprop would minimize.

{{ r.mark }}

{{ r.ctx }} _

{{ r.pred }} {{ r.predP }}

{{ r.target }}

{{ r.loss }}

THE TRAINING DATA

A dataset is just text: no labels, no answers. Teacher forcing slices every sentence into one example per position: everything so far → the actual next word. Your 5-token sentence is the entire corpus here, so it yields exactly these {{ tdCount }} examples:

{{ p.n }}

{{ c }}

→ target {{ p.target }}

model now: p = {{ p.p }} {{ p.mark }}

real models do exactly this, but over web text, books and code: ~15 trillion tokens ≈ 15 trillion (context → next word) examples, batched millions at a time. same loss, same update rule; only the scale changes.

Run W ← W − η·∂L/∂W for real, on all 172 trainable weights, against the loss above. Every tab reads the same weights: watch the prediction chip in the top bar, the ✓ column, and the attention heatmaps reorganize as the loss falls.

training…

EXERCISE · TOUCH ONE WEIGHT

Training is nothing more than search over these numbers. Drag one and watch the loss respond: the curve is the loss landscape along this single weight's axis. Gradient descent just rolls downhill on it, in 172 dimensions at once.

{{ wMapDesc }}

this is the whole model. there is nothing else for training to touch. the 16 LayerNorm params (γ, β) are fixed at 1, 0 in this toy. GPT-scale models are the same picture with bigger rectangles: more dims per matrix, more heads, N stacked blocks.

COMPUTE · WHAT TRAINING COSTS

THE STEP BUTTON ABOVE 172 weights · 345 forward passes per step (2 per weight, finite differences) · ≈0.7 MFLOP · ~7 ms in this tab

Real training never brute-forces the gradient like that button does. Backpropagation reuses the forward pass's intermediate values to get all N gradients for roughly the cost of 2 extra forward passes, so one step costs ~3 forwards' compute regardless of N. At 8 B parameters, finite differences would need 16 billion forward passes per step; backprop needs the equivalent of ~3. That algorithm is the only reason any of the rows below are possible. The totals follow one rule of thumb: FLOPs ≈ 6 · N params · D tokens.

{{ c.k }}

{{ c.v }}

{{ c.sub }}

{{ scaleNote }}

wall-clock = 6·N·D ÷ (GPUs × ~0.4 PFLOP/s sustained per H100-class GPU) · real runs add data pipelines, checkpoint restarts and node failures · architecture is unchanged across every row: same attention, same FFN, same loss as this page

TRAINING: THIS TOY vs PRODUCTION-GRADE

The objective is the one thing that stays the same. everything about scale, data and machinery is where production lives.

{{ r.label }}

{{ r.toy }}

{{ r.real }}

GARAGELM.ORG · ABOUT THIS PROJECT

A working transformer forward pass, small enough to read every number.

This is a complete decoder-only transformer, the same architecture behind GPT-class models, shrunk to a size where nothing is hidden. Four dimensions, two heads, one block, an eight-word vocabulary. Type in five tokens and watch text turn into a next-token probability distribution, one matrix multiply at a time. Nothing is pre-baked: every value on screen is computed live in your browser from the input you choose.

THE COMPLEXITY TRUTH

What's real here is the shape of the computation: the operations, the data flow, the causal masking, the way dimensions transform stage to stage. What's not real is everything about scale and training. The numbers are illustrative, the weights are hand-chosen rather than learned, and a single four-dimensional block obviously cannot model language. Read this as a wiring diagram you can run, not a small language model.

{{ r.label }}

{{ r.toy }}

{{ r.real }}

DELIBERATELY LEFT OUT

These live in every real model but would only obscure the mechanism at this scale. Their absence is the edge between this toy and a production system:

{{ o }}

Same architecture, honest about its scale · computed live · no data leaves your browser