garagelm600× fewer tokens: tying Pythia-160M on one Mac
1. the gates, stated before the results
Nothing here was decided after seeing the numbers. Before launch, the yardstick was written down: no 0.5B-token local run beats SmolLM2-135M (2T tokens), so the honest targets were (a) beat Cerebras-GPT-111M everywhere (if refined data can't beat unfiltered Pile data at 4× our tokens, the data thesis fails), and (b) approach Pythia-160M and OPT-125M on HellaSwag and PIQA at 360–600× fewer tokens. Reference models are re-run through the identical harness and example slices rather than trusting published full-set numbers.
2. the recipe
The architecture came out of milestones 03–05: a 232M decoder with RoPE, RMSNorm, SwiGLU, grouped-query attention, and the hybrid pattern: a 64-token sliding window in most layers, full attention every 4th (layers 3/7/11/15). That keeps the KV cache at ~30% of a full-attention twin while beating it on val loss (replicated 3/3 seeds in milestone 08). Data is the milestone 05 refined pool. Training ran resumable, state saved every 500 steps; run 09 survived two external process kills with zero lost steps.
3. results, identical harness
Val loss 3.1246 (PPL 22.7) at 0.5B tokens; 3.0700 (PPL 21.5) at 1.0B. Head-to-head, acc/acc_norm, all models run through the same local pipeline:
| model | tokens | hellaswag | piqa | arc-e | winogrande |
|---|---|---|---|---|---|
| hybrid-232m @ 1.0B | 1.0B | .357/.397 | .633/.610 | .477/.450 | .517 |
| hybrid-232m @ 0.5B | 0.5B | .353/.393 | .617/.593 | .447/.433 | .533 |
| gpt2 (124M) | ~10B | .353/.427 | .610/.617 | .420/.380 | .530 |
| pythia-160m | 300B | .350/.390 | .627/.627 | .460/.397 | .510 |
| smollm2-135m | 2,000B | .403/.553 | .653/.680 | .583/.437 | .490 |
MMLU is at chance (.267) for every model at this scale, SmolLM2 included, and is omitted. Local re-run numbers are deliberately not comparable to published full-set numbers.
Reading the table: against Pythia-160M (600× the tokens at 0.5B; 300× at 1.0B) the result is a statistical tie at 0.5B and ahead-or-tied everywhere at 1.0B, clearly ahead on ARC-Easy. Against gpt2, one clear win (ARC-E, the only gap beyond 2×SE), PIQA leaning our way inside the noise, and two ties. Against Cerebras-GPT-111M's published numbers, ahead across the board: gate (a) met. Against SmolLM2, clearly behind, exactly as pre-registered: that gap is 2T refined tokens, not architecture.
4. long-range behavior
The risk with windowed attention is going blind past the window edge. The per-position probe says otherwise at 232M: val loss improves from 3.67 over positions 0–63 to ~3.12 by position 256 and holds through position 1000. The four global layers keep earning deep context; the cheap local layers handle the rest.
5. where the lever saturates
Run 09's clearest contribution is negative-shaped: doubling tokens from 0.5B to 1.0B bought 0.055 nats of val loss, +3.0 ARC-E, +1.6 PIQA, and ~0 HellaSwag. Refined data bought a 600× token-efficiency tie at 0.5B; the second 0.5B bought low-single-digit points. The steep part of the curve is behind. Next capability jumps need a different axis: bigger model, longer context, distillation, or post-training. That is what milestone 11 (intelligence per training hour) took on next.
6. deployment
The MLX port passes logit parity and decodes at ~310 tok/s in fp16 with 72ms time-to-first-token on 896-token prompts; the KV cache is 4.97MB at full context, ~30% of a full-attention equivalent. 4-bit quantization costs +0.016 nats and runs ~530–550 tok/s. (The published weights are the 1.0B, 300× run.) Samples read as fluent encyclopedic prose with 232M-grade factual reliability, confidently wrong astronomy included.