Building an LLM from Scratch: What 9 Million Parameters Reveal

In this instalment, Jeremy Andrews, Founding Partner and CEO of Tag1, documents how early configurations from tiny presets to GPT-2–sized variants failed to behave as expected. Wider models did not reliably outperform deeper ones, and validation metrics alone proved insufficient for judging quality.

Controlled experiments revealed several structural lessons. Multi-head attention consistently outperformed single high-resolution heads. Longer context windows stabilised optimisation and improved coherence. Most critically, increasing parameter count without increasing training data led to memorisation rather than generalisation—an accidental demonstration of modern scaling laws.

The resulting 9 million parameter configuration, dubbed “Pocket Bard,” learned to reproduce Shakespearean rhythm, stage formatting, and character conventions. Yet it also generated plausible but meaningless words, underscoring a key boundary: statistical learning can reproduce style without achieving semantic understanding.

A further experiment with transfer learning strengthened that conclusion. By pre-training on TinyStories before fine-tuning on Shakespeare, the same architecture gained grammatical stability and structural coherence far more quickly. Foundational language competence proved more valuable than raw size.

The full guide walks through vocabulary experiments, architectural trade-offs, debugging insights, and implementation details—including a complete Rust-based transformer built without external ML frameworks. For readers interested in how scaling behaviour actually manifests at small model sizes, the original tutorial provides the detailed logs, code references, and experimental breakdown.