Before transformers
Until 2017, NLP used RNNs and LSTMs — sequential models that processed text one token at a time, preventing efficient parallelism and making long-range dependencies difficult to capture.
Self-attention
Each token is projected into three vectors: Query, Key, Value. Attention scores: softmax(QK-transpose divided by square root of d_k) multiplied by V. The result is a weighted sum capturing relevant context from across the full sequence.
Multi-head attention
The Transformer runs several attention heads in parallel. Each head learns different relationship types: syntax, semantics, position. Outputs are concatenated and projected.
Encoder-decoder
Original Transformers had an encoder processing input and a decoder generating output. GPT-family models use decoder-only. BERT uses encoder-only.
Why transformers scale
Every token attends to every other simultaneously, not sequentially. Training is fully parallelisable across GPU clusters. This enabled scaling from billions to hundreds of billions of parameters, producing qualitatively new capabilities.
