What LLMs do
An LLM predicts the next token. Given a sequence of text, the model assigns probabilities to every token in its vocabulary and samples from them. To predict the next word well, the model must encode vast implicit knowledge about the world.
The transformer architecture
Modern LLMs use the Transformer (2017). Self-attention lets each token attend to every other token, learning relevant relationships. Transformers train in parallel, enabling the scale-up that makes LLMs possible.
Pretraining and fine-tuning
Pretraining: trains on trillions of tokens via self-supervised learning. Fine-tuning: adapts using labelled examples or RLHF to produce helpful, safe, instruction-following models.
Emergence
Certain capabilities appear unpredictably as scale increases. Multi-step reasoning, code generation, and analogical thinking emerged at scale. This makes predicting future capabilities genuinely hard.
Limitations
LLMs hallucinate plausible-but-wrong facts with no uncertainty signal. They reflect training data biases, have limited context windows, and cannot update knowledge in real-time without retrieval augmentation.
