LLMs Are Next-Token Predictors
A large language model is, at its core, a next-token predictor: given the tokens so far, predict the next one. Every impressive capability — writing, coding, reasoning — emerges from this loop repeated thousands of times. The “large” refers to billions of parameters learned from enormous amounts of text.
Tokens and Tokenizers
Models don’t read characters or whole words. They read tokens — sub-word units from a fixed vocabulary (typically 32k–200k entries). Byte-pair encoding merges the most frequent character pairs until the vocabulary is the right size. “unbelievable” might be one token; a rare company name splits into two or three. The tokenizer visualizer on this page traces a sentence from text to token ids to embeddings.
Embeddings: Meaning as Vectors
Each token id maps to a dense embedding vector. Words used in similar contexts end up near each other in vector space — which is why vector math like “king − man + woman ≈ queen” works. The model never sees words; it processes these vectors through its layers.
Self-Attention in One Paragraph
For each token, self-attention computes how much to attend to every other token in the context. The word “it” in “the car rolled down the hill because it was slippery” must be linked to “car” through attention weights. This is what gives transformers their power over older recurrent models: every token can directly look at every other token, with no memory bottleneck. The attention patterns are learned during training.
The Context Window
The context window bounds how many tokens the model can attend to at once. Everything beyond it is invisible. Consequences you’ll hit in practice:
| Problem | Why it happens | Workaround |
|---|---|---|
| Long docs get cut off | Exceeds the window | Chunk + retrieve (RAG) |
| Early conversation forgotten | History overflows the window | Summarize or truncate |
| Answers seem out of date | Knowledge is baked in at training time | Provide fresh context in the prompt |
Autoregressive Generation and Sampling
At each step the model outputs a probability distribution over the whole vocabulary. Sampling picks the next token:
- Greedy — always the most likely token (dull, repetitive).
- Temperature — scales the distribution; low = conservative, high = creative.
- Top-p — restricts sampling to the smallest set whose cumulative probability exceeds p.
Generation is therefore non-deterministic. Two identical prompts can produce different answers — a core reason evaluation and caching matter in production.
When It’s the Right Tool
Reach for an LLM when the task is open-ended language: summarization, drafting, code generation, conversational interfaces. For deterministic, latency-critical or low-cost decisions, a rule, a classifier, or a smaller model is usually the better tool — LLMs are slow and expensive per token.
Practice Trajectory
- Tokenize a sentence by hand and compare with a real tokenizer’s output.
- Explain in one sentence why “it” needs attention to resolve correctly.
- Give a scenario where a 4k context window silently breaks an answer.
- Predict what happens to output quality if temperature is set to 0.0.
- Decide when you would use a rule instead of an LLM for a task you know.