Continual learning: A model keeps learning from new experience after deployment, without retraining from scratch and without losing what it already knew.
Catastrophic forgetting: Training on new data degrades what the model knew before, because the same weights get overwritten.
Finetuning, SFT: More gradient descent after pretraining. Supervised finetuning (SFT) trains on fixed input → gold-answer pairs.
Recitation vs. derived questions: Recitation repeats the taught fact. Derived questions need the fact to be used: applied, composed with other facts, or reasoned about counterfactually.
Augmentation: Showing the same fact in many phrasings and contexts instead of one fixed sentence (paraphrases, rewrites, Q&A about it).
i.i.d.: Independent and identically distributed. In pretraining, every batch is a random sample of the whole mixture, so no topic arrives all at once.
In-context learning (ICL): Using information placed in the prompt at inference time, with no weight update.
KV cache: The stored attention keys and values for every earlier token. Its memory cost grows linearly with context length.
Compaction: Compressing a long context (summaries, notes) so an agent can keep working past its context window. Can be trained with RL.
Cartridges: A small KV cache trained offline, on synthetic conversations about a document, to stand in for that document in the prompt.
Mixture of experts (MoE): A layer with many feed-forward “experts” and a router that sends each token to a few of them.
Policy: The model seen as a distribution over its outputs. In RL, the thing being trained.
On-policy: Training on the model’s own samples rather than on fixed external data.
Reward: A score for a sampled output. RL makes high-reward samples more likely.
KL divergence: How far one probability distribution is from another. Here: how far the trained model’s outputs have drifted from the base model’s.
Distillation: Training a student model to match a teacher model’s output distribution, token by token.
On-policy distillation: The student samples its own answers and a stronger teacher scores every token of them.
Self-distillation (SDFT, OPSD, SDPO): The teacher is the same model, given a hint in its context: a demonstration (SDFT), a worked solution (OPSD), or the error from its own failed attempt (SDPO).
Self-edit (SEAL): Training data the model writes for itself about a passage, which it is then finetuned on.
Weight-space programming: A model reads another model’s raw weights and edits them directly, without gradients.
Context Language Model (CLM): A model trained to edit its own working context like a file, instead of only appending to it.
Memory layer, sparse memory finetuning: A layer holding a huge table of key–value “slots”, of which each token reads only a few. Sparse memory finetuning updates only the slots most specific to the new data.
Engram: A large embedding table indexed by hashing the last few tokens (an n-gram), mixed into the hidden state through a learned gate.
Test-time training (TTT): Updating some weights during inference, on the input itself (e.g. Titans, TTT-E2E).
Linear attention, hybrids: Attention variants whose cost grows linearly with context length, often interleaved with standard attention layers.
Training horizon: How far ahead a training run can reward a behaviour. Payoffs that arrive later than that never shape what the model learns to keep.
Harness: Everything around the model in an agent: tools, memory files, context management.
Rate–distortion: Information theory’s trade-off between how much you keep (rate) and how much error that causes later (distortion).

My thoughts draw heavily on conversations with, and posts from, Karthik Balaji, and on posts from Will Brown, Charlie O’Neill, Beren Millidge, Jessy Lin, Ilija Lichkovski and Andrew Lampinen.