Self-Supervised Learning
Training where the data generates its own labels — the trick that made large language models possible.
Self-supervised learning invents a prediction task out of unlabelled data, most famously ‘predict the next token’. Because the label is just the next word, the entire internet becomes training data with no human annotation. This is why LLMs could scale in a way supervised models never could.
In practice: Hide the last word of a sentence, ask the model to guess it, repeat trillions of times.