RLHF (Reinforcement Learning from Human Feedback)
Also called: RLHF
Training a model on human preference judgements so it becomes helpful and well-behaved rather than merely fluent.
RLHF collects human rankings of model outputs, trains a reward model to predict those preferences, then optimises the language model against it. This post-training step is most of the difference between a raw next-token predictor and a usable assistant. It also encodes the preferences of whoever did the ranking, which is a real and underdiscussed limitation.
In practice: Two answers, a human picks the better one, repeat a few hundred thousand times.