RLHF (Reinforcement Learning from Human Feedback)
RLHF collects human rankings of model outputs, trains a reward model to predict those preferences, then optimises the language model against it. This post-training step is most of the difference between a raw next-token predictor and a usable assistant. It also encodes the preferences of whoever did the ranking, which is a real and underdiscussed limitation. In practice: Two answers, a human picks the better one, repeat a few hundred thousand times. ...