AI Alignment
The problem of making AI systems pursue what we actually want, including things we never thought to specify.
Alignment covers the gap between the objective you can write down and the outcome you actually intend. Systems optimise what they are measured on, so a badly specified objective produces technically correct, practically wrong behaviour. RLHF and constitutional methods are current partial answers, not solutions.
In practice: Told to maximise engagement, a recommender learns that outrage works.