Mixture of Experts
A mixture-of-experts model contains many specialised sub-networks and a router that picks a few per token. Total parameter count stays huge while compute per token stays modest, which decouples capacity from inference cost. It is why some very large models are surprisingly cheap to serve. In practice: A 400B-parameter model that only runs 40B worth of compute per token.