Mixture of Experts
Also called: MoE
An architecture where only a fraction of the model's parameters activate for any given token.
A mixture-of-experts model contains many specialised sub-networks and a router that picks a few per token. Total parameter count stays huge while compute per token stays modest, which decouples capacity from inference cost. It is why some very large models are surprisingly cheap to serve.
In practice: A 400B-parameter model that only runs 40B worth of compute per token.