Foundations intermediate

Hyperparameter

A setting you choose before training, as opposed to a parameter the model learns during it.

Learning rate, batch size, number of layers, dropout rate — these are hyperparameters. They are picked by the developer and tuned against validation data. The distinction from parameters is simple: you set hyperparameters, training sets parameters.

In practice: Running the same job 20 times with different learning rates to find the one that converges best.