Learning Rate
How big a step training takes each update — the hyperparameter most likely to ruin a run.
Too high a learning rate and training overshoots and diverges; too low and it crawls or gets stuck. Most real runs schedule it, starting larger and decaying over time. It is the classic example of a hyperparameter: set by you, not learned by the model.
In practice: Same data, same architecture, learning rate 10× too high — the loss goes to infinity.