Activation Function
The non-linear step applied at each unit, without which a deep network would collapse into a single linear one.
An activation function decides what a unit passes forward. The non-linearity is the point: stack a hundred linear layers and you still have a linear model, but insert a non-linear function between them and the network can represent curves, interactions, and structure. ReLU is the common default.
In practice: ReLU: negative input becomes zero, positive input passes through unchanged.