Apply the ReLU activation function to an input value or vector.
ReLU is the most widely used activation function in modern deep learning: ReLU(x) = max(0, x), passing positive inputs through unchanged and zeroing out negative inputs entirely. Its derivative is simply 1 for x > 0 and 0 for x < 0 (undefined exactly at x=0, conventionally taken as 0 or 1), making it extremely cheap to compute and avoiding the vanishing gradient problem for positive inputs. The tradeoff is the 'dying ReLU' problem: once a neuron's input becomes negative, its gradient is exactly 0, so it can get permanently 'stuck' and never update again — this motivated variants like Leaky ReLU and ELU.
ReLU(x) = max(0, x)
If a neuron's weighted input becomes negative and stays negative across training (e.g. due to a large negative bias update), ReLU outputs exactly 0 and its gradient is also exactly 0, meaning that neuron receives no further weight updates and effectively stops contributing to the network permanently.
It requires only a single comparison against zero (max(0, x)), with no exponentials, divisions, or trigonometric functions — this simplicity is a major reason it became the default choice for hidden layers in large-scale deep learning, where activation functions are computed billions of times.
Mathematically the derivative is undefined at exactly x=0 since the function has a sharp corner there, but in practice frameworks conventionally define it as either 0 or 1 at that single point — the choice has negligible practical impact since inputs are rarely exactly 0.