Dropout Probability Calculator
Calculate the expected number of active neurons given a dropout probability.
Inputs
Expected Active Neurons
819.2
Expected Dropped Neurons
204.8
Retained Fraction
80.0%
Inverted-Dropout Scale Factor
1.2500
Guidance
Typical hidden layer size — dropout of 20–50% is common for fully connected layers.
Step by step
Retained fraction: 1 − p
1 − 0.20
= 0.800
Active neurons (expected): total × (1 − p)
1,024 × 0.80
= 819.2
Inverted-dropout inference scale: 1 ÷ (1 − p)
1 ÷ 0.80
= 1.2500
How it works
Dropout randomly zeroes each neuron's activation with probability p during training, so the expected number of active neurons in a layer of size N is N × (1 − p). At inference time, standard ('inverted') dropout implementations instead keep all neurons active and scale outputs by 1 / (1 − p) during training, so the effective magnitude of activations matches what the network will see at inference (or apply no scaling at inference and scale by (1-p) then instead — frameworks like PyTorch and TensorFlow use inverted dropout by default, applying the 1/(1−p) scaling during training only).
Formulas
Expected active neurons
active_neurons = total_neurons × (1 - p)
- total_neurons
- Total neurons in the layer
- p
- Dropout probability
Inverted-dropout scale factor
scale = 1 / (1 - p)
- p
- Dropout probability
Frequently Asked Questions
What's a typical dropout rate?
0.5 (50%) is the classic default for fully connected layers from the original dropout paper, though modern architectures often use lower rates (0.1–0.3) for convolutional layers and even lower or no dropout in transformer feed-forward blocks when other regularization is present.
Why does inference use a scale factor?
During training, only (1−p) fraction of neurons contribute to each forward pass, so their outputs are scaled up by 1/(1−p) to keep the expected activation magnitude constant; at inference all neurons are active without dropout, so no scaling is needed if this 'inverted dropout' convention was used during training.
Does dropout apply per-neuron or per-weight?
Standard dropout zeroes entire neuron activations (not individual weights), which is why it's described in terms of 'active neurons' — variants like DropConnect instead drop individual weight connections.
Should dropout rate differ by layer depth?
It's common practice to use lower dropout near the input (or none at all) and higher dropout in deeper, wider layers with more redundant capacity, since early layers often extract lower-level features that are more sensitive to being randomly zeroed.