Work out batch size selection instantly with clear inputs, formula shown and shareable results.
GPU memory splits into a fixed part — weights, gradients and optimiser state — and a part that scales with batch size, which is dominated by stored activations for the backward pass. The largest micro-batch is therefore the free memory divided by activation memory per sample. Gradient accumulation then reaches any target effective batch by summing gradients over several micro-batches before stepping the optimiser.
Batch and accumulation
free = total - fixed model memory; micro-batch = floor(free / activation per sample); accumulation steps = ceil(target / micro-batch)
Almost. The gradient is mathematically the same, but batch-normalisation statistics are computed per micro-batch, so networks using BatchNorm behave slightly differently.
Gradient checkpointing recomputes activations in the backward pass, cutting activation memory by roughly the square root of the layer count at about 30 percent extra compute.