Calculate the effective batch size achieved through gradient accumulation steps.
Gradient accumulation simulates training with a large batch size on hardware that can only fit a small one: effective_batch = micro_batch_size × accumulation_steps × gpu_count. Instead of updating weights after every forward/backward pass, gradients are summed (accumulated) over several micro-batches before a single optimizer step is taken, so peak GPU memory stays at the micro-batch level while the optimizer still sees the statistical benefit of a much larger effective batch. This is the standard technique for training large models on memory-constrained GPUs, or for matching a specific large-batch training recipe without needing proportionally more hardware.
effective_batch_size = micro_batch_size * accumulation_steps * gpu_count
It increases wall-clock time per optimizer step (since you run multiple forward/backward passes before each update) roughly proportionally to the number of accumulation steps, but it doesn't increase total training compute — it trades time for the ability to use a larger effective batch size on limited memory.
Numerically it's very close — gradients are summed/averaged the same way — though details like batch normalization statistics (computed per micro-batch rather than per full batch) can introduce small differences from true large-batch training.
First find the largest micro-batch size that fits in GPU memory (see the Batch Size Calculator), then divide your target effective batch size by (micro_batch × GPU count) to get the required accumulation steps.
No — peak memory is set by the largest single micro-batch forward/backward pass; accumulation lets you reach a larger effective batch without exceeding that peak, but it doesn't reduce the micro-batch memory requirement itself.