Batch Size Calculator
Estimate the maximum training batch size that fits in available GPU VRAM.
Inputs
e.g. Adam stores two extra moment buffers per parameter, roughly 2x model size in FP32.
Activation memory consumed by a single training sample.
Max Batch Size
240samples
Recommended Batch Size (Power of 2)
128samples
Free Memory for Activations
12.00GB
Step by step
Free memory for activations: available VRAM − model size − optimizer states
24 − 4 − 8
= 12.00 GB
Max batch size: free memory ÷ per-sample memory
12.00 GB ÷ 0.0500 GB
= 240
Recommended batch size: largest power of 2 ≤ max batch size
2^⌊log2(240)⌋
= 128
How it works
The maximum training batch size that fits in GPU memory is: max_batch = (available_vram − model_size − optimizer_states) / per_sample_memory. Model weights and optimizer states (like Adam's momentum and variance buffers) occupy a fixed amount of VRAM regardless of batch size, so subtracting them first leaves the memory actually available for activations, which scale linearly with batch size. Batch sizes are conventionally rounded down to a power of 2 for better GPU kernel efficiency and predictable memory alignment.
Formula
max_batch_size = floor((available_vram - model_size - optimizer_states) / per_sample_memory)
- available_vram
- Total available GPU VRAM in GB
- model_size
- Memory occupied by model weights in GB
- optimizer_states
- Memory occupied by optimizer buffers in GB
- per_sample_memory
- Activation memory per sample in GB
Frequently Asked Questions
Why round to a power of 2?
GPU kernels and memory allocators are often optimized for power-of-2 sizes, which can improve throughput and reduce memory fragmentation compared to arbitrary batch sizes.
What if optimizer states take more memory than expected?
Adam-style optimizers store two additional buffers (first and second moment estimates) per parameter, typically 2x the model size in the same precision, plus a FP32 master copy if using mixed precision — this can add up to 3-4x model size total.
How do I measure per-sample memory accurately?
The most reliable way is to run a single training step with a small batch size and observe actual GPU memory usage via nvidia-smi or a profiler, then divide the activation memory by the batch size used.
Can I exceed this maximum with gradient accumulation?
Gradient accumulation lets you simulate a larger effective batch size by accumulating gradients over multiple smaller forward/backward passes before updating weights, without needing the full batch in memory at once.