Estimate an optimal training batch size based on dataset size and memory limits.
The maximum training batch size that fits in GPU memory is: max_batch = (available_vram − model_size − optimizer_states) / per_sample_memory. Model weights and optimizer states (like Adam's momentum and variance buffers) occupy a fixed amount of VRAM regardless of batch size, so subtracting them first leaves the memory actually available for activations, which scale linearly with batch size. Batch sizes are conventionally rounded down to a power of 2 for better GPU kernel efficiency and predictable memory alignment.
max_batch_size = floor((available_vram - model_size - optimizer_states) / per_sample_memory)
GPU kernels and memory allocators are often optimized for power-of-2 sizes, which can improve throughput and reduce memory fragmentation compared to arbitrary batch sizes.
Adam-style optimizers store two additional buffers (first and second moment estimates) per parameter, typically 2x the model size in the same precision, plus a FP32 master copy if using mixed precision — this can add up to 3-4x model size total.
The most reliable way is to run a single training step with a small batch size and observe actual GPU memory usage via nvidia-smi or a profiler, then divide the activation memory by the batch size used.
Gradient accumulation lets you simulate a larger effective batch size by accumulating gradients over multiple smaller forward/backward passes before updating weights, without needing the full batch in memory at once.