Estimate total model training time from dataset size, batch size, and hardware speed.
Training time can be estimated from the total compute (FLOPs) required divided by the effective throughput of your GPU cluster: time = (dataset_samples × epochs × 6 × model_params) / (gpu_count × gpu_tflops × 1e12 × utilization). The factor of 6 approximates the combined multiply-add FLOPs of a forward pass (≈2×params) and backward pass (≈4×params) per training sample. Utilization accounts for the fact that real training rarely reaches 100% of a GPU's theoretical peak throughput due to communication overhead, data loading, and memory-bound operations.
time_hours = (samples * epochs * 6 * params) / (gpu_count * gpu_tflops * 1e12 * utilization * 3600)
A forward pass costs roughly 2 FLOPs per parameter per token (one multiply, one add), and the backward pass costs about twice that (roughly 4 FLOPs per parameter), for a combined ~6 FLOPs per parameter per training sample.
Large multi-GPU training runs commonly achieve 30-50% of theoretical peak FLOPs due to communication, memory bandwidth limits, and pipeline bubbles. Well-optimized single-GPU runs can reach higher, 50-70%.
Use the GPU's advertised peak FLOPs at the precision you're training in (e.g. BF16/FP16), such as ~312 TFLOPs for an A100 or ~989 TFLOPs for an H100 at BF16 with sparsity disabled.
No, this is a pure compute-bound estimate. Real wall-clock time is often longer due to I/O, checkpointing, and evaluation passes.