Training Time Calculator
Estimate total model training duration from dataset size, epochs, model size, and GPU throughput.
Inputs
Effective TFLOPs per GPU, e.g. ~312 for an A100 at BF16.
Realistic sustained utilization, typically 30-50% for large training runs.
Estimated Training Time
0.00days
Estimated Training Time
0.0hours
Total Training FLOPs
1.260 × 10¹⁷
Step by step
Total training FLOPs: samples × epochs × 6 × parameters
1,000,000 × 3 × 6 × 7,000,000,000
= 1.260e+17 FLOPs
Effective cluster throughput: GPUs × TFLOPs × 1e12 × utilization
8 × 312 × 1e12 × 0.40
= 9.984e+14 FLOPs/sec
Training time: total FLOPs ÷ effective throughput
1.260e+17 ÷ 9.984e+14
= 0.0 hours
How it works
Training time can be estimated from the total compute (FLOPs) required divided by the effective throughput of your GPU cluster: time = (dataset_samples × epochs × 6 × model_params) / (gpu_count × gpu_tflops × 1e12 × utilization). The factor of 6 approximates the combined multiply-add FLOPs of a forward pass (≈2×params) and backward pass (≈4×params) per training sample. Utilization accounts for the fact that real training rarely reaches 100% of a GPU's theoretical peak throughput due to communication overhead, data loading, and memory-bound operations.
Formula
time_hours = (samples * epochs * 6 * params) / (gpu_count * gpu_tflops * 1e12 * utilization * 3600)
- samples
- Total dataset samples
- epochs
- Number of training epochs
- params
- Model parameter count
- gpu_count
- Number of GPUs
- gpu_tflops
- Peak TFLOPs per GPU
- utilization
- Sustained utilization fraction (typically 0.3-0.5)
Frequently Asked Questions
Where does the factor of 6 come from?
A forward pass costs roughly 2 FLOPs per parameter per token (one multiply, one add), and the backward pass costs about twice that (roughly 4 FLOPs per parameter), for a combined ~6 FLOPs per parameter per training sample.
What utilization percentage should I use?
Large multi-GPU training runs commonly achieve 30-50% of theoretical peak FLOPs due to communication, memory bandwidth limits, and pipeline bubbles. Well-optimized single-GPU runs can reach higher, 50-70%.
What TFLOPs value should I use for my GPU?
Use the GPU's advertised peak FLOPs at the precision you're training in (e.g. BF16/FP16), such as ~312 TFLOPs for an A100 or ~989 TFLOPs for an H100 at BF16 with sparsity disabled.
Does this account for data loading or checkpointing overhead?
No, this is a pure compute-bound estimate. Real wall-clock time is often longer due to I/O, checkpointing, and evaluation passes.