Calculate the teraFLOPs throughput required or delivered for a training workload.
Achieved TFLOPS measures actual sustained compute throughput: TFLOPS = operations / (time_seconds × 1e12). Comparing this to hardware's advertised theoretical peak gives Model FLOPs Utilization (MFU) = achieved / theoretical × 100 — a key efficiency metric in large-scale training. Well-optimized large language model training runs typically achieve 30-55% MFU; the gap from 100% comes from memory bandwidth limits, kernel launch overhead, communication between GPUs, and imperfect operator fusion, all of which prevent hardware from sustaining its theoretical peak FLOPs in real workloads.
Achieved TFLOPS
achieved_tflops = operations / (time_seconds × 1e12)
Model FLOPs Utilization
MFU = (achieved_tflops / (hardware_peak_tflops × gpu_count)) × 100
Well-tuned large-scale training runs (e.g. GPT-3/PaLM-scale models on modern GPU clusters) typically report MFU in the 30-55% range; achieving above 50% is considered excellent, since perfect 100% utilization is essentially unreachable due to memory bandwidth and communication overhead.
The theoretical peak TFLOPs figure assumes every compute unit is perfectly fed with data every cycle, but real training involves memory bandwidth limits, inter-GPU communication for gradient synchronization, kernel launch overhead, and non-matmul operations (normalization, activation functions) that don't run at peak matmul throughput.
MFU is only meaningful relative to a specific hardware's theoretical peak — the same achieved TFLOPS represents a much higher MFU on slower hardware (e.g. an RTX 4090) than on faster hardware (e.g. an H100), since the denominator (theoretical peak) differs substantially.