TFLOPs Calculator
Calculate the teraFLOPs throughput required or delivered for a training workload.
About TFLOPs Calculator
Achieved TFLOPS measures actual sustained compute throughput: TFLOPS = operations / (time_seconds × 1e12). Comparing this to hardware's advertised theoretical peak gives Model FLOPs Utilization (MFU) = achieved / theoretical × 100 — a key efficiency metric in large-scale training. Well-optimized large language model training runs typically achieve 30-55% MFU; the gap from 100% comes from memory bandwidth limits, kernel launch overhead, communication between GPUs, and imperfect operator fusion, all of which prevent hardware from sustaining its theoretical peak FLOPs in real workloads.
Formulas
Achieved TFLOPS
achieved_tflops = operations / (time_seconds × 1e12)
- operations
- Total floating-point operations performed
- time_seconds
- Wall-clock time taken
Model FLOPs Utilization
MFU = (achieved_tflops / (hardware_peak_tflops × gpu_count)) × 100
- achieved_tflops
- Measured throughput in TFLOPS
- hardware_peak_tflops
- Advertised peak TFLOPS for the GPU
- gpu_count
- Number of GPUs
Worked example
With the default values below, TFLOPs Calculator returns Achieved TFLOPS of 416.67. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Total Operations (FLOPs)accepts 1–1,000,000,000,000,000,000,000,000,000,000
- 500,000,000,000,000
- Time Takenaccepts 0–1,000,000 seconds
- 1.2 seconds
- Hardware (for MFU comparison)one of NVIDIA A100 (312 TFLOPs), NVIDIA H100 (989 TFLOPs), NVIDIA RTX 4090 (165 TFLOPs)
- NVIDIA A100 (312 TFLOPs)
- GPU Countaccepts 1–100,000
- 1
Results
- Achieved TFLOPS
- 416.67 TFLOPS
- Model FLOPs Utilization (MFU)
- 133.55%
- Theoretical Peak TFLOPS
- 312.00 TFLOPS
Step by step
- 1Achieved TFLOPS: operations ÷ (time × 1e12)= 416.67 TFLOPS
- 2Theoretical peak: hardware TFLOPs × GPU count= 312.00 TFLOPS
- 3MFU: achieved ÷ theoretical peak × 100= 133.55%
Assumptions and limits
- The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What is a good MFU percentage for LLM training?
Well-tuned large-scale training runs (e.g. GPT-3/PaLM-scale models on modern GPU clusters) typically report MFU in the 30-55% range; achieving above 50% is considered excellent, since perfect 100% utilization is essentially unreachable due to memory bandwidth and communication overhead.
Why can't real training runs reach 100% MFU?
The theoretical peak TFLOPs figure assumes every compute unit is perfectly fed with data every cycle, but real training involves memory bandwidth limits, inter-GPU communication for gradient synchronization, kernel launch overhead, and non-matmul operations (normalization, activation functions) that don't run at peak matmul throughput.
Why does hardware selection matter for the MFU calculation?
MFU is only meaningful relative to a specific hardware's theoretical peak — the same achieved TFLOPS represents a much higher MFU on slower hardware (e.g. an RTX 4090) than on faster hardware (e.g. an H100), since the denominator (theoretical peak) differs substantially.