Inference Time Calculator
Estimate model inference latency per request based on hardware throughput.
About Inference Time Calculator
This calculator estimates single-inference latency directly from compute requirements: latency = model_flops / (hardware_tflops × utilization × 1e12). Unlike a full prefill/decode breakdown, this treats inference as a single compute-bound pass — appropriate for non-autoregressive models (e.g. image classifiers, single-pass encoders) where the entire forward pass's FLOPs are known upfront. The utilization factor accounts for the gap between a GPU's theoretical peak and what real inference workloads sustain, driven by kernel efficiency, batch size, and memory bandwidth limits for smaller models.
Formula
latency_ms = (model_flops / (hardware_tflops × utilization × 1e12)) × 1000
- model_flops
- Floating-point operations for one forward pass
- hardware_tflops
- Peak TFLOPS of the hardware
- utilization
- Fraction of peak actually sustained (e.g. 0.4)
Worked example
With the default values below, Inference Time Calculator returns Estimated Inference Latency of 0.112. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Model FLOPs (per inference)accepts 1–100,000,000,000,000,000,000
- 14,000,000,000
- Hardwareone of NVIDIA A100 (312 TFLOPs), NVIDIA H100 (989 TFLOPs), NVIDIA RTX 4090 (165 TFLOPs)
- NVIDIA A100 (312 TFLOPs)
- Expected Utilizationaccepts 1–100 %
- 40 %
Results
- Estimated Inference Latency
- 0.112 ms
- Max Inferences per Second
- 8,914.29 /s
- Effective Throughput
- 1.248 × 10¹⁴ FLOPs/s
Step by step
- 1Effective throughput: hardware TFLOPs × 1e12 × utilization= 1.248e+14 FLOPs/s
- 2Latency: model FLOPs ÷ (hardware TFLOPs × utilization × 1e12)= 0.112 ms
- 3Max inferences/sec at this latency= 8914.29 /s
Assumptions and limits
- The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax or compounding frequency into account.
- Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
How do I find a model's FLOPs per inference?
Profiling tools like PyTorch's torch.profiler, fvcore's FlopCountAnalysis, or a model's published technical report typically report FLOPs per forward pass; for transformer-based models, this is commonly approximated as roughly 2 × parameters × sequence_length for a single forward pass.
Why does utilization matter so much for small models?
Small models often don't have enough parallel work to keep a large GPU's compute units fully occupied, so they tend to sustain lower utilization (sometimes under 20%) than large models — checking utilization against real benchmarks for your model size and hardware avoids over-optimistic latency estimates.
Is this calculator suitable for autoregressive LLM generation latency?
Not directly — LLM token-by-token generation is typically memory-bandwidth bound rather than purely compute-bound; use the Inference Latency Calculator (prefill + decode) for autoregressive generation, and reserve this FLOPs-based calculator for single-pass models like classifiers or encoders.