Inference Time Calculator
Estimate model inference latency per request based on hardware throughput.
Inputs
Total floating-point operations for a single forward pass/inference.
Realistic achieved fraction of peak hardware TFLOPs for this workload.
Estimated Inference Latency
0.112ms
Max Inferences per Second
8,914.29/s
Effective Throughput
1.248 × 10¹⁴FLOPs/s
Step by step
Effective throughput: hardware TFLOPs × 1e12 × utilization
312 × 1e12 × 0.4
= 1.248e+14 FLOPs/s
Latency: model FLOPs ÷ (hardware TFLOPs × utilization × 1e12)
1.400e+10 ÷ 1.248e+14
= 0.112 ms
Max inferences/sec at this latency
1 ÷ 1.122e-4 s
= 8914.29 /s
How it works
This calculator estimates single-inference latency directly from compute requirements: latency = model_flops / (hardware_tflops × utilization × 1e12). Unlike a full prefill/decode breakdown, this treats inference as a single compute-bound pass — appropriate for non-autoregressive models (e.g. image classifiers, single-pass encoders) where the entire forward pass's FLOPs are known upfront. The utilization factor accounts for the gap between a GPU's theoretical peak and what real inference workloads sustain, driven by kernel efficiency, batch size, and memory bandwidth limits for smaller models.
Formula
latency_ms = (model_flops / (hardware_tflops × utilization × 1e12)) × 1000
- model_flops
- Floating-point operations for one forward pass
- hardware_tflops
- Peak TFLOPS of the hardware
- utilization
- Fraction of peak actually sustained (e.g. 0.4)
Frequently Asked Questions
How do I find a model's FLOPs per inference?
Profiling tools like PyTorch's torch.profiler, fvcore's FlopCountAnalysis, or a model's published technical report typically report FLOPs per forward pass; for transformer-based models, this is commonly approximated as roughly 2 × parameters × sequence_length for a single forward pass.
Why does utilization matter so much for small models?
Small models often don't have enough parallel work to keep a large GPU's compute units fully occupied, so they tend to sustain lower utilization (sometimes under 20%) than large models — checking utilization against real benchmarks for your model size and hardware avoids over-optimistic latency estimates.
Is this calculator suitable for autoregressive LLM generation latency?
Not directly — LLM token-by-token generation is typically memory-bandwidth bound rather than purely compute-bound; use the Inference Latency Calculator (prefill + decode) for autoregressive generation, and reserve this FLOPs-based calculator for single-pass models like classifiers or encoders.