Estimate model inference latency per request based on hardware throughput.
This calculator estimates single-inference latency directly from compute requirements: latency = model_flops / (hardware_tflops × utilization × 1e12). Unlike a full prefill/decode breakdown, this treats inference as a single compute-bound pass — appropriate for non-autoregressive models (e.g. image classifiers, single-pass encoders) where the entire forward pass's FLOPs are known upfront. The utilization factor accounts for the gap between a GPU's theoretical peak and what real inference workloads sustain, driven by kernel efficiency, batch size, and memory bandwidth limits for smaller models.
latency_ms = (model_flops / (hardware_tflops × utilization × 1e12)) × 1000
Profiling tools like PyTorch's torch.profiler, fvcore's FlopCountAnalysis, or a model's published technical report typically report FLOPs per forward pass; for transformer-based models, this is commonly approximated as roughly 2 × parameters × sequence_length for a single forward pass.
Small models often don't have enough parallel work to keep a large GPU's compute units fully occupied, so they tend to sustain lower utilization (sometimes under 20%) than large models — checking utilization against real benchmarks for your model size and hardware avoids over-optimistic latency estimates.
Not directly — LLM token-by-token generation is typically memory-bandwidth bound rather than purely compute-bound; use the Inference Latency Calculator (prefill + decode) for autoregressive generation, and reserve this FLOPs-based calculator for single-pass models like classifiers or encoders.