Skip to content
ENExplore Calculators

RAG Pipeline Latency Calculator

Work out rag pipeline latency instantly with clear inputs, formula shown and shareable results.

About RAG Pipeline Latency Calculator

A RAG request is a serial chain: embed the query, search the index, optionally rerank, then generate. Total latency is the sum, and generation almost always dominates because it produces tokens one at a time. That matters for optimisation priorities — shaving 20 ms off vector search is invisible next to a 1.8 second generation, so streaming the response and shortening the output are the effective levers.

Formula

Pipeline latency

retrieval = embed + search + rerank; total = retrieval + generation; p95 ~ 1.5 x mean plus reranker variance

Worked example

With the default values below, RAG Pipeline Latency Calculator returns End-to-end latency of 1,955. Change any input above and every figure here is recalculated in your browser.

Inputs used

Query embedding timeminimum 0 ms
25 ms
Vector search timeminimum 0 ms
40 ms
Reranking timeminimum 0 ms
90 ms
Generation timeminimum 0 ms
1,800 ms

Results

End-to-end latency
1,955 ms
Share spent retrieving
7.93%
Share spent generating
92.07%
Rough p95 latency
2,978 ms

Step by step

  1. 1Values used
  2. 2Pipeline latency
  3. 3End-to-end latency= 1,955 ms
  4. 4Share spent retrieving= 7.93
  5. 5Share spent generating= 92.07
  6. 6Rough p95 latency= 2,978 ms

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

What is the fastest way to improve perceived latency?

Stream tokens as they are generated. Time to first token drops to roughly the retrieval time plus prefill, even though total time is unchanged.

Can retrieval steps run in parallel?

Search depends on the embedding and reranking depends on search, so the chain is inherently serial. You can however run several retrieval strategies concurrently and merge the results.

Related Calculators

RAG Pipeline Latency Calculator | Calcrivo