Work out rag pipeline latency instantly with clear inputs, formula shown and shareable results.
A RAG request is a serial chain: embed the query, search the index, optionally rerank, then generate. Total latency is the sum, and generation almost always dominates because it produces tokens one at a time. That matters for optimisation priorities — shaving 20 ms off vector search is invisible next to a 1.8 second generation, so streaming the response and shortening the output are the effective levers.
Pipeline latency
retrieval = embed + search + rerank; total = retrieval + generation; p95 ~ 1.5 x mean plus reranker variance
Stream tokens as they are generated. Time to first token drops to roughly the retrieval time plus prefill, even though total time is unchanged.
Search depends on the embedding and reranking depends on search, so the chain is inherently serial. You can however run several retrieval strategies concurrently and merge the results.