Work out flops estimate instantly with clear inputs, formula shown and shareable results.
A forward pass costs about two floating-point operations per parameter per token, because each weight participates in one multiply and one add. The backward pass costs roughly twice the forward pass, giving the widely used 6ND estimate for training and 2ND for inference. These figures ignore attention's quadratic term, which is small until context length approaches the hidden dimension.
Compute estimate
training FLOPs = 6 x N x D; inference FLOPs = 2 x N x D
At very long contexts. Attention adds roughly 12 x layers x d x seq^2 per sequence, which starts to dominate once sequence length exceeds the hidden dimension.
Memory bandwidth, kernel launch overhead and communication mean real utilisation is typically 35 to 55 percent of peak on large training runs.