Skip to content
ENExplore Calculators

GPU Memory For Training Calculator

Work out gpu memory for training instantly with clear inputs, formula shown and shareable results.

About GPU Memory For Training Calculator

Training memory is weights plus gradients plus optimiser state plus activations. Mixed-precision Adam is the expensive case: it keeps fp32 master weights and two fp32 moment estimates, roughly 12 bytes per parameter on top of the fp16 weights and gradients. That is why a 7B model needs well over 100 GB to train but only about 14 GB to serve.

Formula

Training memory

total = params x bytes (weights) + params x bytes (gradients) + params x optimiser bytes + activations; Adam optimiser bytes ~ 12, SGD-momentum ~ 4

Worked example

With the default values below, GPU Memory For Training Calculator returns Total GPU memory required of 112.31. Change any input above and every figure here is recalculated in your browser.

Inputs used

Model parametersminimum 1
7,000,000,000
Bytes per weightaccepts 0.5–4
2
Optimiserone of Adam / AdamW (fp32 moments + master weights), SGD with momentum, SGD without momentum, 8-bit Adam
Adam / AdamW (fp32 moments + master weights)
Activation memoryminimum 0.1 GB
8 GB

Results

Total GPU memory required
112.31 GB
Weights
13.04 GB
Gradients
13.04 GB
Optimiser state
78.23 GB

Step by step

  1. 1Values used
  2. 2Training memory
  3. 3Total GPU memory required= 112.31 GB
  4. 4Weights= 13.04 GB
  5. 5Gradients= 13.04 GB
  6. 6Optimiser state= 78.23 GB

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

How do ZeRO and FSDP change this?

They shard optimiser state, gradients and optionally weights across devices, so per-GPU memory falls roughly by the device count at the cost of extra communication.

Why is LoRA so much cheaper?

Only the adapter parameters get gradients and optimiser state. The frozen base weights contribute their bytes once, removing the 12-bytes-per-parameter Adam cost for the bulk of the model.

Related Calculators

GPU Memory For Training Calculator | Calcrivo