Work out gpu memory for training instantly with clear inputs, formula shown and shareable results.
Training memory is weights plus gradients plus optimiser state plus activations. Mixed-precision Adam is the expensive case: it keeps fp32 master weights and two fp32 moment estimates, roughly 12 bytes per parameter on top of the fp16 weights and gradients. That is why a 7B model needs well over 100 GB to train but only about 14 GB to serve.
Training memory
total = params x bytes (weights) + params x bytes (gradients) + params x optimiser bytes + activations; Adam optimiser bytes ~ 12, SGD-momentum ~ 4
They shard optimiser state, gradients and optionally weights across devices, so per-GPU memory falls roughly by the device count at the cost of extra communication.
Only the adapter parameters get gradients and optimiser state. The frozen base weights contribute their bytes once, removing the 12-bytes-per-parameter Adam cost for the bulk of the model.