Estimate the total cost of fine-tuning an LLM on a custom training dataset.
Fine-tuning cost is estimated by first computing total training compute (FLOPs), then converting to time using cluster throughput, then to dollars using the GPU's hourly rate: time_hours = (tokens × epochs × 6 × params) / (gpu_tflops × 1e12 × utilization × 3600), and cost = time_hours × gpu_rate × gpu_count. The factor of 6 approximates combined forward (≈2×params FLOPs/token) and backward (≈4×params FLOPs/token) pass compute per training token, and a 40% utilization factor accounts for realistic overhead from data loading, gradient synchronization, and imperfect kernel efficiency in real training runs.
Training time
time_hours = (tokens * epochs * 6 * params) / (gpu_tflops * 1e12 * utilization * gpu_count * 3600)
Total cost
cost = time_hours * gpu_rate * gpu_count
Real training workloads rarely sustain a GPU's advertised peak throughput due to communication overhead between GPUs, memory bandwidth limits, and non-compute time spent on data loading and checkpointing; 40% is a reasonable, commonly observed average for multi-GPU fine-tuning.
The same FLOPs-based formula applies to both, but fine-tuning typically uses far fewer training tokens (millions to low billions) than pre-training from scratch (trillions), so fine-tuning costs are usually orders of magnitude lower.
No, this estimates only the raw GPU compute time and cost for the training loop itself — data preprocessing, evaluation runs, and hyperparameter search are separate costs not included here.
Yes — parameter-efficient fine-tuning methods like LoRA train a small fraction of additional parameters instead of the full model, dramatically reducing both compute (FLOPs) and memory requirements compared to full fine-tuning, though this calculator assumes full-parameter fine-tuning.