Work out training time estimate instantly with clear inputs, formula shown and shareable results.
Single-device work is total steps times step time. Adding devices divides that, but not perfectly: gradient synchronisation grows with device count, so an efficiency factor of roughly 1/(1 + 0.05 log2 devices) captures the observed sub-linear scaling. Device-hours, not wall-clock hours, are what the invoice is based on, and they rise as you add devices.
Distributed training time
single-device hours = steps x step time; efficiency = 1 / (1 + 0.05 log2 devices); wall clock = single-device hours / (devices x efficiency)
Communication overhead is pure extra work. You buy shorter wall-clock time with more total compute, which is usually worth it for iteration speed but costs more money.
Larger per-device batches, gradient compression or bucketing, faster interconnect such as NVLink or InfiniBand, and overlapping communication with backward computation.