Work out data pipeline throughput instantly with clear inputs, formula shown and shareable results.
A pipeline that must clear a day of records inside a shorter window needs throughput equal to records divided by the window in seconds, not by 86,400. Compressing a day's work into six hours triples the required rate. Worker count follows from measured per-worker throughput, and the bandwidth figure checks whether the network or object store can actually feed them.
Pipeline sizing
required rate = records / (window hours x 3600); workers = ceil(required rate / per-worker rate); bandwidth = bytes x 8 / window seconds
Because capacity must cover the peak rate, not the average. A nightly batch that must finish before business hours has a much tighter rate requirement than the same volume spread over a day.
Deserialisation and small-file overhead far more often than the transformation itself. Batching records and using columnar formats typically gives the largest gains.