Skip to content
ENExplore Calculators

Data Pipeline Throughput Calculator

Work out data pipeline throughput instantly with clear inputs, formula shown and shareable results.

About Data Pipeline Throughput Calculator

A pipeline that must clear a day of records inside a shorter window needs throughput equal to records divided by the window in seconds, not by 86,400. Compressing a day's work into six hours triples the required rate. Worker count follows from measured per-worker throughput, and the bandwidth figure checks whether the network or object store can actually feed them.

Formula

Pipeline sizing

required rate = records / (window hours x 3600); workers = ceil(required rate / per-worker rate); bandwidth = bytes x 8 / window seconds

Worked example

With the default values below, Data Pipeline Throughput Calculator returns Throughput needed of 23,148. Change any input above and every figure here is recalculated in your browser.

Inputs used

Records to process per dayminimum 1
500,000,000
Average record sizeminimum 1 bytes
512 bytes
Records per second per workerminimum 0.01
800
Processing windowaccepts 0.1–24 h
6 h

Results

Throughput needed
23,148 rec/s
Workers required
29
Data volume per day
238.42 GiB
Sustained ingest bandwidth
94.82 Mbps

Step by step

  1. 1Values used
  2. 2Pipeline sizing
  3. 3Throughput needed= 23,148 rec/s
  4. 4Workers required= 29
  5. 5Data volume per day= 256,000,000,000
  6. 6Sustained ingest bandwidth= 94.82 Mbps

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why does the window matter more than the volume?

Because capacity must cover the peak rate, not the average. A nightly batch that must finish before business hours has a much tighter rate requirement than the same volume spread over a day.

What usually limits per-worker throughput?

Deserialisation and small-file overhead far more often than the transformation itself. Batching records and using columnar formats typically gives the largest gains.

Related Calculators

Data Pipeline Throughput Calculator | Calcrivo