Dataflow Cost Calculator

cost = average workers × (vCPU × vCPU-hour price + GB × GB-hour price) × hours + GB processed × shuffle or Streaming Engine price + worker disk. Dataflow bills worker resources by the hour and shuffle volume by the gigabyte, so a pipeline with a wide GroupByKey can spend more on data processed than on workers. Enabling Streaming Engine shifts state off the workers, which lets each worker use a 30 GB disk instead of hundreds of gigabytes — a real saving that the disk line here reflects.

Autoscaling means the maximum worker count is a safety limit rather than a cost, so estimating from peak workers overstates the bill badly while ignoring shuffle understates it. Confirm the per-vCPU-hour and per-GB data-processed rates for your region in the Google Cloud pricing calculator.

This calculator takes 13 inputs: Maximum workers, Average share of maximum workers used, vCPUs per worker, Memory per worker, Pipeline hours per month, Price per vCPU-hour, Price per GB-hour, Data processed by shuffle, Batch shuffle price per GB, Use Streaming Engine, Streaming Engine price per GB, Persistent Disk per worker, Disk price per GB-month. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.

Because shuffle is billed per GB moved and a single GroupByKey or join can move many times your input volume. Reducing key cardinality, filtering earlier in the graph and using combiners to pre-aggregate all cut shuffle bytes directly.