Firehose Buffer Calculator

time to fill = buffer size MB ÷ throughput MB/s; delivery latency = min(time to fill, buffer interval); average object = throughput × delivery latency. Firehose flushes when either hint is satisfied, whichever comes first, so on a busy stream the size hint dictates latency and on a quiet one the interval does. That single choice sets both your end-to-end freshness and your destination object size, and small objects are what make Athena and Spark queries slow. Billing rounds every record up to 5 KB, so 1 KB records cost five times what their raw volume suggests — confirm the current per-GB rate for your region.

Buffer settings are the one Firehose knob that trades latency against query performance and cost simultaneously: a 1 MB buffer gives fresh data and thousands of tiny files, while a 128 MB buffer gives efficient Parquet objects and minutes of lag.

This calculator takes 7 inputs: Records per second, Average record size, Buffer size hint, Buffer interval hint, Compressed size as a share of raw, Firehose ingestion price per GB, Destination storage price per GB-month. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.

For Athena, Spark or Redshift Spectrum, objects of roughly 128 MB or more read far more efficiently than many small files. If freshness forces a small buffer, compact the output later with a scheduled job rather than fighting the buffer hint.

Yes. A Lambda transformation applies its own buffering before delivery, and the billed ingest is measured on the data Firehose receives, not on the smaller output your transformation produces. Dropping records in a transform does not reduce the ingest charge.