Size Fluentd buffer chunks and queue length to avoid backpressure and log loss.
Fluentd buffers accumulate log chunks between flushes to the output destination: buffer_size = ingestion_rate × flush_interval × retry_factor. The retry factor provides headroom so that if a downstream sink (Elasticsearch, S3, Kafka) becomes slow or briefly unavailable, Fluentd can keep buffering without hitting `buffer_queue_limit` and dropping or blocking new events.
buffer_size_MB = ingestion_rate_MBps × flush_interval_sec × retry_factor
Once `total_limit_size` is reached, Fluentd applies its configured overflow action — by default it blocks upstream input, but it can also be configured to drop the oldest chunks, both of which risk data loss or backpressure into your applications.
3-5× is a reasonable starting point for destinations with occasional transient slowness; retry-heavy or unreliable network paths (e.g. cross-region shipping) may warrant 5-10× headroom.
Memory buffers are faster but lost on crash/restart; file buffers persist across restarts at the cost of disk I/O — most production Fluentd deployments shipping critical logs use file buffers sized per this calculation.
Longer flush intervals mean more data accumulates between flushes, requiring a proportionally larger buffer — shortening the interval reduces peak buffer size but increases flush frequency and output-side request volume.