Estimate storage savings from compressing log data before archival or shipping.
Compressing logs before long-term storage shrinks both disk footprint and cost: compressed = raw × (1 − compression_ratio), with savings = raw − compressed. Text-based logs compress especially well because of repetitive structure (timestamps, field names, common log levels), often reaching 80-90% size reduction with standard codecs like gzip or zstd.
compressed_GB = raw_volume_GB × (1 - compression_ratio_percent / 100)
Plain text and JSON logs typically compress 80-90% with gzip due to high repetition (field names, timestamps, common values); already-compressed or binary payloads embedded in logs will compress far less.
Yes — compressed logs generally must be decompressed before searching unless your storage/search engine supports compressed-block scanning, so there's a tradeoff between storage savings and query latency for cold/archived data.
Compressing before shipping (e.g. gzip at the agent) reduces network bandwidth too, not just storage — many log shippers support this natively (e.g. Fluent Bit's gzip output compression).
zstd generally offers a better compression-speed tradeoff than gzip at similar or better ratios, and is increasingly the default choice for high-throughput log pipelines that need to compress in real time.