Estimate the size of a newline-delimited JSON dataset.
JSON Lines trades the enclosing array for one object per line, which makes the file streamable and splittable at any newline. That property is what allows parallel processing without parsing the whole file. Shard sizing matters for distributed processing because each shard becomes one task, and shards that are too large prevent even work distribution.
JSON Lines Size
Size = lines × (average line bytes + one newline byte)
Size = lines × (average line bytes + one newline byte) JSON Lines trades the enclosing array for one object per line, which makes the file streamable and splittable at any newline. That property is what allows parallel processing without parsing the whole file.
Shard sizing matters for distributed processing because each shard becomes one task, and shards that are too large prevent even work distribution.
This calculator takes 4 inputs: Number of lines, Average line size, Gzip compression saving, Target shard size. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.