Determine the right Prometheus scrape interval balancing granularity and storage cost.
Scrape load = targets ÷ interval, and multiplying by average scrape duration gives the effective number of concurrent in-flight scrapes: concurrent_load = (targets / interval) × avg_duration. Comparing this against your Prometheus instance's sustainable scrape capacity (bounded by scrape_timeout, network fan-out, and worker concurrency) shows whether your current interval/target combination is safe or will cause scrape delays and gaps.
samples_per_day = (86400 / scrape_interval_sec) × targets
Scrapes start queuing and some may exceed `scrape_timeout`, resulting in missed/failed scrapes that show up as gaps in your metrics and `up == 0` for affected targets during the overload window.
It depends on your goals — if you need higher-resolution data, scale capacity (more Prometheus replicas/shards, faster hardware, network tuning); if the current resolution is already sufficient, raising the interval is usually the cheaper fix.
A target that's slow to respond (large metrics payload, slow app) ties up a scrape worker longer, so a smaller number of slow targets can create the same concurrent load pressure as many fast targets.
Watch `prometheus_target_scrapes_exceeded_sample_limit_total`, `scrape_duration_seconds`, and `up` metric flapping under load — sustained increases in scrape duration as target count grows is a sign you're approaching capacity.