Estimate the crawl rate a site can sustain given server capacity.
Search engines throttle crawling when response times rise, so server speed indirectly sets crawl rate. Allocating around 10 per cent of capacity to crawlers is typical without affecting users. A site whose full crawl takes over a month will always have stale content in the index, regardless of how often it is updated.
Crawl Rate
Crawl rate = server capacity × allocated share; pages per day = rate × 86,400
Crawl rate = server capacity × allocated share; pages per day = rate × 86,400 Search engines throttle crawling when response times rise, so server speed indirectly sets crawl rate. Allocating around 10 per cent of capacity to crawlers is typical without affecting users.
A site whose full crawl takes over a month will always have stale content in the index, regardless of how often it is updated.
This calculator takes 4 inputs: Server capacity, Share of capacity allocated to crawlers, Average response time, Total URLs. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.