Skip to content
ENExplore Calculators

Crawl Rate Calculator

Estimate the crawl rate a site can sustain given server capacity.

About Crawl Rate Calculator

Search engines throttle crawling when response times rise, so server speed indirectly sets crawl rate. Allocating around 10 per cent of capacity to crawlers is typical without affecting users. A site whose full crawl takes over a month will always have stale content in the index, regardless of how often it is updated.

Formula

Crawl Rate

Crawl rate = server capacity × allocated share; pages per day = rate × 86,400

crawl allowance
Portion of capacity reserved for crawlers without harming users

Worked example

With the default values below, Crawl Rate Calculator returns Sustainable Crawl Rate of 12.000. Change any input above and every figure here is recalculated in your browser.

Inputs used

Server capacityminimum 0.01
120
Share of capacity allocated to crawlersaccepts 0–100 %
10 %
Average response timeminimum 1 ms
380 ms
Total URLsminimum 1
45,000

Results

Sustainable Crawl Rate
12.000 req/s
Pages Crawled per Day
1,036,800
Days for a Full Crawl
0.04 days
Share of Server Capacity Used by Crawling
10.00%
Recommendation
Crawl capacity is adequate for the site size

Step by step

  1. 1Values used
  2. 2Crawl Rate
  3. 3Sustainable Crawl Rate= 12.000 req/s
  4. 4Pages Crawled per Day= 1,036,800
  5. 5Days for a Full Crawl= 0.04 days
  6. 6Share of Server Capacity Used by Crawling= 10.00
  7. 7Recommendation= Crawl capacity is adequate for the site size

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

How is Crawl Rate calculated?

Crawl rate = server capacity × allocated share; pages per day = rate × 86,400 Search engines throttle crawling when response times rise, so server speed indirectly sets crawl rate. Allocating around 10 per cent of capacity to crawlers is typical without affecting users.

Why does Crawl Rate matter?

A site whose full crawl takes over a month will always have stale content in the index, regardless of how often it is updated.

What values do I need to enter?

This calculator takes 4 inputs: Server capacity, Share of capacity allocated to crawlers, Average response time, Total URLs. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.

Related Calculators

Crawl Rate Calculator – Calculate Sustainable Crawl Rate