Crawl budget
The resources a search engine devotes to crawling a site in a given period, determined by the load the server can handle and the demand for its pages.
- 01
Why it matters
For sites with a few hundred pages it is usually not an issue; for shops, forums and news sites with tens of thousands of pages, if crawling is spent on the wrong pages, new and updated pages are found late. Needless parameter URLs, redirect chains and slow server responses waste the budget.
- 02
Example
On a large forum, the crawler is crawling millions of addresses with sort and filter parameters. Once those addresses are closed off and the sitemap lists only canonical pages, new topics start being crawled the same day.
- 03
Common mistake
Worrying about crawl budget on a small site. With few pages, the real problem is usually content quality or internal links; crawl budget is a concern for large sites.
- 04
Let's talk about your project.
Tell us what you need; we will define the scope together.