Marketing
Saved & recent

Web crawling

Also called: crawling, search crawler, spider

Web crawling is the process in which automated software fetches pages and resources to discover what is on the web. Links and sitemaps can help a crawler find URLs.

A discovered URL may not be fetched, and a fetched page may not be indexed. Crawling is therefore one stage in search visibility, not a promise that a page will appear in results.

Related
Search indexing, Internal linking, Search engine optimization
Further reading
Google Search Central โ€” In-depth Guide to How Google Search Works
crawler home fetch guide A guide B fetched queued sitemap URL C fetch URL discovery
Conceptual crawl state. Discovering a URL does not mean its content has been fetched, indexed, or selected for results.

View in the full Marketing collection