Web crawling
Also called: crawling, search crawler, spider
Web crawling is the process in which automated software fetches pages and resources to discover what is on the web. Links and sitemaps can help a crawler find URLs.
A discovered URL may not be fetched, and a fetched page may not be indexed. Crawling is therefore one stage in search visibility, not a promise that a page will appear in results.