Concept

Crawler

Crawling is the process by which a search engine, using a crawler (also known as a bot or spider, such as Googlebot), automatically visits web pages and analyzes their content.

How a crawler explores the web

A crawler starts with known URLs and then follows every link it encounters to new pages, thereby mapping out the web step by step. What a crawler encounters and understands on a page determines whether and how that page will ultimately appear in search results.

Controlling Crawlers

Two files largely determine how crawlers access a website:

  • Robots.txt: specifies which parts of the site may or may not be crawled.
  • XML sitemap: indicates which pages exist and their priority.

Crawl Budget

Large websites are subject to a crawl budget: the number of pages a crawler visits within a given period. A confusing site structure, numerous broken links, or unnecessary duplicate pages waste that budget on unimportant content, at the expense of the pages that really matter.

Are you unsure whether your website is being crawled effectively? Contact us or schedule a strategy meeting.

← Back to the Glossary