Web Crawler

A web crawler (also spider or search bot) is an automated program that systematically traverses the web, reads content and stores it in an index. Search engines like Google, Bing or DuckDuckGo are based entirely on such crawlers, without them there would be nothing to search. Starting from a list of known URLs (seed URLs), they follow every link, find new pages, refresh their data basis and make the found information available to other services. Anyone running a website who wants to appear in Google search must make access easy for the crawlers while steering them into the right channels.

How does a crawler work?

A crawler works in a loop:

1. Seed: it starts from a list of known URLs (e.g. from earlier crawls, sitemap submissions, inbound links).
2. Fetch: it retrieves each URL and loads the HTML.
3. Parse: it reads out links (<a href>), images (<img src>), scripts, structured data and metadata.
4. Index: relevant content moves into the search index.
5. Queue: new links are added to the crawl queue if not already known.
6. Re-crawl: known URLs are re-checked at intervals to capture updates.

Modern crawlers like Googlebot additionally render pages with a real browser (JavaScript execution) to capture dynamically generated content too, but with delay and a resource limit.

Well-known web crawlers

Every crawler identifies itself via its user agent string, by which you recognise it in logs and address it in robots.txt.

Crawl budget

Search engines allocate each website a limited crawl budget: how many URLs they visit within a period. For small sites, that easily suffices for the entire domain. For large websites (online shops, magazines, forums with filter URLs), the budget quickly gets tight:

• Weak URLs (filter permutations, empty lists, duplicate content) devour budget without delivering new value
• Important URLs get crawled less often because the budget is used up elsewhere

Optimisation: lock out unimportant paths via robots.txt or noindex, set canonical URLs cleanly, speed up slow servers (see LCP).

Controlling crawlers

robots.txt

The robots.txt in the web root is the most important control instrument, a simple text file telling crawlers where they may go and where not:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://www.aceart.de/sitemap.xml

Important: robots.txt is a request, not technical protection. Reputable crawlers respect it; spam bots ignore it. Confidential data must never be “hidden” via robots.txt alone.

Meta robots in the HTML

Per page, indexing can be controlled via the meta tag meta robots:

<meta name="robots" content="noindex, follow">

Common values: index / noindex, follow / nofollow, noarchive, max-snippet:0. Also targetable at individual crawlers (<meta name="googlebot" content="noindex">).

The HTTP header X-Robots-Tag

For non-HTML resources (PDFs, images, JSON APIs) there’s the HTTP header X-Robots-Tag: noindex as the meta tag’s counterpart.

The canonical link

With several URLs sharing similar content, the <link rel="canonical"> specifies which version Google should rank, saving crawl budget and preventing duplicate content problems.

The sitemap as a signpost

A sitemap lists all of a site’s important URLs, a friendly hint to the crawler of what’s to be found where. Particularly helpful for large sites and newly published content.

Excluding crawlers – when does it make sense?

• Staging and test environments: definitely block with Disallow: / (plus basic auth!)
• Admin/login areas: no indexing value, a risk with accidentally exposed URLs
• Filter/sort URLs: ?sort=…&color=… produces a combinatorial explosion of URLs without added value
• PDF duplicates of HTML content: lock out via X-Robots-Tag: noindex
• AI training crawlers (GPTBot, ClaudeBot, CCBot): a conscious decision about whether your own content is released for LLM training

Recognising crawlers in your own server logs

Incoming crawlers leave traces in the access logs. Typical patterns:

66.249.79.42 - - [22/May/2026:10:15:22 +0200] "GET /sitemap.xml HTTP/2" 200 - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The user agent reveals the bot, and the IP address can be checked against official crawler IP lists (Google offers a reverse DNS check under googlebot.com). Spam bots frequently fake their user agent – only an IP reputation check or a WAF helps there.

Frequently asked questions about web crawlers

How often does Googlebot visit my site? Depends on update frequency and authority. News sites get crawled several times a day, static marketing sites every few weeks. The exact frequency can be tracked in Search Console.

What’s the difference between crawling and indexing? Crawling = the bot visits the page. Indexing = the content lands in the search index. A page can be crawled yet still not indexed (e.g. due to noindex or quality problems).

Can I see when Googlebot visits my site? Yes: in the server access logs, and summarised in Google Search Console under “Crawling → statistics”.

Does a crawler consume bandwidth? Yes, but usually marginally; Googlebot crawls a few MB per day even on mid-sized websites. Conspicuous load tends to come from aggressive SEO tools or badly configured third-party systems.

Conclusion

Web crawlers are the invisible helpers behind every search engine. Understand how they work (seed URLs, crawl budget, robots.txt, meta robots) and you can open your website up to them in a targeted way and deliberately lock out unimportant areas. Misconfigured crawler directives are one of the most common causes of sudden visibility losses in Google. We check and optimise the crawl strategy as part of every SEO audit – feel free to reach out for a no-obligation consultation.