robots.txt
Standard file that instructs crawlers and agents which paths they may or may not access.
Definition
robots.txt is a text file at /robots.txt that tells crawlers and bots which URLs they are allowed or disallowed to request. It is used by search engines and, increasingly, by AI crawlers (e.g. GPTBot, ClaudeBot).
Relevance to GEO
A correct robots.txt is part of GEO: it defines the crawl contract for agents. Combining it with llms.txt and structured data gives agents a clear picture of what they may access and how to use your site.
See also
Related: GEO, LLM Indexing.