What is Robots.txt? The Complete 2026 Guide for SEO & AI Bots
The first thing to learn about robots.txt is the thing most people get backwards: it controls crawling, not indexing. A page you "block" in robots.txt can still show up in Google as a bare link, because you told the bot not to read the page, not to forget it exists. Once that distinction clicks, every other robots.txt decision gets easier, including the newer 2026 one of whether OpenAI, Anthropic, and Google get to train on your content. This guide builds from that foundation up.
Quick verdict: Robots.txt is a plain text file at yoursite.com/robots.txt that tells crawlers which pages they may read. It manages crawl, not indexing (use noindex for that) and not privacy (use a password for that). Major search and AI bots respect it voluntarily. Use it for crawl budget, admin/duplicate paths, and AI-training opt-outs. Generate one in 30 seconds with our free generator.
How Robots.txt Actually Works
Every legitimate web crawler — Googlebot, Bingbot, the social media link previewers, the AI training crawlers — performs the same first action when it visits a new domain. Before crawling any page, it requests https://yoursite.com/robots.txt.
If the file exists, the bot reads it and applies the rules to its own user-agent. If it doesn't exist (HTTP 404), the bot assumes everything is allowed. If the file errors out (HTTP 500 or unreachable), Google specifically pauses crawling of your site entirely until it can fetch the file successfully — one reason a broken robots.txt is much worse than a missing one.
The file uses a simple text format defined by the Robots Exclusion Protocol (now formalized as RFC 9309 in 2022). Each rule block targets a specific bot or group of bots and lists which paths are allowed or disallowed.
The Syntax: Five Directives You Need to Know
| Directive | What it does | Required? |
|---|---|---|
User-agent: | Specifies which bot the following rules apply to. Use * to target all bots. | Yes |
Disallow: | Prevents the user-agent from crawling URLs matching this pattern. | Optional |
Allow: | Overrides a previous Disallow to permit a specific path or sub-path. | Optional |
Sitemap: | Points to your sitemap.xml URL. Helps crawlers discover your URL list. | Recommended |
Crawl-delay: | Asks the bot to wait N seconds between requests. Google ignores this; Bing and Yandex respect it. | Limited support |
Real-World Examples
Allow everything (default behavior, but explicit)
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlEvery bot is allowed to crawl every path. Sitemap is provided for discovery. This is a perfectly valid baseline robots.txt.
Block admin paths from all bots
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /?s=
Disallow: /search?
Sitemap: https://yoursite.com/sitemap.xmlBlocks common admin, e-commerce checkout, and internal-search paths from all bots.
Block AI training bots, allow search engines
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlAI training crawlers are blocked. Googlebot, Bingbot, and other search-engine crawlers continue normally. This pattern is increasingly common in 2026.
Allow Googlebot only, block everything else
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /Aggressive setup — Googlebot crawls freely, every other bot (including Bing) is blocked entirely. Generally not recommended unless you have a strong reason.
The 2026 Question: AI Training Bots
The single biggest shift in robots.txt usage since 2023 is the rise of AI training crawlers. Every major AI company now operates a publicly-identified bot that respects robots.txt, giving website owners a choice about whether to participate in AI training data.
| User-agent | Company | What they use it for |
|---|---|---|
GPTBot | OpenAI | Training data for ChatGPT models |
ChatGPT-User | OpenAI | Real-time browsing when ChatGPT answers user questions |
ClaudeBot | Anthropic | Training data for Claude models |
Google-Extended | Training data for Gemini (separate from Googlebot for search) | |
PerplexityBot | Perplexity | Indexing for Perplexity's AI search |
CCBot | Common Crawl | Open-source web archive used as training data by many AI labs |
anthropic-ai | Anthropic (legacy) | Older Anthropic crawler — still respected if listed |
cohere-ai | Cohere | Training data for Cohere models |
GPTBot stops OpenAI from training on your content. It does NOT stop ChatGPT from showing your existing content if it's already in training data or if a user pastes your URL into a conversation. Robots.txt is forward-looking only.
What Robots.txt Cannot Do
This is where most confusion lives. Robots.txt is not a security tool. It tells well-behaved bots what to do; it cannot prevent bad actors from accessing anything.
- It can't hide content from indexing. Google can index pages from external links even when they're blocked in robots.txt. To prevent indexing, use a
<meta name="robots" content="noindex">tag in the page HTML. - It can't make pages private. Anyone can read your robots.txt, which actually reveals the paths you're trying to hide. Don't list sensitive admin URLs — you're publishing a map of what's worth attacking.
- It can't stop scrapers. Malicious bots, AI scrapers operating without identifiable user-agents, and content thieves routinely ignore robots.txt. For protection, use authentication, IP blocking, or web application firewalls.
- It can't remove content from Google. If a page is already indexed, blocking it in robots.txt actually prevents Google from re-crawling to see the removal — meaning the old listing stays in search results longer. Use Search Console's Remove URL tool or noindex tag instead.
- It can't enforce rules. The entire protocol is voluntary. Compliance is a courtesy, not a guarantee.
When to Use Robots.txt vs Other Tools
| Goal | Right tool | Why |
|---|---|---|
| Prevent crawling of a path | Disallow: in robots.txt | Stops bots from fetching |
| Prevent indexing of a page | noindex meta tag | Robots.txt blocks crawling but not indexing |
| Protect sensitive data | Authentication + access control | Robots.txt is public; bad actors ignore it |
| Manage crawl budget | Disallow: low-value paths | Keeps bots focused on important content |
| Opt out of AI training | Block AI user-agents in robots.txt | Voluntary but respected by major AI companies |
| Speed up bot crawling | Sitemap: directive | Helps bots discover all your URLs |
| Slow down aggressive crawling | Crawl-delay: (limited support) | Bing/Yandex honor it; Google doesn't |
How to Test Your Robots.txt
- Visit it in a browser. Go to
https://yoursite.com/robots.txtand confirm it loads. If it 404s, the bots won't find it. - Use Google Search Console's robots.txt report. Settings → Robots.txt — shows what Google sees and flags any parse errors.
- Test specific URLs. Use Google's "URL Inspection" tool to confirm whether a given URL is blocked or allowed.
- Check Bing too. Bing Webmaster Tools has its own robots.txt tester — the behavior usually matches Google but not always.
- Monitor crawl errors. Watch Search Console's Crawl Stats for unexpected drops in crawl rate after publishing a new robots.txt.
Pattern Matching: The Wildcards That Trip People Up
Robots.txt supports limited pattern matching. Getting this slightly wrong is one of the most common ways to accidentally block important content.
*— matches any sequence of characters.Disallow: /*.pdfblocks all PDFs.$— matches the end of the URL.Disallow: /*.pdf$blocks only URLs that end with .pdf (not/page.pdf?download=1).- Trailing slash matters.
Disallow: /adminblocks both/adminand/admin-stuff.Disallow: /admin/blocks only paths inside the /admin/ folder. - Case sensitivity. URLs are case-sensitive, so
Disallow: /Admin/doesn't block/admin/. Block both if needed. - Specificity wins. If multiple rules match a URL, the most specific rule applies.
Allow: /admin/public/overridesDisallow: /admin/for any URL under /admin/public/.
The analogy I use to make the crawl-vs-index distinction stick: robots.txt is a "please don't come in" sign on a room's door, not a shredder for what's inside. If a bot honors the sign, it won't enter and read the room, but if a hundred other websites have a signpost pointing at that door ("the thing you want is in there"), Google will still list the door in its directory, it just can't describe what's inside. Students always assume the sign deletes the room. It doesn't. So the question I make them answer before choosing a tool is "do I want the bot to not read this, or to not list this?" Not read, robots.txt. Not list, noindex (which means the door has to stay open so the bot can see the noindex note inside). Get that one question right and you stop blocking pages you meant to hide and accidentally hiding pages you meant to keep.
Frequently Asked Questions
Crawl, not index, is the whole foundation
Robots.txt controls which bots crawl which pages, full stop. It is not an index-control tool (that's noindex) and not a privacy tool (that's a password). Once that distinction is clear, the rest is easy: use it for crawl-budget management, admin paths, and 2026's new question of whether AI services train on your content. It's voluntary but universally respected by the majors. Our free generator builds a correct robots.txt in 30 seconds, but understanding crawl-vs-index is what keeps you from misusing it.