Is the robots.txt generator free?
Yes. It's completely free with no sign-up.
Quick Presets
Block AI Training Bots
Stops GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended and more from training on your content.
User-Agent Rule Groups
Sitemap URLs
Use absolute URLs (https://...). Add a separate line for each sitemap, including news / image / video sitemaps.
Preferred Host (optional, Yandex)
Only respected by Yandex. Tells Yandex which host to treat as canonical when you have multiple mirrors.
100% private — nothing is uploaded. Upload the generated robots.txt to your site's root directory.
robots.txt Preview
Live
0 bytes
0 rules
URL Tester
How to install
1
Click Download robots.txt above (or Copy & paste into a new file named
robots.txt).2
Upload to your site's root:
/public_html/robots.txt (cPanel) or via FTP.3
Verify by visiting
yoursite.com/robots.txt.4
Test in Google Search Console → Robots.txt report.
More Free Tools
Saved Configurations
Stored locally in your browser.
No saved sets yet. Build a configuration and click Save.
Benefits
Why You Need a Robots.txt
One small file controls how search engines and AI crawlers see your site.
Protect content
Block AI training bots from scraping your work.
Save crawl budget
Stop Google wasting crawls on /admin or /cart.
Announce sitemaps
Tell crawlers where to find your sitemap(s).
Cleaner SERPs
Avoid duplicate-content from tag/search pages.
Reference
Common Bots You Might Want to Manage
| Bot | Owner | Purpose | Typical Action |
|---|---|---|---|
Googlebot | Search index | Allow | |
Bingbot | Microsoft | Bing search | Allow |
Google-Extended | Gemini AI training | Block (optional) | |
GPTBot | OpenAI | ChatGPT training | Block (recommended) |
ClaudeBot | Anthropic | Claude training | Block (recommended) |
PerplexityBot | Perplexity | Perplexity AI search | Block (optional) |
CCBot | Common Crawl | Used by many AI projects | Block (recommended) |
Bytespider | ByteDance / TikTok | AI training | Block (recommended) |
Amazonbot | Amazon | Alexa / AI | Block (optional) |
AhrefsBot | Ahrefs | SEO crawler | Block if not subscriber |
SemrushBot | Semrush | SEO crawler | Block if not subscriber |
MJ12bot | Majestic | SEO crawler | Often blocked |
Guide
From Blank to Live in 4 Steps
1
Pick a preset
WordPress, Shopify, blog, e-commerce — preset fills the rules.
2
Block the AI bots
One click adds blocks for GPTBot, ClaudeBot, CCBot, etc.
3
Add sitemaps
Paste your sitemap.xml URL(s). Test paths in the URL Tester.
4
Download & upload
Save as robots.txt and put it in your site's root folder.
FAQ
Frequently Asked Questions
It stops well-behaved ones. OpenAI (GPTBot), Anthropic (ClaudeBot), Common Crawl (CCBot) and Google (Google-Extended) all honour robots.txt. Less-scrupulous scrapers ignore it — for those you'd need server-level blocks (firewall, Cloudflare bot rules).
Always at the root of your domain:
https://yoursite.com/robots.txt. Crawlers only look there. Subdirectories don't work. On cPanel, that's /public_html/robots.txt.No — it just stops crawling. A page already in the index can still show in search results (just without a snippet). To remove a page, use a
noindex meta tag and let Google crawl it. Counter-intuitively, blocking it in robots.txt prevents Google from seeing the noindex.Disallow blocks a path. Allow creates exceptions to a Disallow. Example:
Disallow: /wp-admin/ + Allow: /wp-admin/admin-ajax.php blocks everything in /wp-admin/ except admin-ajax. Most specific (longest) rule wins.Yes.
* matches any sequence. $ at end means exact match. Examples: Disallow: /*.pdf$ blocks all PDFs. Disallow: /search?* blocks anything starting with /search?.By Bingbot and YandexBot — yes. Google has ignored Crawl-delay since 2019 (manage Google's crawl rate inside Search Console instead). Set it for Bing/Yandex if you have small servers.
Grow Your Business with Smart Marketing Tools
All the tools you need to grow faster, rank higher & earn more.