Robots.txt for WordPress & Shopify: The 2026 Perfect-Setup Guide

What WordPress's Default Robots.txt Actually Says

If your WordPress site has no physical robots.txt file, WordPress generates one virtually. The default looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

That's it. It blocks the admin area but allows the AJAX endpoint (which needs to be accessible for many plugins). It does NOT include:

• A sitemap reference
• AI training bot blocks
• Login or registration page blocks
• XML-RPC or REST API blocks
• Internal search or pagination blocks
• Comment author URL blocks

For most production WordPress sites, this is insufficient. The improved version follows.

Recommended WordPress Robots.txt (Copy-Paste)

# === AI Training Crawlers (block to opt out) ===
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

# === All other bots ===
User-agent: *
# Block admin and login
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-register.php
Allow: /wp-admin/admin-ajax.php

# Block system paths
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /xmlrpc.php

# Block internal search and pagination noise
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
Disallow: /comments/feed/

# Block trackbacks
Disallow: /trackback/
Disallow: /*/trackback/

# Allow important assets
Allow: /wp-content/uploads/
Allow: /*.css$
Allow: /*.js$

# === Sitemaps ===
Sitemap: https://yoursite.com/sitemap.xml
Sitemap: https://yoursite.com/sitemap_index.xml

Replace yoursite.com with your actual domain. The two sitemap entries cover both standard sitemaps and Yoast/RankMath sitemap indexes — include whichever matches your setup.

How to Install Custom Robots.txt on WordPress

Method 1: Yoast SEO (most common)

1. Yoast SEO → Tools → File Editor
2. If no robots.txt exists, click "Create robots.txt file"
3. Paste the recommended content from the previous section
4. Save changes
5. Verify by visiting yoursite.com/robots.txt

Method 2: Rank Math

1. Rank Math → General Settings → Edit robots.txt
2. Paste content, save
3. Rank Math will replace WordPress's virtual file

Method 3: Manual upload (FTP / file manager)

1. Save the recommended content as robots.txt in a plain text editor (VS Code, Sublime, or Notepad++ — not Word)
2. Connect to your WordPress install via FTP or your host's file manager
3. Upload to the same level as wp-config.php (the WordPress root)
4. Visit yoursite.com/robots.txt to verify — it should show your custom content, not the virtual default

The physical file takes precedence over the virtual one automatically — no additional configuration needed.

Method 4: Code snippet plugin (advanced)

For programmatic control, use a plugin like Code Snippets and add:

add_filter('robots_txt', function($output) {
  return "User-agent: GPTBot\nDisallow: /\n\n" . $output;
}, 10);

This prepends your custom rules to whatever WordPress generates dynamically. Useful if you want WordPress's defaults plus a few extras.

What Shopify's Default Robots.txt Includes

Shopify's auto-generated robots.txt is more comprehensive than WordPress's. It includes blocks for cart, checkout, accounts, search, policies, and Shopify-internal paths. The default also includes a sitemap reference automatically.

Sample Shopify default (paths vary slightly by store config):

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*?q=
Disallow: /policies/
Disallow: /search
Disallow: /apple-app-site-association

Sitemap: https://yourstore.myshopify.com/sitemap.xml

This is solid for the basics — better than WordPress's default. What it lacks: AI bot blocks, custom collection or tag blocks, and any store-specific cleanup rules.

Recommended Shopify Robots.txt Additions

You don't replace Shopify's default — you append to it via the robots.txt.liquid template. The full template should look like this:

{% raw %}{%- for group in robots.default_groups -%}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{%- endfor -%}{% endraw %}

# === Custom additions ===

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

# Block filtered URLs (duplicate content)
User-agent: *
Disallow: /*?filter=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?pf_

# Block /collections/all (large duplicate index)
User-agent: *
Disallow: /collections/all

The Liquid template at the top preserves Shopify's defaults; your custom rules are appended after.

How to Install Custom Robots.txt on Shopify

1. Admin → Online Store → Themes
2. Click the ⋯ menu on your live theme → Edit code
3. Templates folder → Add a new template
4. Choose "robots" from the dropdown — Shopify creates a robots.txt.liquid file with the default content already populated
5. Modify the file to add your custom rules (see the recommended template above)
6. Save — Shopify regenerates robots.txt immediately
7. Verify by visiting yourstore.com/robots.txt

WordPress vs Shopify: Robots.txt Comparison

Other Platforms (Quick Reference)

Verify Your Setup Works

1. Visit yoursite.com/robots.txt in a private browser window. You should see your custom content.
2. Open Google Search Console → Settings → Robots.txt. Click "Request a recrawl" to force Google to fetch the new version.
3. Use URL Inspection on a few URLs to test:
   • An admin URL — should show "blocked"
   • A normal blog/product URL — should show "allowed"
   • An AI training crawler test — not testable in Search Console, but verify the syntax in your file
4. Wait 2-3 days for crawlers to refresh their cached version.
5. Monitor Coverage report in Search Console for the next 2 weeks. "Blocked by robots.txt" entries should match your intent.

Frequently Asked Questions