Blog

Short answer: ticking “Discourage search engines” in WordPress does not put Disallow: / in your robots.txt. It leaves robots.txt almost untouched, removes the sitemap line, and adds to every page. That is a stronger action than the wording suggests, because noindex removes pages that are already ranking.

Nearly everyone I talk to believes that checkbox writes a Disallow: / into robots.txt. I did too, until I tested it.

What actually changes when you discourage search engines in WordPress

I took a clean WordPress install, captured robots.txt and the page source, ticked the box in Settings then Reading, and captured both again.

the disappearance of the sitemap line and the arrival of a noindex tag on every page. Neither of those is what the setting appears to promise.*

Two consequences worth sitting with.

The setting is worded as a polite request and behaves as a removal order. noindex does not merely discourage. It asks search engines to drop pages they already have.

And because robots.txt is untouched, anyone auditing robots.txt alone will find nothing wrong. That is the failure I keep meeting on inherited sites: the SEO plugin looks fine, robots.txt looks fine, and the site is being actively removed from the index by a checkbox somebody ticked on a staging copy eight months ago and never unticked after launch.

Disallow and noindex are not two flavours of the same thing

This is the distinction that causes the most damage, because the two get used interchangeably.

Disallow a URL that serves a noindex, the crawler is forbidden from fetching it, so it never reads the instruction to drop the page. The URL can sit in results indefinitely.*

The same comparison as text:

The trap: Disallow a page that also carries noindex, and the noindex can never be read. The crawler is forbidden from fetching the page, so it never sees the instruction to drop it. The URL can sit in results indefinitely, which is the opposite of what both settings were trying to do.

The rule I use:

• Want it gone from results? Allow crawling, serve noindex.
• Want it never fetched? Disallow, and accept it may still show as a bare URL.
• Doing both is almost always a mistake.

Checking a client site in thirty seconds

The method is three requests against the live domain: read robots.txt, check the response for a noindex tag, and confirm WordPress’s own indexing setting. Here is each one:

# 1. does robots.txt block anything wholesale
curl -s https://example.com/robots.txt | grep -A2 -i "user-agent: *"

# 2. is a noindex being served on the pages that matter
curl -s https://example.com/ | grep -i "name=["']robots["']"

# 3. does WordPress think it should be indexed at all
#    (wp-admin, Settings, Reading, or via WP-CLI)
wp option get blog_public     # 1 = indexable, 0 = discouraged

Check two and three on the homepage and a money page, not just the homepage. Plenty of sites carry a stray noindex on a template that only affects one post type.

blog_public returning 0 on a production site is nearly always a launch that never got finished.

What good robots.txt files look like

For calibration, here is what four production sites publish today.

not spend their time filling shopping baskets. Block internal search results so they do not become indexable thin pages. And list every sitemap you publish, not only the index.*

None of them uses Disallow: /. On a live site that line is almost always an accident.

The one exception almost every real site carries, and why

There is one Allow line that shows up on nearly every production WordPress robots.txt alongside Disallow: /wp-admin/, and it is worth understanding rather than copying blindly:

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

admin-ajax.php lives inside /wp-admin/, which is exactly the directory the line above it just disallowed, but it is not an admin screen. It is the endpoint themes and plugins call for ordinary frontend behaviour: infinite scroll, a live search box, an “add to cart” button that updates without a page reload, a comment form that submits without leaving the page. Block it wholesale and a crawler that respects robots.txt strictly can be denied the one URL a page’s own JavaScript depends on to finish rendering what a visitor sees.

I checked whether real sites actually carry the exception, rather than assuming the pattern is universal:

for d in wordpress.org woocommerce.com sitec.example sitea.example siteb.example; do
  echo "=== $d ==="
  curl -s "https://$d/robots.txt" | grep -iE "admin-ajax|wp-admin"
done

Four out of five carry the exception. The fifth, sitec.example, is the interesting row, and it is not a mistake: this site is a headless Next.js frontend with no /wp-admin/ surface to protect in the first place, so there is nothing to disallow and nothing to carve an exception out of. Its own robots.txt disallows /admin, /dashboard and /api/ instead, which are the actual route names this stack uses. Reading that as “sitec.example forgot the WordPress rule” would be wrong. It is a different architecture answering a question that does not apply to it.

Real robots.txt files often have two blocks fighting for the same user-agent

Pulling the full files instead of grepping for one line surfaced something none of the earlier tables in this post caught, because every table above only ever showed one canonical block per site.

$ curl -s https://woocommerce.com/robots.txt | grep -c "User-agent: *"
2
$ curl -s https://wordpress.org/robots.txt | grep -c "User-agent: *"
3

Both sites declare User-agent: * more than once, in separate, non-adjacent blocks. Here is woocommerce.com’s file in full:

User-agent: *
Disallow: /wp-content/uploads/wc-logs/
Disallow: /wp-content/uploads/woocommerce_transient_files/
Disallow: /wp-content/uploads/woocommerce_uploads/
Disallow: /*?add-to-cart=
Disallow: /*?*add-to-cart=
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://woocommerce.com/sitemap_index.xml

# START YOAST BLOCK
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php*
Disallow: /wp-json/
Disallow: /sso?*
Disallow: /sso/*
Disallow: /*?q=*
Disallow: /*?quid=*
Disallow: /?s=
Disallow: /start/square
Sitemap: https://woocommerce.com/sitemap_index.xml
# END YOAST BLOCK

The comment answers the question before I had to guess at it: the second block is machine- generated by the Yoast SEO plugin, appended after a first block someone wrote by hand. Neither tool checked whether a User-agent: * section already existed. Each just wrote its own.

wordpress.org’s file has the same shape in miniature: a Disallow: /wp-admin/ / Allow: admin-ajax.php block, a second block a few lines later disallowing internal search (/search, /?s=), a run of named-crawler blocks for GPTBot, ClaudeBot, Google-Extended and others, and then a third, separate block at the very end disallowing /plugins/search/, almost certainly added later by whoever last touched the plugin directory’s search feature without noticing the other two * blocks already sitting in the file above it.

The robots.txt specification (RFC 9309) says a crawler that finds multiple groups matching the same user-agent should union their rules, treating them as one combined group rather than letting a later block override an earlier one. Google, Bing and the other major crawlers do implement it that way, so none of these files are actually broken. What they are is hard to read, because “what does do on this site” now requires collecting rules from two or three places in the file and combining them yourself, and smaller or less-maintained crawlers are not guaranteed to implement the union behaviour correctly. A tool that only reads the first block it finds, which is a plausible shortcut for a hastily written crawler, would miss admin-ajax.php on wordpress.org’s third block entirely, or every Yoast-added disallow on woocommerce.com.

The practical cause, both times, is the same one this whole post keeps returning to in different clothing: more than one thing thinks it owns robots.txt. A manually written block from whoever first set up the site, plus a plugin that generates its own block and appends it rather than merging into what is already there, produces a file that is correct by the letter of the spec and genuinely confusing to a human reading it top to bottom.

# check your own site for the same pattern
curl -s https://example.com/robots.txt | grep -c "User-agent: *"

A result of 1 means one source of truth. Anything higher is worth opening the file and reading both blocks together, not because it is necessarily broken, but because the next person to edit it, plugin or human, is about to add a third.

The AI crawlers get their own named blocks now, separate from *

Reading these files in full also surfaced something that would not have shown up in a single grep for admin-ajax: several of them carry dedicated, named blocks for AI crawlers, sitting apart from the User-agent: * rules entirely.

wordpress.org names ten of them individually: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, PerplexityBot, Bytespider, CCBot, Copilot, each with its own Allow: /. That is deliberate, not decorative. A blanket Disallow in the group does not automatically apply to a named crawler; per the same union-by-group logic from the section above, a crawler that identifies itself as GPTBot matches the GPTBot group specifically and ignores the group’s rules for anything the GPTBot group also covers. If a site wants an AI crawler treated differently from search engines, generically, it has to say so by name.

Two of the other domains checked earlier carry a second layer worth naming: a “Content-Signal” directive, which is a newer, separate proposal (contentsignals.org) sitting inside robots.txt rather than replacing it:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

That line is not part of the original robots.txt exclusion standard the rest of this post has been describing. It answers a different question: not “can you crawl this,” which Disallow already covers, but “what are you allowed to do with what you crawled,” search indexing, AI training, or feeding a live AI answer. A site can Allow a crawler’s access and still say no to training on the content, and the two settings are read from completely different lines, so checking one and assuming it answers the other is the same category of mistake as confusing Disallow with noindex earlier in this post.

# check whether a site declares AI-specific crawler rules or content signals
curl -s https://example.com/robots.txt | grep -iE "gptbot|claudebot|anthropic|content-signal"

An empty result does not mean AI crawlers are blocked. Most of them fall back to whatever the * group says, exactly like any other crawler that is not named individually. It means the site has not made a specific decision about them yet, one way or the other, and is currently letting the general-purpose rule decide by default.

Why this survives so long on real sites

Because nothing complains.

The site loads. Analytics still records the visitors it has. Nobody gets an email. The traffic decline reads as seasonality for the first month and as a content problem for the second.

By the time someone connects it to a checkbox, a quarter has gone. It is the same economics as every other unbilled check: a thirty-second fix that nobody is scheduled to perform.

That is the same shape as the scheduled jobs that stop without a word and the backup file left sitting in the web root: cheap to fix, invisible while it runs, and expensive precisely because it is quiet.

Protuno’s free audit reads robots.txt and the indexing signals from the domain alone, so you can see across a portfolio which sites are telling search engines to go away. Iris is live now.

In the meantime, run the three commands above against the client site you launched most recently. That is where I find it.

Straight from Google on the question this post keeps circling

Google Search Central’s own answer to exactly the kind of robots.txt judgement call this post walks through: when blocking a path helps, and when it just hides a problem instead of fixing it.

Short answer: a wordpress crawl audit checks the relationships between pages, which is the one thing an SEO plugin sitting inside a single page cannot do. It finds pages nothing links to, pages buried too deep to matter, and links pointing at URLs that no longer exist. Every individual page can return 200 while the site as a whole is broken.

I ran one against protuno.com while writing this. It found two things I did not know, and one of them is on every page of the site.

What a crawl actually produces

A crawler starts at the homepage, follows every internal link, and records what it finds. The useful column is not the status code. It is how many other pages point at each URL.

single image missing an alt attribute. One row is red. It is linked from all 24 pages.*

The good news first, because a crawl should tell you what is working: no orphan pages, no missing alt attributes, nothing buried deeper than three clicks.

Then the row I did not expect.

The defect that was on every single page

link to an endpoint that is not being served. Every page carrying that link returns 200, so nothing anywhere reports a problem.*

Click the contact email in our footer and you land on a 404.

Nothing told us. The CMS is happy, because it does not know what Cloudflare rewrote. Analytics is happy, because a 404 on a link nobody clicks generates no events. An uptime monitor is happy, because every page it checks returns 200.

This is the entire category in one example. The pages are fine. The relationship between them is broken. No per-page check will ever find it, because the problem does not live in any single page.

Why your SEO plugin will not catch this

Not a criticism of SEO plugins. They are answering a different question.

own link graph. Everything in the bottom half of that table is a relationship, which is exactly what code running inside one page cannot see.*

Caveat: “partly” in the last row is generous. A plugin can flag a noindex tag on the page it is looking at. It has no way to know robots.txt disallows that same URL two directories over, which is exactly the mismatch below.

The second finding, which I had just written a post about

The crawl cross-references robots.txt against the robots meta tag on each page. Ours disagree.

/blog as its canonical, so it cannot compete with itself. The first two rows are the mismatch.*

/auth/login is linked from 40 pages, disallowed in robots.txt, and tells search engines to index it. Because it is disallowed, that instruction can never be read. The URL can sit in results as a bare link with no description.

I wrote about that exact failure mode a couple of days ago, and then found it on our own site by crawling it. Which is the argument for crawling rather than assuming.

Four days later, neither finding is fixed

I went back and checked both findings again while expanding this post, rather than let them sit as a screenshot from the day I found them.

Neither is fixed. I want to sit with that rather than gloss over it, because it is a more honest demonstration of this post’s actual argument than the original finding was.

A crawl audit is not a service you buy once and file away. Both of these are cheap fixes: the first is a footer link, the second is one line in robots.txt or one meta tag. Neither has happened, four days on, on a site whose owner wrote a public blog post about finding them. That is not carelessness so much as it is the exact mechanism this whole category of check exists to describe: a finding with no invoice attached, competing against everything else on a Tuesday, loses. It is the same shape as the unbilled half hour that swallows every other quiet check, and writing about it does not exempt the writer from it.

Both are on the list now, again, with a second date attached instead of just the first one. That is the honest version of what a re-check accomplishes. It does not fix anything by itself. It turns “we should fix that” into a line with two dates on it, which is a harder thing to keep ignoring than a line with one. A crawl result that nobody looks at a second time is exactly as useful as no crawl at all, and this section is me not being the exception to that rule either.

The orphan report from the crawl tool was wrong

While re-checking the earlier findings I ran a second, larger crawl, 103 pages this time, and asked it for orphan pages: URLs nothing else on the site links to. It reported 89.

That is a number worth being suspicious of on its own. Eighty-six percent of a site being orphaned is not a plausible finding, it is a sign something upstream broke, and I want to show you how I caught it rather than just state the correction.

The same crawl records, for every single page, exactly which other pages link to it. So instead of trusting the orphan summary, I looked up four of the pages it had just listed as orphaned, in the crawl’s own raw per-page data from the same run.

Every one of them has real, recorded inbound links. /blog alone is linked from twenty-one other pages, footer navigation on nearly everything the crawler visited. The orphan-detection feature and the raw link data disagreed with each other, inside a single crawl, from a single run, and the raw data was the one telling the truth.

I do not know the exact cause inside the tool, and I am not going to guess at someone else’s code to sound more authoritative than the evidence supports. What I do know, because I checked it directly, is that the summary figure was wrong and the underlying data was right, which is exactly the shape of every other finding in this post: the higher-level report is the one that lies, and the raw evidence underneath it is where the truth was sitting the whole time.

The only genuinely orphaned page in that list, by the crawl’s own per-page data rather than its summary, is the homepage itself, and that is a known convention rather than a defect: a crawl starting from a URL does not count arriving there as a followed link, so the seed page always shows zero inbound links in tools built this way. Knowing that distinction is the difference between reading a crawl report correctly and repeating whatever number is largest on the page.

Running a WordPress crawl audit yourself

You do not need a subscription. The check that matters most is inbound link counts, and any crawler will give you that. What to look at, in order:

Anything returning a non-200 with inbound links above zero. This is the highest-value output of any crawl. Fix these first, always.

Pages with zero inbound links. If nothing links to a page, you are asking search engines to value something you do not value enough to link to.

Click depth above three. Depth is a rough proxy for how important your own site says a page is. A commercially important page at depth five is a structural problem.

robots.txt against the meta robots tag. They should agree. When they disagree, the one that can actually be read wins, and it is usually not the one you intended.

Canonical tags on parameterised URLs. Sorting, filtering and pagination parameters should canonicalise back to the clean URL, as our ?order=asc does.

One practical note on crawling a headless site, because it caught me out mid-write. Two of the posts I published this week returned 404 to the crawler for a few minutes after going live, then 200 once the front end revalidated. If you crawl a site that renders pages on demand and caches them, a fresh URL can look broken purely because nothing has asked for it yet. Re-crawl before you report a 404 on a page you have just published, or you will file a bug against yourself.

Crawl as a logged-out visitor. It is easy to run a crawl in a browser where you are signed in and see admin-only links, draft content and navigation that no visitor gets. The crawler should carry no cookies at all. What you want is the site a stranger sees.

Why this ends up never being done

A crawl takes minutes and needs redoing after every content change, every migration, every theme update and every time a third party rewrites your HTML, which is what happened to us.

It produces nothing to show a client when it passes. So it gets run once during onboarding and then never again, exactly like the backup file left in the web root and the scheduled jobs that stopped.

Iris, Protuno’s SEO agent, runs a read-only Technical SEO Audit on a connected site, inside your own AI app. See the playbooks page for the other SEO playbooks.

The free audit checks robots.txt, the sitemap and the indexing signals from the domain today, which catches the second finding above but not the first. The broken-link finding needs a real crawl.

Run one on your own site before you run one on a client’s. I did, and it found something on every page.

What is actually doing the crawling

A look inside how Googlebot itself decides what to fetch and how often, straight from the team that builds it, useful context for why a crawl audit like this one finds what it finds.