Should you block GPTBot? Training and Search are independent

What result are you trying to get?

A small publisher might have one concern about training and another about distribution. Keeping its latest reporting available to search can serve readers even when the publisher declines automatic training collection. There is no need to give those choices the same answer.

The table is the decision I would write before opening a hosting dashboard. It prevents a crawler preference from being mistaken for a privacy control or a promise about where the publication will appear.

What does a GPTBot disallow change?

GPTBot is the automatic crawler whose documented purpose includes gathering material that may be used for OpenAI foundation-model training. A disallow communicates your choice for that use. OpenAI’s search agent has a different token, and ChatGPT-User handles certain user-initiated actions; OpenAI says robots rules may not apply to the latter. OpenAI’s agent definitions.

This makes the wording of a change request matter. “Decline GPTBot collection of public essays” is implementable. “Make the site disappear from AI” leaves unanswered which service, which workflow and which copy of the material the owner means.

If the concern is sensitive information, move the decision to the server. A URL listed under Disallow remains a publicly disclosed path, and an HTTP client can request it without following the file. RFC 9309 defines the protocol as crawler rules rather than authorization. The practical boundary is covered in why robots.txt cannot protect a private page.

How do you keep search open while blocking GPTBot?

This complete example starts with a public site that excludes its internal search and previews from ordinary crawling. It repeats those exclusions in the named search group so that adding the group does not reopen them. Replace the example hostname and reconcile every existing rule before using it.

The repeated paths are deliberate. Under the protocol, a matching named group supplies the crawler’s rules; the wildcard group is the fallback. Two groups for the same product token combine, so appending another GPTBot group is not a reliable way to overwrite an earlier policy. RFC 9309 grouping rules.

User-agent: *
Disallow: /search/
Disallow: /preview/

User-agent: OAI-SearchBot
Allow: /
Disallow: /search/
Disallow: /preview/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

What would count as a successful change?

I would accept the configuration when the public host serves the intended file and the parser returns the expected decisions for an article and an excluded path, checking the public response because a CMS, cache, or CDN can still serve a version different from the one in the repository.

Access deserves its own check. Preserve a normal GET response for a public article, including the body. If a real OAI-SearchBot request later reaches the edge, join its identity evidence to the delivered response using the server-log verification method. A curl request that copies the bot’s name comes from your network and tests only that synthetic request.

Write down the observation window. A missing attributed request leaves provider delivery unverified during that period, while a denied request can show the WAF acting before the crawler refreshes its policy. Either result needs its event time beside the policy version.

• Save the old and new public robots.txt bodies with the deployment time.
• Check GPTBot against a public article and OAI-SearchBot against both allowed and excluded URLs.
• Inspect a returned article body for the expected text, login screens or challenges.
• Keep any later attributed request with its timestamp, response and policy version.

When would I leave GPTBot allowed?

An owner may want public material used in model development or have an agreement that makes collection worthwhile. Allowing GPTBot is then a deliberate choice based on rights and the publication’s business model.

I would judge the choice against the owner’s goals and measured costs or referrals. The broader crawler guide maps the other workflows. Here my default is to keep Search working and make the training choice explicit, without attaching an unsupported ranking reward or penalty to it.