---
title: "GEO: how ChatGPT, Claude and Gemini find and read websites"
url: "https://romankryvolapov.com/en/geo/"
description: "GEO in practice: how AI assistants find and read a site, which robots to let in, what to set up in robots.txt, Cloudflare and other CDNs, how to prepare text and markup, and how to measure the result."
language: en
published: 2026-11-11
updated: 2026-11-17
---
**Hi!**

It used to be that a person asked a search engine a question and got ten links. Now they increasingly ask ChatGPT, Gemini, Claude, Perplexity or Copilot and get a ready-made answer. Such an answer has two or three links, and sometimes none. Sometimes the assistant goes further: it opens a website itself, fills in a form and puts a product in a cart. If a site is not ready for this, it drops out of the answers or shows up in them with mistakes.

This article is for people who run a blog, a company website or an online store and want to understand what to do about it. You do not need to program for this: code appears only in a few examples, and you can skip them. What is inside:

- what GEO is and how it differs from ordinary SEO;
- how assistants find and read websites, which robots come and how to manage them;
- what to check in Cloudflare, AWS and other CDNs, in site builders and on Astro;
- how to prepare content and markup;
- how to check that everything works, and which pieces of advice do not work;
- what has been done on this very site.

How an assistant can not only read a site but also act on it, and how purchases through AI work, is covered in the second article: ["MCP and WebMCP: how AI assistants act on websites and shop in online stores"](/en/mcp-webmcp/).

If you are short on time, start with three checks. Each takes a few minutes:

1. Look in your hosting or CDN dashboard to see whether AI robots are blocked there. Where to look is in the section on Cloudflare and other providers.
2. Make sure the text of your pages is visible without JavaScript. How to do it is in the section on content.
3. Open the reports on impressions in AI answers in Search Console and Bing Webmaster Tools. They are covered in the section on measurement.

Everything here describes the state of things in November 2026. The field changes fast. Wherever I rely on a retelling rather than on a company's own documentation, I say so: "according to…", "reportedly". Links to the primary sources are in the text.

## What is GEO and how does it differ from SEO?

**SEO** (Search Engine Optimization) is the work of making a site findable in Google, Bing and other search engines. **GEO** (Generative Engine Optimization) is the work of making AI assistants find your site, cite it and retell it correctly. A generative engine is any system that does not just show a list of links but writes an answer itself: AI Overviews (the AI summary above Google's results) and AI Mode (Google's conversational mode), ChatGPT with search, Perplexity, Copilot, Claude with web search.

Two more terms show up alongside it: **AEO** (Answer Engine Optimization) and **LLMO** (Large Language Model Optimization). In essence they are the same thing. From here on I write only GEO.

The main difference is in the result. In classic search the goal is to reach the first page and get a click. In an assistant's answer, a site is either mentioned as a source or it is not. Such links get clicked less often: the person has already received the answer. So what matters is not your position in the results but whether you are cited and what is said about you.

At the same time, GEO stands almost entirely on ordinary SEO. In its [guide to AI optimization](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), Google says outright that AI features in search rely on the same ranking and quality systems as regular search and need no special optimization. The same guide lists what you do need: original, useful text, clear headings, content that is easy to skim and, for stores, a catalog in Merchant Center and a business profile. Google's official position: GEO and AEO are still SEO.

And yet a separate discussion of GEO is justified, for two reasons.

1. **There are more robots now, and they are different.** Some collect texts to train models, others work as search, and still others arrive the second a person asks something. You have to treat them differently, and an ordinary SEO checklist says nothing about this.
2. **Agents that act have appeared.** An assistant does not only read a page, it also calls a site's functions: searches for a product, checks availability, submits a form. There are new protocols for this, MCP and WebMCP. They have [an article of their own](/en/mcp-webmcp/).

### Why does this matter right now?

Imagine a tea store. Before, a customer typed "buy sencha green tea" into a search engine, saw ten links, opened three of them and compared prices on their own. Now they write to an assistant: "find a good green tea under 15 euros with delivery to Germany in three days". The assistant compares stores itself and shows a short list. If your store's page does not state the delivery time in a form a machine can read, or the assistant's robot could not open the page, you are not on the list. The customer will not even know: they never saw that you could have been chosen.

It is the same with a blog. Someone asks: "how do I set up hooks in Claude Code". The assistant reads several sources, assembles an answer and says where it took each part from. Whether you end up among those sources depends on three things: whether search found you, whether the robot could read the page, and whether the answer is easy to find on it.

That is the point of GEO: to give the assistant no reason to lose you at any of these steps. To do that you need to understand how it works. We start with how assistants learn about sites at all.

## How does an AI assistant learn about a site?

An assistant learns about the world through two independent channels. It is important to tell them apart, because each is managed differently.

**The first channel is training.** A company collects a huge set of texts, including pages from websites, and trains a model on it. The model remembers general patterns and facts, but it does not remember addresses and cannot quote yesterday's article. New models are trained once every few months, so what you write today will not reach a model for a long time, and may never reach it. The robots that collect texts for training are called training crawlers.

**The second channel is search and reading at the moment of the question.** When the assistant is not sure of an answer, or the question needs fresh data, it goes to the internet. There are two kinds of robots here:

- **search robots** crawl sites in advance and put them into an index, the way an ordinary search engine does;
- **user-initiated fetchers** come to a page the second a person asks: "read this link" or "look at the site and tell me about it".

That gives three kinds of visitors: training robots, search robots and "on request" robots. Their goals differ, and you can decide about each kind separately. Search robots almost always deserve to be let in: without them you are not in the answers. With training robots the choice is yours. Many site owners keep them out so as not to give their texts away for free. Others let them in so that models know about them even without search. This site, for example, is open to all of them; why is explained in the section on how it is set up.

### Whose index does each assistant use?

An index is a list of pages that search runs over. Here is where each assistant gets it:

| Assistant | Where it gets pages from |
|---|---|
| Google AI Overviews and AI Mode | the regular Google Search index; the page must be indexed and eligible to show snippets, there are no other conditions ([Google documentation](https://developers.google.com/search/docs/appearance/ai-features)) |
| Gemini (app) | the Google Search index; whether your pages may be used is decided by the Google-Extended token (more on it below) |
| ChatGPT with search | sites open to the OAI-SearchBot robot get into answers ([OpenAI](https://developers.openai.com/api/docs/bots)); the company does not disclose what role the Bing index plays |
| Perplexity | its own index, filled by PerplexityBot ([documentation](https://docs.perplexity.ai/guides/bots)) |
| Claude | by all signs, web search runs on Brave Search: Anthropic has not announced this, but in March 2025 it added Brave to its official list of subprocessors that receive data ([Simon Willison](https://simonwillison.net/2025/Mar/21/anthropic-use-brave/)) |
| Copilot | the Bing index (according to secondary sources) |
| Meta AI | reportedly, the Meta-WebIndexer robot collects the sources it cites |

The conclusion is simple. To get into Google's answers, you have to be in Google's index; into Copilot, in Bing's index; into ChatGPT, let in the OAI-SearchBot robot. All of this is ordinary site hygiene: a sitemap (a file listing all your pages), normal indexing and open access for robots.

### What happens when a person asks a question?

In simplified form, an assistant with search does this:

1. It decides whether its own knowledge is enough. If the question is about recent events, prices or documentation, it goes to search.
2. It turns the question into one or several search queries. They are usually longer and more specific than what a person types.
3. It gets a list of pages from the index.
4. It reads the selected pages, whole or in fragments.
5. It writes an answer and links to the sources the facts came from.
6. If the person then asks "open this link", a user-initiated robot comes.

A site can drop out at any step:

- the site is not in the index: it does not get into the list at step 3;
- the robot is blocked: the page is not read at step 4;
- the page is empty without JavaScript: it is read as empty;
- the answer is buried in the middle of a long text: it is not cited at step 5.

Next we go through each of these cases.

### What does an assistant do with a page it has read?

A page that has been read does not go into the answer whole. The assistant pulls the facts it needs out of it and retells them. Three consequences follow, and they are worth keeping in mind when you write text.

1. **A retelling can distort.** If one place on the page says "delivery 4.9 euros" and another says "free delivery from 30 euros", and nothing connects them, the assistant may pick the wrong one. Put the condition next to the claim: "delivery is 4.9 euros, and free for orders over 30 euros".
2. **The link goes to whoever stated the fact clearly.** If two sites say the same thing and one did it in a single clear sentence, that one is more likely to get the link. There is no guarantee, but the logic is clear: it is easier for the assistant to take a ready-made wording.
3. **Contradictions hurt.** If a site names different prices, versions or dates in different places, the assistant will either choose at random or drop the source altogether. Keep prices, deadlines and versions in one place: this matters more than any "optimization".

All of this makes a site convenient for people too, so you do not have to choose between "writing for people" and "writing for assistants".

## Which robots come from AI, and how do you manage them?

Each robot introduces itself with a string in the request header. It is called the **user agent**, and the server uses it to tell who has come. **robots.txt** is a text file in the root of a site that says, for each user agent, what it may do. It is a request, not protection: well-behaved robots follow it, ill-behaved ones may ignore it.

### What kinds of robots are there?

| Robot | Whose and what for | How it treats robots.txt |
|---|---|---|
| GPTBot | OpenAI, collecting texts for training | obeys |
| OAI-SearchBot | OpenAI, ChatGPT search; it decides whether a site gets into answers | obeys |
| ChatGPT-User | OpenAI, a visit on a user's request | OpenAI's documentation says robots.txt rules may not apply |
| ClaudeBot | Anthropic, collecting for training | obeys |
| Claude-SearchBot | Anthropic, Claude search | obeys |
| Claude-User | Anthropic, a visit on a user's request | obeys |
| PerplexityBot | Perplexity, the search index; not used to train base models | obeys |
| Perplexity-User | Perplexity, a visit on a user's request | according to the documentation, usually ignores it |
| Google-Extended | Google, permission to train Gemini on your pages and use them in Gemini answers | not a robot, only a name for rules in robots.txt: the pages are crawled by Google's ordinary robot |
| Applebot-Extended | Apple, permission to train Apple's models | also only a name for rules; it does not affect search through Applebot |
| Meta-ExternalAgent | Meta, collecting texts to train models and for Meta's products | obeys |
| Meta-ExternalFetcher | Meta, a visit on a user's request | according to the documentation, may bypass robots.txt |
| CCBot | Common Crawl, an open archive of the internet that many models are trained on | obeys |

The table shows the main thing: robots that act "on a user's request" mostly do not have to obey robots.txt. The companies' logic is clear: the person asked to open the page themselves, which is almost the same as opening it in a browser. To close a page to such a visit, you need a password login or protection on the server, not robots.txt.

A few clarifications:

- **Google-Extended** does not affect ordinary search. Google writes that this token "does not affect inclusion in Google Search and is not a ranking signal". AI Overviews are part of Google Search, so it does not remove a site from them either. AI summaries can be limited only with the same rules as ordinary snippets (`nosnippet`, `max-snippet`, `noindex`), and those would also cut the site down in ordinary results.
- **Anthropic** honours the `Crawl-delay` directive (a pause between a robot's requests) and does not bypass captchas.
- **A common ban on training.** The IETF working group, which sets internet standards, is preparing a common way to say "do not train on my texts" (AIPREF). Anthropic has understood its draft since 30 October 2026, and Google [promises](https://developers.google.com/search/blog/2026/10/ai-training-control) support in 2027. Until the standard is adopted, it is safer to list training robots by name, as in the example below.
- **Perplexity** ended up at the centre of a scandal in August 2025: Cloudflare accused it of hidden robots that passed themselves off as ordinary browsers, and removed Perplexity from its list of verified bots. The lesson is a general one: a robot's name is easy to fake, so reliable checking relies on addresses and signatures (more on them below).

### How do you allow AI search but block training in robots.txt?

The example below splits robots into three groups. AI search robots we let in. Collectors of training texts we keep out. Everyone else, including Googlebot and Bingbot, we let in. The block for everyone also gets a `Content-Signal` line, covered right after the example.

```text title="robots.txt"
# AI search robots: let them find the site and link to it in answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Robots that collect text to train models: keep them out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://example.com/sitemap.xml
```

How to read this. Several `User-agent` lines in a row form one group, and the rule below them applies to all the robots listed. A robot looks for the group where its name is mentioned exactly, and if there is none, it takes the `*` group. I checked this file with a standard Python robots.txt parser. AI search robots get access, training robots do not, and Googlebot gets through by the `*` group. The `Content-Signal` line does not break the parser: it skips lines it does not know.

Two warnings. First: rules for GPTBot do not apply to ChatGPT-User and OAI-SearchBot, which are separate robots. Second: if you close OAI-SearchBot, the site disappears from ChatGPT search. OpenAI picks up changes to robots.txt in about a day.

### What are Content Signals?

**Content Signals** is an extension of robots.txt that Cloudflare proposed in September 2025. It lists not robots but what the content may be used for. There are three signals:

- `search` — build a search index and show links and snippets;
- `ai-input` — give the content to an AI in response to a query, that is, use it as a source for an answer;
- `ai-train` — train and fine-tune models.

The line `Content-Signal: search=yes, ai-input=yes, ai-train=no` means "fine for search and answers, not for training". So far there are few such sites: according to Cloudflare's own data for April 2026, about 4% of the two hundred thousand most visited domains had declared signals.

A signal can also carry legal weight. In the EU, the copyright directive (Article 4) lets an owner forbid automated analysis of their texts if the ban is written in a machine-readable form, and such a line may qualify. But that is a question for lawyers. Technically a signal is still a request: a robot may ignore it.

### Why is robots.txt not enough, and what else can get in the way?

It happens that robots.txt is written correctly and a robot still cannot read the site. The cause is bot protection. It is set up by CDNs (networks of servers that deliver a site to visitors and also protect it) such as Cloudflare and Akamai, by hosting providers and by security plugins. If the protection is configured harshly, it cannot tell a harmful bot from OAI-SearchBot and turns both away. You may not even know: robots.txt allows everything, yet the robot gets a refusal or a "prove you are not a robot" page.

How to check:

1. Open the server's access log or your protection service's dashboard and find the robot you need: does it come, and what response does it get. Responses 403 ("forbidden") and 429 ("too many requests") to a search robot are a warning sign.
2. Do not trust the name alone: anyone can introduce themselves under someone else's name. OpenAI and Perplexity publish lists of their robots' addresses, for example `openai.com/searchbot.json` and `perplexity.ai/perplexitybot.json`. With them you can confirm that a visitor is genuine and add these addresses to the allowed ones in your protection rules.
3. Do not forget the "user-initiated" robots. If you want the assistant to be able to open a page whose link a person has sent, the protection has to let them in too. As we saw, they often ignore robots.txt, so the protection settings decide everything.

What to choose depends on the site. A store or blog that lives on visibility benefits from letting in search robots and user-initiated robots; each owner decides about training robots. A site with paid or closed content needs the opposite, and there you need a password login, not a request in robots.txt. Where all these switches are at Cloudflare and other providers is in the next section.

## What do Cloudflare and other providers let you configure?

Between AI robots and your site there is almost always someone else: a CDN, a hosting provider or a site builder. Their settings act before robots.txt does. They can let a robot in or turn it away at the door, write robots.txt for you, change a page on its way to the reader and show who came. One forgotten switch is enough for a site to drop out of AI answers, and nothing on the site itself will show it.

Below is the state of things on 12 November 2026. Cloudflare comes first and in the most detail: this site runs on it, and I went through every relevant screen of its dashboard on the free plan. For other CDNs, clouds and site builders I rely on their documentation.

### Where are the AI settings in Cloudflare?

A domain's dashboard has three places for them:

- **AI Crawl Control**, a separate menu item: the list of AI robots with a block switch for each, statistics of their visits and the state of robots.txt;
- **Agent Readiness**, the next menu item: a check of how ready the site is for agents, plus Markdown for Agents and WebMCP;
- **Security → Settings**, the "Bot traffic" group: the main access policies, Bot Fight Mode and AI Labyrinth.

Almost everything here works on the free plan; the exceptions are named where they come up.

### Who does Cloudflare let in?

The main switch is **AI Crawler and Agent access policies** (Security → Settings, filter "Bot traffic"). In older guides it is called "Configure AI bot policies" or "Block AI bots". Robots in it are split into three groups by what they do:

- **Search** — robots that index a site so that they can later answer questions about it;
- **Training** — collectors of texts for training models, including mixed ones that serve both search and training;
- **Agents** — those that act on a person's behalf right now: they open a page from a chat or work in a browser.

For each group you choose: allow, block everywhere, or block only on pages with ads. Training has one more option, **Disallow**. It works only together with robots.txt synchronization (Bot Preference Sync, covered just below), so turn them on together. Then Cloudflare writes the ban on training into robots.txt and also blocks training robots on its own side, before they reach the site. Mixed robots such as Googlebot, Bingbot and Applebot are not affected by this blocking, so the site does not drop out of ordinary search. One caveat from [Cloudflare's announcement](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/): Bing does not yet understand this ban in robots.txt, and Microsoft promises support by early 2027. Until then, you can close training to Bing with its own `NOARCHIVE` meta tag.

The Agents group has a separate list of **personal agents**: assistants that work for one particular person. They are Claude Duet (Anthropic), dots (OpenAI), Gemini Spark (Google) and Muse (Meta). By default they follow the Agents policy, but each can be given its own. Cloudflare recommends letting them in.

Check this switch even if you have never touched it. New domains start with everything allowed. But if you said during setup that the site makes money from ads, training will be forbidden and agents blocked on pages with ads. And for those who had "Block AI bots" on, Cloudflare itself changed the settings in September 2026: search allowed, training forbidden, agents blocked on pages with ads.

Next to the policies there are three more settings that decide who gets in:

- **Bot Fight Mode** challenges every suspicious bot on the domain, and on the free plan you cannot make an exception for it. If you want AI agents and MCP clients (they are covered in [the article on MCP and WebMCP](/en/mcp-webmcp/)) to reach you, keep it off. Paid plans have Super Bot Fight Mode instead: there you choose an action for each class of bots and can make exceptions.
- **Browser Integrity Check** turns away requests with missing or suspicious headers. On this site it is on and does not get in the way: every AI robot I tried, and even a request with no user agent at all, got the page.
- **AI Labyrinth** adds to pages links with a nofollow mark ("do not follow") that people cannot see and that lead into a maze of generated pages. Honest robots do not follow such links, so the trap only catches those that ignore the rules.

The list of individual robots is in **AI Crawl Control → Security**: each has its company, its category and a block switch. Besides "allow" and "block" there is a third action, **Charge**, a fee for every page (Pay per crawl). A robot that has not paid gets the response 402 ("payment required") with a price. For now this is a closed beta.

How does Cloudflare tell a real robot from one that has taken its name? By the addresses the robot comes from and by **Web Bot Auth**, a digital signature on requests. The agent signs every request with its private key and publishes the public key on its own site. Cloudflare checks the signature against that key. A name is easy to fake, but a signature cannot be forged without the private key. Since July 2026 Cloudflare counts agents with a valid signature as verified bots. Technically this is the RFC 9421 standard with Ed25519 keys.

### What does Cloudflare write to robots.txt?

**Bot Preference Sync** (formerly "Managed robots.txt") carries your policies over into robots.txt. It puts its own block at the top of the file: a ban for the robots of the blocked groups and a `Content-Signal` line with an explanation. Your own lines stay below, and if there was no robots.txt, it creates one. One detail matters: the block appears only if at least one policy forbids something. If everything is allowed, the file stays as it is, with no signals, and you have to write the `Content-Signal` line yourself. New domains have the sync turned on; on older ones you turn it on in the same window as the policies.

You can check the result on the **Signals** tab in AI Crawl Control. It shows whether robots.txt answers, whether it has Content Signals, and which AI robots went where robots.txt does not let them.

### What does Cloudflare show about AI traffic?

- **Overview and Metrics** in AI Crawl Control: how many requests each robot and each company made, what responses they got and which pages they requested most.
- **Optimization**: how many times pages were fetched for answers, what assistants tried to open but could not, and whose assistants come most often.
- **Security → Analytics → Events**: who was blocked or challenged and by which rule. On the free plan this is a sample, but it is enough to find a robot that is being turned away.

For an example, here is a week of this site in November 2026. AI robots made about 4,000 requests: about 2,500 got a page, 844 a redirect, 416 "not found" and exactly one a refusal. The "not found" answers were mostly for guessed addresses such as `/sitemap.xml` and for requests for `/.env` files with passwords, most likely from vulnerability scanners posing as AI robots. By the number of fetches for answers Perplexity came first, followed by OpenAI, Apple and Anthropic.

### How does Cloudflare help agents read and use the site?

- **Markdown for Agents** (Pro plans and above): Cloudflare converts a page into markdown on the fly for agents that ask for it. More in the section on llms.txt.
- **Agent Readiness → Diagnostics** checks the site against about twenty signs, from robots.txt and the sitemap to the MCP server card and WebMCP. For each failed check the dashboard explains what is wrong and offers a ready task for a programmer's AI assistant. Not everything on the list is needed by every site, and this check does not look at llms.txt.
- **WebMCP** (beta): Cloudflare itself adds ready-made tools for agents in the browser to your pages, for example it opens the tools of your MCP server to them. What that means is explained in [the article on MCP and WebMCP](/en/mcp-webmcp/). If the site registers such tools itself, as this one does, do not turn it on: the agent would see them twice.
- **AI Search**: Cloudflare indexes the site and provides search over it together with its own MCP server, a ready-made replacement for a server of your own.
- **Crawler Hints** (Caching → Configuration, free): when a page changes, Cloudflare itself tells Bing, Yandex and other search engines that support the IndexNow protocol. Bing learns about new pages sooner, and Copilot answers from Bing's index.

### Which Cloudflare settings quietly spoil pages for AI?

Some settings block no one but change the page on its way.

- **Email Address Obfuscation** (Security → Settings, the "Client side abuse" group) hides email addresses from spammers. Cloudflare replaces every address in the HTML with the placeholder "[email protected]", and a script in the browser later puts the address back. A robot does not run scripts and sees the placeholder. The trap is that the setting finds addresses in code examples too. It was on at this site, and a check on 12 November 2026 found 32 replacements in each language, 30 of them in code examples. A model reading such a page got broken code. The setting has been turned off. If your site has no real addresses to hide, turn it off too; if it does, exclude the pages with code using a separate rule (a configuration rule).
- **Rocket Loader** changes the order in which scripts load. If the text of a page is visible without JavaScript, robots lose nothing; if scripts draw the content, something may break.
- **Under Attack mode** shows a challenge to everyone, robots included. Turn it on only during a real attack, and do not forget to turn it off.

### What do other CDNs and clouds offer?

The principle is the same everywhere: a provider can stop an AI robot before it gets to robots.txt, and some switches block more than their names promise.

- **Vercel.** In Security → Firewall → Rules there is a set of rules called **AI Bots** for AI crawlers and AI assistants, with the action "log" or "deny"; it works only once you turn it on. Next to it is **Bot Protection**, a JavaScript check for everything that is not a browser. It lets verified bots through, but a deny in AI Bots is stronger.
- **Netlify.** There are no ready-made AI rules: robots are blocked by name with the **User Agent Blocker** extension or with your own function on the CDN side. Netlify itself warns that a robot can call itself by someone else's name.
- **Fastly.** **AI Bot Management** tells AI crawlers from on-request robots, verified from suspected, and lets you block, allow or intercept them.
- **Akamai.** **Bot & Agent Control** divides AI robots into training, search and agents, with its own policy for each group. Web Bot Auth signatures are checked.
- **AWS.** The biggest trap is here. If you turn on the Bot Control rule group in AWS WAF (Amazon's web application firewall) and do not switch its **CategoryAI** rule to Count ("only count"), it blocks all AI robots, even verified ones. Exceptions are made with a separate rule after Bot Control, and an agent with a valid Web Bot Auth signature passes these rules.
- **Google Cloud.** Cloud Armor has no rules for AI robots; agents are handled by Cloud Fraud Defense, the successor to reCAPTCHA. Firebase Hosting and Cloud CDN have nothing of the kind.
- **Azure.** The Bot Manager in the WAF for Front Door (Premium plan) knows only good, bad and unknown bots; there is no separate category for AI. By default it only logs unknown ones. But if unknown bots are given a JavaScript check, AI agents will be stopped by it.

### And if the site is built with a site builder?

There is less control here, but the switches are easier to find.

- **WordPress.com.** "Prevent third-party sharing" in Settings → Reading adds known AI robots to the ban in robots.txt. It is about training and does not touch ordinary search. Do not confuse it with "Discourage search engines from indexing this site": that one hides the site from ordinary search too. On self-hosted WordPress, robots.txt and llms.txt are handled by plugins.
- **Squarespace.** "Block known artificial intelligence crawlers" in Settings → Crawlers is off by default. It closes 26 AI robots, among them GPTBot, ClaudeBot and Google-Extended, and does not touch ordinary search. The AI Visibility tool shows how often assistants mention the site.
- **Wix.** There is no separate switch: robots are closed in the robots.txt editor (SEO & GEO → Tools and settings). llms.txt is created automatically on paid plans with a connected domain, and, according to Wix, every site has its own MCP server for agents.
- **Webflow.** Site settings → SEO → Indexing has switches for search robots and for AI robots, both allowed by default. There is also a robots.txt editor whose documentation describes the `Content-Signal` line. llms.txt is uploaded as a file.
- **Shopify.** robots.txt is edited through the `robots.txt.liquid` template, and there is no separate AI switch. Something else matters more: **Agentic Storefronts**. Shopify itself sends the catalog to ChatGPT, Perplexity and Copilot, and each channel is turned on separately. Banning AI robots in robots.txt does not stop that feed. Shopify storefronts also have WebMCP tools.
- **Tilda.** robots.txt cannot be edited. llms.txt and llms-full.txt can be uploaded in the site's SEO settings.

### What should you check if the site is on AWS?

On AWS a site usually sits behind CloudFront, Amazon's CDN, with the files in an S3 bucket or on Amplify Hosting. AWS itself does not block robots, but it is easy to block them here by mistake. What to check:

1. **The web application firewall.** If the Bot Control rule group is enabled in AWS WAF, its CategoryAI rule blocks every AI robot except agents with a valid Web Bot Auth signature (more on this above, in the list of providers). Switch that rule to Count, or add your own rule after Bot Control that lets the robots you want through by their labels.
2. **Pages in folders.** CloudFront fills in `index.html` by itself only for the root of the site. For an address like `/blog/post/`, a private S3 bucket answers with error 403, and the robot sees that instead of the article. You need a small CloudFront Function that appends `index.html` to such addresses.
3. **An honest 404.** Single-page apps are often advised to answer any 403 or 404 with the home page and code 200. That is worse for robots: every address that does not exist looks like a real page with the same text. If your site is not a single-page app, serve a real 404 page.
4. **Service files.** robots.txt, the sitemap and llms.txt sit in the root and are served as text or XML. After you change robots.txt, clear the CloudFront cache (an invalidation), or robots will keep getting the old version for a long time.
5. **Headers.** Headers such as `Link` (a pointer to llms.txt from the home page) or `X-Robots-Tag` are added in CloudFront by a response headers policy.
6. **Who comes.** The overall picture of AI robots is in the AI Activity Dashboard; the details are in the CloudFront and WAF logs.

Everything else, from text in HTML to markup and llms.txt, does not depend on the hosting and is done as described in this article.

### And if the site is built with Astro or a similar generator?

Static site generators such as Astro, Hugo, Eleventy or Next.js in static export mode already serve ready-made HTML, and that is the main thing. The rest is set up once in the project. Here is how it works in Astro, which this site is built on:

1. **Text in HTML.** Astro builds pages into ready-made HTML. The exception is components with `client:only`: they are not rendered on the server at all, and a robot will not see their text. Do not keep anything important in them.
2. **The site address and the sitemap.** Set `site` in the configuration: full addresses, canonical links and the sitemap are built from it. The `@astrojs/sitemap` integration puts the sitemap at `/sitemap-index.xml`, not `/sitemap.xml`, so list it in robots.txt.
3. **robots.txt.** Put it in the `public/` folder or generate it during the build; it holds the `Sitemap` line and, if you need one, `Content-Signal`.
4. **Languages.** Each language gets its own address (Astro has built-in routing by language), and the shared page template outputs links to the versions in other languages (`hreflang`).
5. **Markup and dates.** It is convenient to assemble JSON-LD in the shared template from the article's frontmatter: title, author, publication and modification dates. Then the markup always matches the page.
6. **llms.txt and markdown copies.** You can build them from the same collection of articles with small endpoints such as `src/pages/llms.txt.ts`: during the build Astro turns them into ordinary files.
7. **Redirects and headers belong to the hosting.** In a static build, redirects from Astro's configuration become pages with an instant jump, not a real 301 response. A real 301 and headers such as `Link` are set by the hosting's files: `_redirects` and `_headers` on Cloudflare Pages and Netlify, `vercel.json` on Vercel. This site does the same: when this article changed its address, the old one was closed with a line in `_redirects`.

### What should you check first?

1. **Who is turned away.** Open your provider's statistics on AI robots and look at the responses search robots and agents get. Responses 403, 429 or a challenge page mean that some switch is blocking them.
2. **Every place where a robot can be stopped:** the provider's AI policy, bot protection, the web application firewall (WAF) rules, the site builder's switch and robots.txt itself. They work independently, and any one of them can stop a robot.
3. **What changes a page on its way:** email obfuscation, script loaders, challenge pages.
4. **Your wishes, declared explicitly:** robots.txt with a `Content-Signal` line, written by hand or by the provider.
5. **The same again in a month.** Providers change their defaults; Cloudflare, for example, did in September 2026.

## How do you make content understandable to AI?

There is no magic here: an assistant reads a page the way a person does, only without the pictures, and pulls facts out of it. Anything that gets in the way of pulling out a fact lowers the chance of being cited.

### Is server-side rendering needed?

Yes. Many sites today reach the browser almost empty, and JavaScript draws all the text on the page once it is there. Google's robot can run this code, but most AI robots cannot: that is what a study by Vercel and MERJ (December 2024) showed. Something may have changed since, but the advice is still safe and cheap: let the text of the page arrive as ready-made HTML, before any script runs. This is what is called server-side rendering.

Checking is easy. Download the page with the `curl` command, or open "View page source" in the browser, and look for your text there. If it is not there, the assistant most likely will not see it either.

A blog on Astro, like this one, serves ready-made HTML anyway. On an online store it is worth checking the product card specifically: the price, availability and description must be in the source HTML, not loaded by a separate request.

### How do you write to get cited?

Rules that follow from how an assistant works with text:

1. **The answer first.** If a heading is a question, the first sentence under it should answer it. Details and caveats come later.
2. **One section, one topic.** An assistant often takes not the whole page but a fragment. The fragment must make sense on its own: not "as we said above" but with the name of the subject.
3. **Specifics.** Numbers, dates, versions, names, definitions. A vague "many believe" does not get cited.
4. **Tables and lists for comparisons.** They are easy to parse both for an assistant and for a person.
5. **Freshness.** A page should have a visible date, and the markup (more on it below) should have the publication and edit dates, `datePublished` and `dateModified`. When a question is about the current state of things, assistants look for fresh material.

Google adds warnings to this. In the same guide it advises against splitting text into small "chunks" specifically for AI, against making separate pages for robots, against chasing llms.txt, and against ordering fake mentions on third-party sites. What matters most is original text for people.

### What does "the answer first" look like in an example?

Take a store's page about returning goods. The weak version begins like this: "We value our customers very much and try to make shopping as comfortable as possible. Our return policy has been designed with…" The actual deadline appears only three paragraphs later. The assistant has to look for the fact among the polite words, and if it takes only the beginning of the page, it will not find it at all.

The strong version: "You can return a product within 30 days of receiving it. We pay for the return shipping. We refund the money to your card within 5 working days after we receive the parcel." Three short sentences, each can be cited separately, and each has a number. After that you can write the details: how to start a return, which goods are not accepted.

The same goes for a blog article. The heading "What is MCP?" and right under it: "MCP is an open protocol through which an AI assistant connects to external services". The definition stands in the first line, not on the third screen.

### How do you organize headings and pages?

- Phrase headings the way a person phrases a question: "How do I return a product?", "How much is delivery to Germany?". The assistant turns the user's question into a search query, and a question heading matches it better than "Delivery information".
- One page answers one question. A huge "Everything about our store" page is worse than separate pages on delivery, returns and payment.
- Every page has its own address and its own heading. Parts of a site that load without changing the address are not seen by the assistant as separate pages.
- Link pages by meaning: a product to the delivery page, an article to a related article. That way both people and robots find related facts.
- Use tables for comparisons and specifications. If you are comparing plans, models or versions, a table is clearer than a paragraph.

### What did the GEO research paper show?

The only work I know of that measured which techniques raise citation rates is the paper "GEO: Generative Engine Optimization" (Aggarwal et al., [arXiv 2311.09735](https://arxiv.org/abs/2311.09735), KDD 2024 conference). The authors took a set of questions, ran them through a generative engine and checked how much more prominent a source becomes in the answer if its text is rewritten in different ways. Prominence was counted in conditional points; unchanged text scored about 19.5. The main points:

- the best techniques raised prominence by about 40%;
- the most useful were quotations from authoritative sources (about 27.8 points), statistics (25.9) and links to sources (24.9);
- keyword stuffing, familiar from old SEO, lowered prominence on the main metric, to 17.8 points, and on Perplexity by about 10%;
- links to sources helped sites that sat fifth in the results especially: their prominence grew by 115.1%. But for sites in first position they did harm: minus 30.3% on average;
- the effect depends on the topic: what works for legal questions does not necessarily work for questions about history.

The caveat: this is a laboratory experiment from 2023 on specific engines and questions, not a measurement of today's ChatGPT. But the conclusion agrees with common sense: a text with quotations, numbers and sources is more useful to the reader and more noticeable to an assistant, while mechanically cramming in keywords does harm.

### What else matters: entities and languages?

An **entity** is a specific object: a person, a company, a product. It matters for an assistant to understand that "Roman Kryvolapov" on the site, in the LinkedIn profile and on GitHub is the same person. Three things help: the same spelling of the name everywhere, an About page with a clear description, and a `sameAs` field in the markup (an example is below) with links to profiles.

**Languages.** If a site is multilingual, each language should have its own address, not be switched on the same one. This site, for example, has addresses like `/en/…`, `/uk/…`, `/ru/…` and `/bg/…`. Then a question in Russian leads the assistant to the Russian page, and a question in English to the English one.

## What schema.org markup do you need?

**Schema.org** is a vocabulary that lets a page tell machines about itself: "this is an article, its author is so-and-so, it was published then" or "this is a product, it costs this much and is in stock". Markup is most often written in the **JSON-LD** format, which is a block of JSON inside a `script` tag on the page.

Markup is not required to get into Google's AI answers. But it is needed for rich search results, the cards with a price, a rating or a date. Microsoft, according to secondary sources, considers that markup helps. One condition Google repeats constantly: **the markup must match what a person sees on the page**. A price of 10 euros in the markup and 12 on the page is an error that undermines trust in both the markup and the site.

### What should you mark up in a blog?

The minimum is an article and an author, linked by `@id`. All the facts about the author sit in one place, and every article refers to it:

```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://example.com/#person",
      "name": "Jane Doe",
      "url": "https://example.com/about/",
      "sameAs": [
        "https://github.com/example",
        "https://www.linkedin.com/in/example"
      ]
    },
    {
      "@type": "BlogPosting",
      "headline": "How to brew sencha",
      "datePublished": "2026-11-11",
      "dateModified": "2026-11-11",
      "author": { "@id": "https://example.com/#person" },
      "inLanguage": "en"
    }
  ]
}
```

This is an abbreviated form. The `sameAs` field lists profiles that confirm this is one person (that same "entity"). Remember that `datePublished` must match the date the reader sees, and `dateModified` must match a real edit.

### What should you mark up in an online store?

For a store, markup matters more than for a blog: Google Shopping and agentic purchases are built on it. The main type is `Product` with a nested `Offer` (an offer: price, currency, availability). Mark up the rating, delivery and return terms separately: these are what assistants are asked about most often.

The example below is a tea product card. Three places in it deserve attention: the `sku` and `gtin13` identifiers (they are used to match the product against catalogs), the `shippingDetails` block (the delivery cost and time) and the `hasMerchantReturnPolicy` block (the return period and method).

```json
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Sencha green tea, 100 g",
  "sku": "TEA-001",
  "gtin13": "4006381333931",
  "brand": { "@type": "Brand", "name": "Example Tea" },
  "image": ["https://shop.example.com/img/tea-001.jpg"],
  "description": "Japanese steamed green tea with a fresh grassy taste. Loose leaf, 100 g.",
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": 4.7,
    "reviewCount": 128
  },
  "offers": {
    "@type": "Offer",
    "url": "https://shop.example.com/p/tea-001",
    "price": 12.5,
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition",
    "shippingDetails": {
      "@type": "OfferShippingDetails",
      "shippingRate": { "@type": "MonetaryAmount", "value": 4.9, "currency": "EUR" },
      "shippingDestination": { "@type": "DefinedRegion", "addressCountry": "DE" },
      "deliveryTime": {
        "@type": "ShippingDeliveryTime",
        "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
        "transitTime": { "@type": "QuantitativeValue", "minValue": 1, "maxValue": 3, "unitCode": "DAY" }
      }
    },
    "hasMerchantReturnPolicy": {
      "@type": "MerchantReturnPolicy",
      "applicableCountry": "DE",
      "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
      "merchantReturnDays": 30,
      "returnMethod": "https://schema.org/ReturnByMail",
      "returnFees": "https://schema.org/FreeReturn"
    }
  }
}
```

How to read this. `handlingTime` is how long it takes to pack the product before shipping (from zero to one day), `transitTime` is how long it travels (from one to three days). Together they give the delivery time. `returnPolicyCategory` ending in `FiniteReturnWindow` means "return within a limited period", and the period itself is in `merchantReturnDays`. The price, availability and deadlines in the markup must match the page, so it is better to generate them from the same data as the visible text than to maintain them by hand in two places.

If a store has a Google Merchant Center account, it is Google's main source of data. You upload a **feed** there: a file with the whole catalog, with products, prices and availability. Shopping and agentic purchases are built from it. The markup on the pages must not contradict the feed.

## llms.txt and markdown copies: do they work?

**llms.txt** is an idea from Jeremy Howard, co-founder of fast.ai and Answer.AI, from September 2024 (the second version came out in August 2026). He proposed putting a markdown file in the root of a site with a short description of the site and links to the most important things. Markdown is plain text with light markup: `#` for headings, `-` for lists. The idea is to give a model a "table of contents" and spare it from parsing HTML. Details are at [llmstxt.org](https://llmstxt.org).

The honest status is this. It is not a standard, it is a proposal. Google says outright that Google Search does not use it, and advises not to spend time on it for ranking. None of the major vendors has confirmed that it takes llms.txt into account when choosing sources. Cloudflare's "Agent Readiness" check does not look at it either.

So why is it on this site? Because it costs nothing and may be useful to quite specific readers: coding agents that read documentation, and people who feed a site to an assistant as a single file. Not as a "ranking factor" but as a convenient format. Here are a few lines from this site's file (long lines shortened):

```markdown title="llms.txt"
# Roman Kryvolapov

> Engineering blog: Java, Kotlin, Android, Spring, microservices, running large language models locally.

## MCP server

- [MCP endpoint](https://mcp.romankryvolapov.com/): Streamable HTTP, no authentication. Tools: `search` — full-text search over every article; …

## English

- [Claude Code Hooks](https://romankryvolapov.com/en/claude-code-hooks.md): A walkthrough of Claude Code hooks: every event, exit codes and JSON decisions, …
```

Each link leads to the markdown version of a page. Every article on the site has such a copy at the same address with `.md` added, for example `/ru/ktor-manual.md`. The page points to the copy in its `<head>` with the line `<link rel="alternate" type="text/markdown">`. And the copy is served with a `Link: <page address>; rel="canonical"` header, that is, it names the page itself as the main version, so that a search engine does not count the copy as a duplicate. There is also `llms-full.txt`: all the articles in all languages in one file, about 15 MB.

A similar idea from Cloudflare is **Markdown for Agents** (Pro plans and above). If an agent asks for a page with the header `Accept: text/markdown`, Cloudflare converts the HTML into markdown on the fly. According to Cloudflare, Claude Code and OpenCode, for example, send such a header. This benefits an agent: markdown is several times shorter than HTML, and so cheaper. One subtlety: if the site has not set its own `Content-Signal` header, Cloudflare adds to such a response a permission for everything, training included. On the free plan you can do the same yourself: a small function on the CDN side answers such a request with the page's ready `.md` copy.

The bottom line: llms.txt and markdown copies are a convenience for agents, not a way to climb the results. Make them if it is cheap, and do not expect a miracle.

## What should a store do, step by step?

The list is ordered by return: the first items are needed by everyone, the last ones can be postponed.

1. **Open access to search robots.** Check the CDN and the web application firewall (WAF): they must not block `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot` and `bingbot`. To let in the real robots, use the published lists of their addresses (they are given in the section on bot protection).
2. **Decide about training.** If you do not want to give your texts away for training, add a block to robots.txt like the one in the example above.
3. **Check that the product card gives its text without JavaScript.** Price, availability, description, delivery and returns must be in the source HTML.
4. **Mark up the products:** `Product`, `Offer`, `AggregateRating`, `OfferShippingDetails`, `MerchantReturnPolicy`. The values must match the page.
5. **Set up Google Merchant Center and a Business Profile on Google.** The feed is configured once and works for several channels at once.
6. **Set up measurement** (the next section) and come back to the numbers in a month.
7. **If you want agents to buy from you,** the next steps are an application to ChatGPT, your platform's ready-made solutions, an MCP server and WebMCP. They are all covered in [the article on MCP and WebMCP](/en/mcp-webmcp/).

### What mistakes do stores make most often?

- **Price and availability are loaded by a script.** The source HTML is empty, and a robot sees a card with no price. The cure is server-side rendering, or at least markup with the key data.
- **The markup diverges from the page.** The markup says "in stock", the page says "out of stock". The data must come from one source.
- **No delivery times or return terms**, neither in the text nor in the markup. These are the questions buyers ask an assistant most often, and a store without an answer loses to those that have one.
- **Bot protection blocks everyone.** AI search robots get a challenge or a refusal, and the store is not in the answers.
- **Identical descriptions on dozens of products.** A description copied from the manufacturer gives the assistant no reason to choose your page.
- **No identifiers.** SKUs and barcodes (`sku`, `gtin`) let a product be matched against catalogs; without them a product is harder to find and compare.
- **Reviews are trusted without a filter.** Any user text in a review can be an attempt to inject an instruction. A review is data, not a command.

### And for a blog or a company website?

The list is shorter here:

1. Indexing: a sitemap, Search Console and Bing Webmaster Tools.
2. The text of pages arrives as ready-made HTML.
3. AI search robots are not blocked, neither in robots.txt nor at the provider.
4. Articles have a visible publication date, and sections have the answer in the first line.
5. The text has quotations, numbers and links to primary sources.
6. The markup has the author (`Person`) with links to profiles (`sameAs`).
7. Measurement is set up (the next section is about it).

The `Content-Signal` line and markdown copies of pages are optional. But do not spend effort on llms.txt as a ranking factor.

## How do you know GEO is working?

A direct counter of citations exists so far only at Bing. Google shows impressions in its AI answers, and the rest has to be put together from indirect signs.

### What does GA4 show?

Google Analytics 4 (GA4), according to secondary sources, has had a separate "AI Assistants" traffic channel since May 2026. But it does not catch everything. Visits from Perplexity and Claude often land in "Referral" (visits from other sites), Gemini often arrives without a source address and lands in "Direct", and clicks from AI Overviews and AI Mode are counted as ordinary Google search (`google / organic`). It is more reliable to collect such traffic yourself: add a filter on the session source to a report with the condition "matches regular expression" and paste in the expression below. It catches visits from the main assistants:

```text
(www\.)?(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com|chat\.mistral\.ai|chat\.deepseek\.com|grok\.com|meta\.ai)
```

This is an estimate from below: visits without a source address will not be caught. ChatGPT, according to secondary sources, marks its own links itself (`utm_source=chatgpt.com`), so its traffic is the easiest to see.

### What do Search Console and Bing show?

Since June 2026, according to Google, Search Console has reports on impressions in generative features: AI Overviews, AI Mode and Discover. Only impressions are visible, with no clicks.

Bing has an **AI Performance** report in Webmaster Tools, still in public testing ([Bing blog](https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/)). It shows how many times you were cited, which pages, for which queries and on which days. There are no clicks there either, but it is the only open report that says directly: "you are being cited". It covers Copilot and AI summaries in Bing.

### What is visible in server logs?

Logs show what analytics does not: who comes and how many times. Robots do not run analytics scripts, but they do stay in the server's access log. This command counts visits by the main AI robots from a log in nginx or Apache format:

```bash
grep -oiE '(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-external[a-z]+|Applebot|bingbot|Googlebot)/' access.log |
  tr -d / | sort | uniq -c | sort -rn
```

What it does. `grep -oiE` pulls out of each line only the robot's name with a slash after it, and ignores letter case: Meta, for example, writes its robots' names in lower case. `tr -d /` removes the slash, `sort` and `uniq -c` count how many times each name occurred, and the last `sort -rn` puts the most frequent ones on top. On a test log with the strings that real robots introduce themselves with, the result was this:

```text
      2 meta-externalagent
      2 Googlebot
      2 GPTBot
      1 meta-externalfetcher
      1 bingbot
      1 PerplexityBot
      1 Perplexity-User
      1 OAI-SearchBot
      1 ClaudeBot
      1 Claude-User
      1 Claude-SearchBot
      1 ChatGPT-User
      1 Applebot
```

A name in a user agent can be faked, so for serious conclusions check the addresses against the published lists (they are covered in the section on bot protection). If a site is behind a CDN, its dashboard shows the same without any logs: at Cloudflare it is AI Crawl Control, at Vercel the AI Traffic report, at AWS the AI Activity Dashboard.

### Can you just ask the assistants themselves?

You can, but carefully. Ask ChatGPT, Gemini and Perplexity five to ten questions that your site should answer, and see whether they cite you. A single measurement says nothing: answers change from run to run. Repeat once a month and record the results. There are paid services for this too, but the principle is the same.

## What does not work?

A short list of misconceptions.

- **"I need llms.txt to rank in AI."** No. Google says outright that it does not use it, and no vendor has confirmed the opposite.
- **"I have to cut content into chunks and write separate pages for AI."** Google advises against doing this.
- **"If I block Google-Extended I will vanish from Google."** No, this ban does not affect ordinary search.
- **"I will stuff the text with keywords like before."** In the GEO research this did not help, and on the main metric it even lowered prominence.
- **"robots.txt will protect a page from all assistants."** User-initiated robots often ignore it. To truly close a page, you need a login.
- **"robots.txt allows everything, so every robot gets in."** A switch at the CDN or bot protection can turn a robot away whatever robots.txt says.
- **"I will add markup and the assistant will buy from me."** Markup is needed, but an agentic purchase also requires a feed, supported payment methods and, for a number of channels, an application and approval.

## How is this done on this site?

This site is a blog on Astro: static pages on Cloudflare Pages, four languages (English, Ukrainian, Russian and Bulgarian) and 48 articles in each. Everything described below was checked on the live site in November 2026.

- **Text copies.** Every article has a markdown copy, and so do the About and privacy pages: 200 in all. There is [`llms.txt`](https://romankryvolapov.com/llms.txt) with a table of contents and `llms-full.txt` with all the articles in one file.
- **Markup.** One JSON-LD graph: the author (`Person`), the site (`WebSite`), a `BlogPosting` with dates on an article, breadcrumbs, `ProfilePage` on "About me" and `ContactPage` on the contact page.
- **MCP server and WebMCP.** The site has its own MCP server and WebMCP tools for agents in the browser; how they work is described in [the article on MCP and WebMCP](/en/mcp-webmcp/).
- **Cloudflare**, free plan. All three access policies, search, training and agents, are set to "allow", and so are the personal agents, and no robot is blocked. Training is open on purpose: it benefits a blog if models know about it even without search. Bot Fight Mode is off: ChatGPT and Claude come to the MCP server from their own clouds, and it would cut them off. AI Labyrinth and Cloudflare's own WebMCP are off too, since the site has its own tools. In a week AI robots got exactly one refusal.
- **What changed after the check on 12 November.** robots.txt got the line `Content-Signal: search=yes, ai-input=yes, ai-train=yes`. The home pages got `Link` headers pointing to llms.txt and the MCP server card. The address `/sitemap.xml` now leads to the real sitemap. Email Address Obfuscation, the one that spoiled the code examples, is off, and Crawler Hints is on. Agent Readiness counts the MCP server card and WebMCP, but not markdown on request: the copies sit at their own addresses, and Markdown for Agents needs the Pro plan.

## Summary

The main points of the article.

1. GEO is not some separate magic: it is SEO plus managing the new robots. The foundation is the same: indexing, clear text, honest markup.
2. There are three kinds of robots: training, search and "on request". Search robots are worth letting in, each owner decides about training robots, and robots.txt will not protect you from "on request" robots.
3. The text must be in HTML without JavaScript, with the answer at the start, numbers, quotations and sources.
4. Measure: GA4, Search Console, Bing AI Performance and server logs.

Where to start: the three checks from the beginning of the article, namely the bot settings at your provider, the page text without JavaScript, and the new reports in Search Console and Bing. The rest can be done gradually.
