AI Crawlers (GPTBot, ClaudeBot, PerplexityBot): Allow or Block in robots.txt
AI crawlers in robots.txt: what GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do, how to separate training from search, with copy-ready rules and limits.
Key takeaways
AI Overviews are the AI-generated summaries Google shows on the results page, with links to sources. To be used as a supporting link, your content has to meet one technical condition: the page must be indexed and eligible to appear in Google Search with a snippet. There is no special markup, no special file and no "AI setting." The rest depends on how useful and original your content is.
This article explains, based on Google's documentation, what we know for sure, which controls you have, how to check in Search Console whether you appear, and what nobody can promise. For the big picture of visibility across AI engines, start with our GEO guide; for hands-on implementation, see our generative engine optimization service.
AI Overviews are automatically generated summaries that Google shows above or among the classic results for certain queries. AI Mode is a separate, conversational experience for questions that call for exploration, reasoning or complex comparisons. In both cases, the answer comes with links to web pages.
One important technical detail from Google's documentation: these features may use "query fan-out," meaning they run several related searches across subtopics and data sources to build the answer. The consequence for you: the links shown can come from a wider range of pages than the top positions of a single ordinary search. A page that answers one sub-question well can be used even if it does not dominate the main query.
Availability differs by country, language and query type. Check in your Search Console account and through real searches whether the features are active for your audience. If you sell to Romanian-speaking users, test in Romanian as well as in English, because the two can behave differently.
The documentation says a page must be indexed and eligible to be shown in Search with a snippet. Beyond that, there are no extra requirements and no special optimizations needed. In practice, you have three checks:
nosnippet or a very small max-snippet reduce or remove eligibility to be shown with text.Because eligibility depends on a page being crawled and indexed, sites whose content only exists after JavaScript runs, with no useful rendered HTML, risk problems; details in our JavaScript SEO guide.
Google's documentation stresses unique, helpful, people-first content rather than commodity content that repeats what is found everywhere. A few ideas worth turning into practice:
Google also says what is not required: writing in a special way just for generative AI, or breaking your content into tiny pieces. Creating several versions of the same page primarily to manipulate rankings, on the other hand, can fall under its spam policies.
Suppose someone asks: "How do I choose an accountant for a new LLC?" A system using query fan-out might break the question into subtopics, such as: what a subscription includes, how pricing works, what documents accountants ask for at the start, how to check credentials, which questions to ask in the first meeting. We do not know exactly which sub-queries Google runs for a specific search, so this is a logical illustration, not a description of the algorithm.
What this means for a page that wants to be useful in such situations:
Notice that nothing on the list is "AI-specific." These are writing practices that also help a person reading the page. That matches the message of Google's documentation: the same SEO fundamentals apply to the generative features as to the rest of Search.
A frequent risk is treating AI Overviews as a separate channel and abandoning classic optimization. The documentation says the opposite: SEO best practices remain relevant. Pages that Google understands and finds useful for ordinary search are the raw material for the generative features. So:
No. The documentation is explicit: there is no special schema.org markup for generative features, and you do not need to create new machine-readable files or "AI text files" (this includes llms.txt) to appear in these features. Structured data remains useful: it supports rich results and entity clarity. Our schema.org JSON-LD guide shows what is worth implementing. But do not let anyone present it as "the key to AI Overviews."
Google points to the same controls as for classic snippets:
| Control | What it does | Trade-off |
|---|---|---|
nosnippet | Prevents any text fragment from the page being shown | You lose the classic snippet too |
data-nosnippet | Excludes a portion of the page (HTML attribute) | Use it surgically, not on all content |
max-snippet:N | Limits fragment length | Too small may reduce eligibility |
noindex | Removes the page from Search | It disappears from results entirely |
| Google-Extended in robots.txt | Controls use for Gemini (training, grounding) | Does not affect Search and does not control AI Overviews |
The last row causes frequent confusion. Google's documentation describes Google-Extended as a robots.txt token for certain Google AI products and states that it does not affect inclusion in Search and is not a ranking signal. If you want to avoid being used in AI Overviews, it is not the right tool; for AI bots in general, read our guide to robots.txt and AI crawlers.
The documentation says traffic from these features is counted in the Performance report under the "Web" search type. In addition, Google announced a dedicated report, "Generative AI performance," which shows impressions in the generative features, grouped by page, country, date and device. According to the help page, Google had rolled the report out to all websites worldwide as of August 31, 2026. If it does not appear in your account or shows little data, the cause may simply be a low number of impressions in these features, or the fact that you excluded your site from them.
How to use it:
The general guide to the tool is Google Search Console for beginners. Be careful with third-party tools that claim to use Google's "internal" metrics: Google warns that no third-party tool has access to its internal ranking or AI systems. Measurement methods for all AI engines are in our guide to visibility in ChatGPT and AI search.
nosnippet without understanding the cost. You lose classic snippets and control over how you appear.AI Overviews do not require a new handbook, just SEO done carefully: indexing, useful and original content, snippet eligibility. As next steps, check the 20 pages, rewrite the openings, clean up overlaps and open the Search Console report. If you want help, our GEO service page describes how we work.
Frequently asked questions
They are two different features in Google Search, but the documentation treats them together for eligibility. AI Overviews appear as a summary on the results page; AI Mode is a broader conversational experience suited to complex questions. Both show links to sources.
Indirectly, yes. Use nosnippet, data-nosnippet, max-snippet or noindex, which limit how your content is shown in Search. The cost is that you also limit classic snippets. Blocking Google-Extended does not have this effect, because that token concerns other products.
No. Google says there is no special schema.org markup for its generative features. Structured data remains useful for rich results and entity clarity, but it is not a condition for appearing in AI Overviews.
Yes. The feature composes its answer from several related searches and may choose links that differ from the top classic positions. A strong ranking does not guarantee a citation, although well-ranked pages generally have more chances of being considered.
In Search Console, traffic and impressions from these features are part of the performance report, and Google also introduced a dedicated performance report for generative AI features. Check whether it is available in your account and compare pages, countries and periods.
Related service
Keep reading
AI crawlers in robots.txt: what GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do, how to separate training from search, with copy-ready rules and limits.
llms.txt explained without hype: format, a full example, what Google says, what is known about its effect, and when it is or is not worth publishing.
Generative engine optimization explained honestly: what works, what Google says, which tactics are myths, and a 30-day plan to get cited in AI answers.
Send us your website address and we’ll reply with a free initial analysis and a concrete SEO strategy — no strings attached.