Is your robots.txt blocking AI crawlers? A 10-minute audit
If AI assistants never name your product, the first thing to rule out is boring: can their crawlers even read your site? It takes about ten minutes.
By the Citeably team · · · 4 min read
1. Open your robots.txt
Go to yourdomain.com/robots.txt. Look for Disallow rules aimed at the user agents below, or a blanket “User-agent: *” with “Disallow: /”. Some of these are training crawlers and some are search or fetch tools. Blocking training but allowing search is a legitimate choice. Blocking everything by accident is not.
- GPTBot and OAI-SearchBot (OpenAI)
- ChatGPT-User (fetches pages on behalf of a user)
- ClaudeBot and Claude-SearchBot (Anthropic)
- PerplexityBot
- Google-Extended (controls use for Google's AI models)
2. Allow what you mean to allow
Decide per bot. Many CDNs and security plugins add bot rules for you, so check them as well as the file. Crawler names change, so confirm them against each vendor's current documentation.
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / Sitemap: https://yourdomain.com/sitemap.xml
3. Check you are not behind a wall
A crawler has to be able to load the page.
- The page loads without a login, a cookie-banner wall or a CAPTCHA.
- Important text is in the HTML, not only drawn by JavaScript after load. View source and search for a sentence from your homepage.
4. Make the page say what you are
A crawler that can read you still has to understand you. One plain sentence near the top, saying what the product is, who it is for and how it differs from the alternatives, helps a person and a model.
5. Then measure
Fixing access does not guarantee you are named. Ask the assistants the questions your buyers ask, without your brand name, and see who comes up and why. Re-run the same questions a month later. Answers vary between runs, so only trust a clear, repeated change.
The free check at citeably.app does this step for you in about a minute, with no signup. We built it, so weigh that accordingly. You can do the same by hand with a list of eight to ten questions.
Frequently asked questions
Should I allow training crawlers?
That is your call. Search and fetch crawlers are the ones that can lead to your pages being cited in an answer; training crawlers affect what a model learns.
Will this get me named by AI?
It removes one common reason you might not be. It does not guarantee a mention.