Free tool
Free robots.txt generator for AI crawlers
If your robots.txt blocks AI crawlers, they cannot read your pages. This generates rules that explicitly allow the major ones, and you can keep any private areas blocked.
Last updated .
How to use it
- Enter your brand and website.
- Copy the rules into your existing robots.txt, keeping any private paths you block today.
- Re-check your site with the readiness checker.
Questions
Should I allow every AI crawler?
That is your decision. Allowing the search crawlers is what lets assistants cite you. Some crawlers are for model training, and you can treat those separately.
Then see what AI actually says about your brand
The files above make your site easier to read. The free check shows whether ChatGPT, Gemini and Perplexity name you when buyers ask.
Run the free AI visibility check →More free tools: llms.txt generator, Organization and software schema generator, FAQ schema generator, AI site readiness checker, AI share of voice calculator.