Robots.txt generator.
Build a correct robots.txt file in a minute. Start from a preset, add the folders to keep crawlers out of, add your sitemap, and decide separately whether AI companies may use your site for training and whether AI search tools may read and cite it.
- Free, no sign-up
- Runs in your browser. Nothing you enter is sent to us.
Your robots.txt
How to use the robots.txt Generator
- Pick a starting point: allow everything, WordPress, or an online store.
- Add any folders or pages to keep crawlers out of, one per line.
- Add your sitemap address, and choose your AI crawler settings.
- Download robots.txt and upload it to the root of your site, then test it with our robots.txt checker.
What to put in robots.txt
Most small business sites need very little: allow everything, block a few private or low-value areas (admin, cart, internal search results) and list your sitemap. The file must be named robots.txt, in lower case, and sit at the root of the domain: example.com/robots.txt.
- Do block: admin areas, internal search results, cart and checkout pages, and filters that create endless URL variations.
- Don't block: CSS, JavaScript or image folders. Google needs them to render the page.
- Don't use robots.txt to hide pages from Google results: use a noindex tag instead, and leave the page crawlable.
- Never block "/" on a live site. It's the most common way sites vanish from Google after launch.
AI crawlers: training and search are separate
AI companies run different crawlers for different jobs. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect pages to build future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user fetchers (ChatGPT-User, Claude-User) read pages to answer questions now, usually with a link to the source.
Many businesses block training but allow search, so they can still be recommended and cited in AI answers. That's the default here. Blocking AI search too means those assistants can't cite your pages, which is rarely what a business wants. More on this in our generative engine optimisation service.
After uploading
Check the live file with our robots.txt checker, which tests any URL against any crawler, and with the robots.txt report in Google Search Console. Changes are picked up within a day or so.
About the robots.txt Generator.
Where do I upload robots.txt?
To the root folder of your website (public_html on cPanel), so it opens at yourdomain.com/robots.txt. In WordPress, SEO plugins like Yoast and Rank Math can edit it for you.
Does WordPress already have a robots.txt?
WordPress serves a virtual one if no file exists. Uploading a real robots.txt file replaces it.
Should I block AI crawlers?
That is a business choice. A common middle ground is to block training crawlers but allow AI search crawlers, so assistants can still cite and link to you.
Will blocking Google-Extended affect my Google rankings?
No. Google-Extended only covers Gemini training and grounding. Google Search uses Googlebot.
Is crawl-delay supported?
Google ignores crawl-delay. Bing and some others respect it, so it is offered as an option, but most sites don't need it.
You might also need these.
Robots.txt Checker & Tester
Read any site's robots.txt and test whether Googlebot, Bingbot or AI crawlers may visit a URL.
Use the toolllms.txt Generator
Create an llms.txt file: a plain summary of your site and key pages for AI assistants.
Use the toolXML Sitemap Validator
Find and validate a site's XML sitemap: format, limits, lastmod dates and a sample of its URLs.
Use the toolWant this done for you?
Free tools show you what to fix. We fix it, measure it, and keep it working while you run the business.