Robots.txt checker and tester.

What robots.txt does, and what it doesn't

robots.txt is a plain text file at the root of a site (example.com/robots.txt) that tells crawlers which parts they may visit. Well-behaved crawlers, including Google, Bing and the major AI companies, follow it.

It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it, just without a description. To keep a page out of search results, let it be crawled and add a noindex tag instead. And it's not security: anyone can read the file, so never use it to hide private areas.

How the rules are matched

• A crawler follows the group that names it most specifically. If GPTBot has its own group, it ignores the "User-agent: *" group entirely.
• Within that group, the longest matching rule wins. "Allow: /blog/" beats "Disallow: /" for /blog/post.
• When an Allow and a Disallow match with the same length, Allow wins.
• * matches any characters and $ marks the end of the URL, so "Disallow: /*.pdf$" blocks every PDF.
• An empty "Disallow:" means everything is allowed.

Common robots.txt mistakes

• "Disallow: /" left over from a staging site, which blocks the whole site from Google.
• Blocking CSS, JavaScript or image folders, so Google can't render the page properly.
• Trying to hide pages with robots.txt instead of noindex.
• No Sitemap line, which costs nothing to add.
• Accidentally blocking AI search crawlers while trying to block AI training.

robots.txt and AI search

AI companies now publish separate crawlers for training and for live search. You can allow OAI-SearchBot, Claude-SearchBot and PerplexityBot so your pages can be cited, while blocking GPTBot and ClaudeBot from training. Our AI search readiness checker tests all of them at once. If you'd like help deciding, that's part of our generative engine optimisation work.