Robots.txt Generator & Tester for Laravel

Robots.txt is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages or sections they can or cannot access. It's important because it helps you control crawl budget, prevent indexing of private areas, and manage how bots interact with your site. Every major search engine respects robots.txt directives.

Robots.txt prevents crawling, but not necessarily indexing. If other pages link to a URL you've disallowed, Google may still index it (showing a "No information is available for this page" result). To truly prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header. Robots.txt is best for controlling crawl access, not for removing pages from search results.

Add specific User-agent rules to your robots.txt. For OpenAI's bots, add "User-agent: GPTBot" and "User-agent: ChatGPT-User" each followed by "Disallow: /". For Google's AI training bot, add "User-agent: Google-Extended" with "Disallow: /". Use this tool's generator to build these rules visually — just select the bot from the dropdown and add a Disallow rule for the root path.

Disallow tells bots not to crawl URLs starting with the specified path. Allow overrides a Disallow for a more specific path — for example, you can disallow /private/ but allow /private/public-page. When both match, the most specific rule (longest matching path) wins. An empty Disallow (no path) means everything is allowed for that user-agent.