Robots.txt Generator & Validator

Add allow and disallow rules for each crawler, set a crawl delay, list your sitemaps, and get a validated robots.txt file ready to upload to your site root.

All options in SEO Utilities

How to Use Robots.txt Generator

Start from a template or an example

Press Load Example to start from a typical configuration, or build rules from scratch.

Add rules per crawler

Use Add Rule to create Allow and Disallow lines for all crawlers (*) or for a named crawler such as Googlebot or Bingbot.

Add sitemaps and optional directives

List the absolute URL of each sitemap and, if a crawler supports it, set a crawl delay.

Validate, then copy or download

Review the validation checks, then copy the file or download robots.txt and upload it to the root of your domain.

Robots.txt Generator Guide

What robots.txt does

robots.txt is a plain text file at the root of a domain, such as https://example.com/robots.txt, that tells crawlers which paths they may fetch. Rules are grouped by User-agent: a group for * applies to every crawler, and a named group such as Googlebot or GPTBot overrides it for that crawler.

It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, just without a description. To keep a page out of results, allow crawling and add a noindex meta tag instead.

Mistakes the validator catches

A single "Disallow: /" under User-agent: * blocks your whole site, and it is surprisingly common after a staging configuration ships to production. Other frequent problems are rules with an empty user agent, paths that do not start with a slash, duplicate rules and sitemap lines with relative URLs. The validator flags these before you upload. One thing to check yourself: do not block the CSS and JavaScript files search engines need to render your pages.

Controlling AI crawlers

Many AI companies publish their crawler names, such as GPTBot, ClaudeBot, Google-Extended and CCBot, and honor robots.txt rules for them. Add a separate group for each one you want to allow or block. Keep in mind that robots.txt is a request, not an access control: well-behaved crawlers follow it, but it does not stop anyone determined to fetch your pages.

Frequently Asked Questions

Where does robots.txt need to go?

At the root of each host, such as https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain, such as blog.example.com, needs its own file. The name must be lowercase.

Will Disallow remove a page from Google?

No. Disallow stops crawling, but a blocked URL can still be indexed from links on other sites. To remove a page from search, let it be crawled and add a noindex meta tag or X-Robots-Tag header, then wait for Google to recrawl it.

How do Allow and Disallow conflicts get resolved?

Google and Bing follow the most specific rule, meaning the one with the longest matching path. If the two match with equal length, Allow wins. So "Disallow: /docs/" with "Allow: /docs/public/" blocks everything under /docs/ except the public folder.

Does Google respect Crawl-delay?

No. Googlebot ignores Crawl-delay and adjusts its rate automatically based on how your server responds. Bing and some other crawlers do honor it. If Google is crawling too fast, return 503 or 429 responses temporarily rather than relying on robots.txt.

Should I block my admin or login pages?

You can, but robots.txt is public, so listing a secret path in it reveals that path to anyone who reads the file. Protect private areas with authentication, and use Disallow only to save crawl effort on pages that have no search value.

Related Guides

HTTP status codes explained: a practical referenceA practical guide to the HTTP status codes you actually meet, grouped by class, with advice on choosing the right code for APIs and websites.