FREE ROBOTS.TXT GENERATOR

A robots.txt file is a plain-text file placed at the root of a website domain that communicates instructions to web crawlers using the Robots Exclusion Protocol (REP). When a crawler visits a site, it first checks for the robots.txt file at the domain root before deciding which pages to crawl.

The file is publicly accessible — any visitor or crawler can read it at https://yourdomain.com/robots.txt. This makes it a communication tool between website owners and crawlers, not a security barrier.

robots.txt controls crawler access. It tells search engine bots and other crawlers which paths on your site they are and are not permitted to crawl. This is useful for preventing crawlers from wasting crawl budget on low-value pages such as admin dashboards, checkout flows, duplicate search results pages, or internal staging directories.

Critically, robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other websites link to it — search engines can infer a page exists from external links even without crawling it directly.

Start by selecting one of the four presets: Allow All creates a permissive file that allows all crawlers; Block All creates a restrictive file that blocks all crawlers; WordPress provides sensible starter rules for WordPress installations; Custom lets you build your own configuration from scratch.

Configure one or more User-agent rule groups. Each group can target a specific crawler (such as Googlebot or GPTBot) or all crawlers using the * wildcard. Within each group, add Disallow paths to block directories and files, and Allow paths to create exceptions inside blocked directories. Add your sitemap URL if available, then copy or download the generated file.

robots.txt must be placed at the root of your domain — accessible at https://yourdomain.com/robots.txt. It cannot be placed in a subdirectory such as /blog/robots.txt and still be effective for the entire domain.

For subdomain configurations, each subdomain requires its own robots.txt file. A robots.txt at https://www.example.com/robots.txt does not apply to https://shop.example.com — that subdomain needs its own file at https://shop.example.com/robots.txt.

The three core directives in any robots.txt file are User-agent, Allow, and Disallow. User-agent specifies which crawler the following rules apply to. Disallow specifies a path the crawler is not permitted to access. Allow specifies a path the crawler is permitted to access, even if a parent directory is disallowed.

The Sitemap directive is also widely supported and specifies the absolute URL of an XML sitemap. Google, Bing, and most major crawlers recognize this directive. Note that crawl-delay is not a standard directive recognized by Google — it is not included in this generator to avoid generating inaccurate output.

To block a crawler from accessing an entire directory and its contents, add a Disallow rule with a trailing slash. For example, Disallow: /admin/ prevents the crawler from accessing /admin/ and all paths beneath it such as /admin/users/ or /admin/settings.php.

Without the trailing slash, Disallow: /admin would block the exact path /admin and any path that begins with those characters — including hypothetical paths like /administrator. Using a trailing slash is the clearer and more precise convention when targeting directories.

Allow rules are used to create exceptions within a blocked directory. A common example is WordPress: you may want to block /wp-admin/ to prevent crawlers from indexing the admin area, while still permitting crawlers to access /wp-admin/admin-ajax.php — which is used by some front-end functionality.

The Allow rule must be placed in the same User-agent group as the Disallow rule. Allow rules take precedence over Disallow rules when both apply to the same path. The more specific rule always wins.

You can declare your XML sitemap directly inside robots.txt using the Sitemap directive. The value must be an absolute URL, for example: Sitemap: https://example.com/sitemap.xml

This directive is recognized by Google, Bing, and most other major crawlers. It allows any crawler that reads your robots.txt to automatically discover your sitemap without requiring a direct submission through webmaster tools. Multiple Sitemap directives can be included on separate lines if your site has more than one sitemap.

A robots.txt file can contain multiple User-agent groups, each with their own set of rules. This allows you to apply different crawling restrictions to different bots. For example, you might allow Googlebot broad access while restricting AI training crawlers such as GPTBot or ClaudeBot from accessing your content.

Each group begins with one or more User-agent lines followed by the corresponding Allow and Disallow rules. Groups are separated by blank lines. The * wildcard group applies to all crawlers that do not have a specific group defined for them.

Google's crawlers respect robots.txt directives. When Googlebot encounters a Disallow rule for a path, it will not crawl that path. However, Googlebot can still index a disallowed URL if it discovers the URL from external links — the robots.txt rule only prevents crawling, not indexing.

Google does not support the crawl-delay directive. If you need to slow down Googlebot's crawl rate, use the crawl rate settings in Google Search Console. Crawl-delay is also not recognized by Googlebot and including it has no effect on Google's crawling behavior.

Several AI companies use named crawlers that respect robots.txt. OpenAI's GPTBot and Anthropic's ClaudeBot both check robots.txt before crawling. You can block these crawlers by adding a specific User-agent group in this generator.

Be aware that not all AI scrapers respect robots.txt. Just as malicious scrapers can ignore Disallow rules, some AI data collection systems may not honor the Robots Exclusion Protocol. robots.txt is a convention, not a technical barrier. For content you want strictly protected, access controls and legal terms of service are necessary.

robots.txt and noindex serve different purposes and operate at different levels. robots.txt controls crawling — whether a crawler is permitted to visit and read a page. The noindex meta tag controls indexing — whether a page that has been crawled should be included in search results.

A critical distinction: if you block a page with robots.txt, a crawler cannot read the noindex tag on that page because it cannot access the page at all. This means blocking a page in robots.txt does not prevent it from being indexed — it only prevents crawling. To reliably exclude a page from Google search results, it must be crawlable so that Googlebot can read the noindex tag.

Accidentally blocking the entire site with Disallow: / under User-agent: * is the most damaging mistake. Always verify the output of your robots.txt file before deploying. This generator flags this condition with a warning.

Other common mistakes include: placing robots.txt in a subdirectory instead of the domain root; using relative paths in Disallow rules (all paths must start with /); blocking CSS or JavaScript files that Googlebot needs to render pages correctly; adding crawl-delay expecting Google to respect it; and confusing robots.txt disallow with secure access control.

robots.txt cannot prevent a URL from appearing in search results. If external sites link to a URL you have blocked in robots.txt, search engines may still show that URL in results — they simply will not have crawled its content.

robots.txt cannot protect sensitive content from being accessed by users or malicious crawlers. It is a public file and only provides guidance to crawlers that choose to respect it. It provides no authentication, encryption, or server-level access control. For genuinely private content, use server authentication, proper HTTP access controls, or content management permissions.

After generating your robots.txt file, upload it to the root directory of your web server so it is accessible at https://yourdomain.com/robots.txt. Most hosting platforms and CMS systems (including WordPress) allow you to set or override robots.txt through their settings or a plugin.

To verify that your file is correctly deployed and accessible, visit https://yourdomain.com/robots.txt in your browser. You can also use Google Search Console's robots.txt tester to check how Googlebot interprets specific rules — though note that this generator produces locally validated, standards-compliant output and does not require external validation to function correctly.