Explore Tools

Robots.txt Generator

Create a valid robots.txt file in seconds. Block private folders, add your sitemap, stop AI crawlers from training on your content, and test any URL against the rules before you upload.

  • Block AI crawlers
  • Built-in URL tester
  • Follows RFC 9309
Block AI crawlers
Block other bots
You can edit the result directly. The tester below checks whatever is in this box.
Test a URL

How to use the Robots.txt Generator tool

  1. Set the defaultAllow all crawlers (normal websites) or block everything (staging and test sites).
  2. List what to blockAdd folders or pages to keep out of search engines, with optional exceptions.
  3. Add sitemaps and bot rulesEnter your sitemap URL, and tick any AI crawlers or SEO bots you want to block.
  4. Test and downloadCheck URLs with the tester, then download robots.txt and upload it to your site’s root folder.

Features

  • AI crawler blockingOne-click rules for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more.
  • URL testerSee instantly whether Googlebot or another crawler may fetch a URL, and which rule decides it.
  • Wildcards supportedUse * for any characters and $ for “ends with”, as supported by Google and Bing.
  • Editable outputFine-tune the generated file by hand; the tester checks your edits live.
  • Standards-basedMatching follows the official robots.txt standard (RFC 9309): longest match wins, Allow wins ties.

What is robots.txt?

robots.txt is a small text file at the root of your website, for example https://www.example.com/robots.txt, that tells crawlers which parts of the site they may visit. Search engines like Google and Bing check it before crawling. It’s useful for keeping crawlers out of admin areas, internal search results, shopping carts and endless filter combinations that waste crawl budget.

robots.txt syntax

LineMeaning
User-agent: *The rules below apply to all crawlers
User-agent: GPTBotThe rules below apply only to OpenAI’s crawler
Disallow: /admin/Don’t crawl anything starting with /admin/
Allow: /admin/help/Exception: this folder may be crawled
Disallow: /*?sort=Block any URL containing ?sort=
Disallow: /*.pdf$Block URLs that end in .pdf
Sitemap: https://…/sitemap.xmlWhere to find your XML sitemap

Important: robots.txt is not a security tool

  • It doesn’t hide pages. The file is public, and listing a secret folder actually advertises it. Protect private areas with a password.
  • It doesn’t remove pages from Google. A blocked URL can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex meta tag instead (see our Meta Tag Generator).
  • Bad bots ignore it. Reputable crawlers follow robots.txt; scrapers and malware usually don’t.

Blocking AI crawlers

Many AI companies crawl the web to train language models. You can opt out of several of them in robots.txt: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s AI models; regular Google Search is not affected), CCBot (Common Crawl, used by many AI projects), Applebot-Extended, PerplexityBot, Bytespider and meta-externalagent. Blocking them doesn’t affect your rankings in Google or Bing search. New crawlers appear regularly, so review your file from time to time.

How to install robots.txt

  1. Download the file. It must be named exactly robots.txt in lowercase.
  2. Upload it to the root folder of your website, so it opens at yourdomain.com/robots.txt. A robots.txt inside a subfolder is ignored.
  3. Each subdomain (such as shop.example.com) needs its own file.
  4. Check it in Google Search Console’s robots.txt report.

Don’t have a sitemap yet? Create one with the XML Sitemap Generator.

Frequently asked questions

Where do I put robots.txt?
In the root folder of your website so it is available at https://yourdomain.com/robots.txt. Files in subfolders are ignored.
How do I block AI bots like GPTBot?
Tick the AI crawlers you want to block. The generator adds a group with their user-agents and “Disallow: /”.
Will blocking AI crawlers hurt my Google rankings?
No. Google-Extended only controls use of your content for Google’s AI models; Googlebot, which powers Search, is separate.
Can robots.txt remove a page from Google?
No. Use a noindex meta tag (and allow crawling) to remove a page from search results. robots.txt only controls crawling.
What does “Disallow:” with nothing after it mean?
An empty Disallow means nothing is blocked: crawlers may visit the whole site.