Robots.txt Generator

# Robots.txt will populate here

What Is a Robots.txt File?

A robots.txt file is a plain text file that is located in the root of your website and communicates to search engine robots which pages they are permitted to crawl and which ones they should skip. It's a part of the Robots Exclusion Protocol, which is a standard that almost all significant search engines (Google, Bing and others) follow while crawling the web.

The first thing that a crawler such as Googlebot does on a site, before crawling any of your content, is to check for /robots.txt. The file that forbids a path will usually be ignored by the crawler. If no robots.txt file, then everything is assumed to be crawlable.

You should be careful to note that this file does not govern the crawling process, but rather the indexing process. Other sites can link to the page, and it may show up in search results (typically without an description) even if it has been blocked in a robots.txt file — robots.txt does not necessarily make a page disappear from search results, just a page that a search bot will not crawl. Instead, use a noindex meta tag, or password protection.

Creating a Robots.txt File (3 Step Guide)

1. Configure default rule. Set whether to allow all crawlers by default (the most common setting — 'Allow All' — is enabled by default) or to block them by default.
2. Add specific rules. Exclude directories that you don't want Google or other bots to crawl, and include per-bot rules for individual Google and Bing crawlers and the GPTBot.
3. Include your sitemap and generate. Enter the URL of your sitemap, click the Generate button, and copy or download the generated robots.txt file. Upload it to your site's root directory, so that you can access it via yourdomain.com/robots.txt, not via yourdomain.com/yourfolder/robots.txt.

Robots.txt Syntax, Explained

A minimal example that leaves everything as it is and refers to a sitemap:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

An example that would deny access to one bot in one folder, and give access to everyone else:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Note that the * rules are "all crawlers", that each group of rules begins with its own User-agent line and that the file paths are case sensitive — /Photo/ is a different directory from /photo/.

Errors That Can Actually Problem Your Site.

This is what people say little about when talking about generators, and it's why a generator is worth using instead of hand-writing the file:

• The inadvertent full disallow. Deny: / under User-agent: * will deny access to all crawlers to your whole site. It's one character off target for Disallow: /admin/ and has inadvertently removed entire sites from search results.
• Blocking personal CSS/JS. Blocking asset folders such as /wp-content/themes/ or /assets/js/ was used to be common practice; but today it is detrimental to your rankings as Google needs to see your page as a real user does to properly evaluate it.
• Using robots.txt as a privacy measure. Occasionally, disallowed pages will appear in search results if they are linked to from other sites. Don't only use robots.txt to exclude a page from search if it really should not be indexed.
• Case-sensitivity mismatches. The Disallow: /photo/ takes no effect if your actual folder is called /Photo/.
• Listing individual files instead of the folder. Don't block the whole directory if you want to block all the files in it; It's easier to maintain and less likely to be mis-blocked.

Common bots that you can target.

If you want it specifically not included in AI training crawls, it's usually addressed through special Disallow rules for AI bots (GPTBot, ClaudeBot) that are distinct from those for search engine indexing crawls.

Frequently Asked Questions