Robots.txt Generator
Take control of your website's crawl budget. Instantly generate a custom file to guide search engines and block unauthorized AI scrapers.
Global Rules
Restricted Directories
Paths are relative to the root and should contain a trailing slash (e.g., /wp-admin/).
XML Sitemaps
Leave blank if you don't have one. Must be an absolute URL.
Search Engines
AI Crawlers (2026)
Live Preview
Generated securely inside your browser. Zero data is sent to our servers.
How To Create a Robots.txt File
Set Global Rules
Choose whether search engines are allowed to crawl your site by default. You can also specify restricted directories (like admin panels) and set a crawl delay to protect server bandwidth.
Manage Bots
Fine-tune access for specific search engines (like Google or Bing). Importantly, use our AI toolset to instantly block aggressive AI data scrapers like GPTBot or ClaudeBot.
Download & Upload
Review the live code preview. Once satisfied, click download. Upload the generated robots.txt file to the root directory of your website via FTP or your hosting file manager.
Premium Generator Features
AI Bot Blocking (2026)
The web has changed. Keep your copyrighted content out of Large Language Models (LLMs) by blocking modern AI scrapers like GPTBot, ClaudeBot, and Applebot-Extended.
Real-Time Compilation
No need to click "Generate" constantly. As you tweak dropdowns and add paths, the live preview window compiles the strict syntax required for a valid robots.txt file instantly.
Multi-Sitemap Support
For massive websites, a single XML sitemap isn't enough. Our tool allows you to seamlessly append multiple sitemap index URLs to guide bots exactly where they need to go.
Robots.txt Syntax Cheatsheet
Core Directives
-
User-agent:
The bot you are targeting. An asterisk (
*) means the rule applies to all crawlers. Specific names (likeGooglebot) apply only to that crawler. -
Disallow:
Tells the targeted bot which directories or files it cannot crawl. Example:
Disallow: /admin/blocks the admin folder. -
Allow:
Overrides a Disallow directive. Used to grant access to a specific file within a blocked directory. Example:
Allow: /admin/public.html
Advanced Wildcards
-
* (Asterisk)
Matches any sequence of characters. Useful for blocking specific file types or dynamic parameters. Example:
Disallow: /*.pdfblocks all PDF files on the site. -
$ (Dollar Sign)
Matches the end of a URL string. Example:
Disallow: /*.php$blocks any URL that ends exactly with .php, but allows URLs like.php?id=1. -
Important Note: Robots.txt is case-sensitive.
/Admin/is treated differently than/admin/.
Frequently Asked Questions
No, a robots.txt file only prevents search engines from crawling the page. If the page is already indexed or if other websites link to it, Google may still display the URL in search results (often with a "No information is available for this page" snippet). To completely remove a page from Google, you must use a noindex meta tag.
Robots.txt relies on voluntary compliance. Legitimate bots like Googlebot, Bingbot, and most AI scrapers (like GPTBot) will respect your rules. However, malicious bots, spammers, and hacking tools will simply ignore the file. To block malicious traffic, you need server-level protection like an .htaccess block, firewall, or Cloudflare.
The file must be named exactly robots.txt (all lowercase) and placed in the top-level root directory of your website. For example, it must resolve exactly at https://yourdomain.com/robots.txt. You can upload it using an FTP client, your hosting provider's cPanel File Manager, or a dedicated CMS plugin (like Yoast for WordPress).
The Crawl-delay directive tells search engines to wait a specific number of seconds between page requests to prevent server overload. However, Googlebot completely ignores this directive. Instead, you must set Google's crawl rate via Google Search Console. Other search engines like Bing, Yandex, and Baidu do still respect the crawl delay tag.
Yes! If you have a large website with a sitemap index or separate sitemaps for posts, pages, and products, you can include multiple Sitemap: [URL] directives in your robots file. Just make sure you are using the absolute URL (including https://) for every entry.