Robots.txt Generator & Validator
Build a robots.txt file, test your rules and block AI crawlers.
Robots Configuration Builder
Customize user-agent directive blocks, crawl limits, and directory permissions.
Quick-Start Presets:XML Sitemaps & Host Directives
Discovery metadataAI Scraping & LLM Harvester Defense
Explicitly block AI model trainers from scraping your unique articles and intellectual property.
User-Agent Directives (1)
robots.txt
# robots.txt generated by Ai Directory SEO Tools# Generated: 2026-11-17# For syntax guidelines: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txtUser-agent: *Disallow: /admin/Disallow: /api/Disallow: /private/Disallow: /search/Disallow: /*?*Allow: /Sitemap: https://example.com/sitemap.xml
Lines: 13 · Chars: 329 · Encoding: UTF-8
Interactive URL Crawl Tester
Rule simulatorBlocked by Disallow: "/admin/"
Matching Directive: Disallow: /admin/
Syntax & SEO Health Audit
1 checks1 sitemap(s) specified for automated crawler discovery.
How to use it
- Pick a preset
Start from a quick-start preset, then add or remove paths, bots and sitemaps.
- Test your rules
Pick a crawler and type a URL path to see if it is allowed or blocked.
- Copy or download
Copy the robots.txt text or download the file and upload it to the root of your site.
Generate syntax-clean crawl instructions for search engines and protect your content from unauthorized AI scrapers. Includes instant preview, rule tester, and direct download.
Mastering Robots.txt for Search & AI in 2026
A properly configured robots.txt file guides search engine crawl budgets effectively, protects confidential application paths, and establishes clear boundaries for autonomous AI crawlers.
The file must reside at the root of your domain: https://example.com/robots.txt. Subdirectories (like /blog/robots.txt) are ignored. It must return an HTTP 200 status code in plain text UTF-8 and stay under 500 KB.
Robots.txt prevents crawling (fetching resources), not indexing! If other websites link to your disallowed URL, Google can index the URL title. To remove a page completely from search results, allow crawling and use the noindex meta tag.
Modern search engines like Googlebot render client-side JavaScript and CSS just like real browsers. Blocking .css, .js, or asset bundles degrades Core Web Vitals and hurts organic search rankings.
Frequently Asked Questions
Where do I upload the robots.txt file?
Upload the robots.txt file to the top-level root directory of your web server (typically public_html, /var/www/html, or the public/ folder in modern frameworks like Next.js, Vite, and Astro). It must be directly accessible in a browser at https://yourdomain.com/robots.txt.
What is the difference between Googlebot and Google-Extended?
Googlebot is Google’s primary search crawler responsible for indexing pages for Google Search. Google-Extended is a separate user-agent used to collect training data for Google Gemini and Vertex AI models. Disallowing Google-Extended will stop AI training data harvesting without hurting your regular Google search rankings.
Does Googlebot respect Crawl-delay?
No. Googlebot ignores Crawl-delay because it uses proprietary algorithms to determine the optimal crawl rate based on your server capacity. Crawl-delay is, however, respected by Bingbot, Yandex, and Baidu.
Can I use wildcards (*) and dollar signs ($)?
Yes. The asterisk (*) matches any sequence of characters, and the dollar sign ($) indicates the end of a URL. For example, "Disallow: /*.pdf$" blocks all files ending in .pdf, and "Disallow: /*?*" blocks URLs containing URL query parameters.
Why should I include a Sitemap directive in robots.txt?
Including "Sitemap: https://yourdomain.com/sitemap.xml" makes it effortless for any compliant web crawler to discover your full inventory of canonical URLs, change frequencies, and media files without guessing.
More free tools
Convert titles and phrases into clean, SEO-friendly, URL-safe kebab-case slugs.
Runs in your browserGenerate HTML meta tags with live Google SERP snippet and social media share card previews.
Runs in your browserAnalyze text with real-time word, character, sentence, paragraph, and reading time statistics.
Runs in your browser