Free robots.txt generator
Build crawl rules from blank fields, check the supported values, and copy one reviewable file.
Crawl configuration
Nothing is published or tested automatically. The generator only creates the text you review and place at your domain root.
Use * for every crawler, or one product token such as Bingbot.
Choose deliberately. A site-wide block can remove crawl access.
Bing supports 1–20 seconds. Google ignores this directive.
Make one rule group, then test the live file
- 1
Choose the crawler
Use * to address every crawler or enter one product token, such as Bingbot, for a crawler-specific group.
- 2
Choose an explicit policy
Allow everything, block everything, or add exact path rules. The preview stays blank until you make this choice.
- 3
Add optional discovery and pacing fields
Sitemap values must be complete URLs. Only add crawl-delay for a crawler that supports it.
- 4
Publish and test
Review the file, publish it at /robots.txt, then test important crawler and URL combinations against the live response.
Know what each line actually controls
| Field | Purpose | Important limit |
|---|---|---|
| User-agent | Selects the crawler group. | Use * or a crawler product token. |
| Allow / Disallow | Permits or prevents path requests. | Not a security or noindex control. |
| Sitemap | Points crawlers to a sitemap. | Use a complete http:// or https:// URL. |
| Crawl-delay | Asks a supporting bot to pause. | Google ignores it; Bing documents 1–20 seconds. |
Robots rules can reduce crawling, but blocked URLs may still be indexed if search engines discover them elsewhere. Use page-level indexing controls when the goal is to keep a crawlable page out of search results.
The file is public. Never put secrets in it or treat it as an access-control system. Use authentication to protect private content.
Check the rule against the crawler that matters
The generator checks the supported fields it accepts. It cannot predict how every crawler will interpret a live site. Review the Robots Exclusion Protocol and the current guidance for your crawler before a high-risk change.
Google documents its robots.txt support. Bing documents both file creation and crawl control. Recheck those sources when crawler behavior matters to a launch or migration.
Robots.txt answers
- What does a robots.txt file do?
- A robots.txt file tells compliant crawlers which paths they may request. It manages crawling, not whether a page can appear in search results.
- Where should I upload robots.txt?
- Publish it as UTF-8 plain text at /robots.txt on the exact scheme, host, and port it controls. A file on one subdomain does not control another subdomain.
- Can robots.txt protect private pages?
- No. The file is public and crawl rules are not access controls. Protect private content with authentication or another real authorization mechanism.
- Does Google support crawl-delay?
- No. Google does not support the crawl-delay field. Bing documents crawl-delay values from 1 to 20 seconds and recommends placing the directive in the relevant user-agent group.
- Should the sitemap URL be absolute?
- Yes. Use a complete http:// or https:// sitemap URL, such as https://www.example.com/sitemap.xml.
Continue with the live site
Robots.txt Tester
Test whether a live file allows one crawler to request one URL.
Redirect Checker
Follow the hops a crawler takes and confirm the final response.
SEO Title & Meta Description
Write snippet copy against a live Google preview.
Open Graph & Twitter Cards
Build social meta tags with live preview.
Robots.txt Examples
Compare practical patterns before choosing rules for your site.
On-page SEO Analysis
Check the indexability and on-page signals around one public page.
Need to check more than crawl rules?
A robots.txt file controls crawler access to paths. It does not review the title, headings, links, content, mobile experience, or conversion path on the page itself.
Submit one public URL for a one-page audit. No site crawl or Search Console access required.