Robots.txt Generator

Create and validate robots.txt files for your website. Control how search engine crawlers access your content.

Define rules for different user agents

User-Agent

Disallow Paths

Allow Paths

Optional directives

Sitemap URL

Host (canonical domain)

Crawl Delay (seconds)

Test if a URL is allowed or blocked

Test as User-Agent

User-agent: *
Allow: /

User-agent: *

Applies to all crawlers

Disallow: /path/

Block access to a directory

Allow: /path/

Explicitly allow (overrides Disallow)

Disallow: /*.pdf$

Block all PDF files

Disallow: /*?*

Block URLs with query strings

Place your robots.txt file in your website's root directory:

https://example.com/robots.txt