Robots.txt Tester

Demo tool — results are sample data, not a live lookup.

About the Robots.txt Tester

robots.txt is a small file at the root of your site that tells search-engine crawlers which paths they may and may not request. Get it wrong and you can accidentally block Google from your whole site — or leave admin paths crawlable that you'd rather keep quiet.

This tool fetches robots.txt and shows the rules it contains, so you can confirm important pages are crawlable, sensitive paths are disallowed, and your sitemap is linked.

How to use it

1Enter your domain.
2Run the check to read the live robots.txt rules.
3Confirm you're not blocking anything important, and that the file links to your sitemap.

What it checks

User-agent: Which crawler a block of rules applies to (* means all crawlers).
Disallow: Paths crawlers are asked not to request, e.g. /wp-admin/.
Allow: Exceptions that override a broader Disallow rule.
Sitemap: A link to the XML sitemap so crawlers can find your page list.

Frequently asked questions

Does robots.txt block a page from Google entirely?

It stops crawling, not indexing. To keep a page out of results, use a noindex meta tag instead — a disallowed page can still be indexed from links.

What should a WordPress robots.txt look like?

Typically it disallows /wp-admin/ but allows /wp-admin/admin-ajax.php, and links to the sitemap. It should not block CSS or JS.

Where is robots.txt located?

Always at the domain root — for example https://example.com/robots.txt. WordPress generates a virtual one if no physical file exists.

XML Sitemap CheckerFind a site's XML sitemap, count its URLs, and flag common errors.Meta Tags CheckerCheck a page's title, description, canonical, and Open Graph tags.Security Headers CheckerCheck a site's HTTP security headers — HSTS, CSP, X-Frame-Options, and more.

Run WordPress on infrastructure built for it.

Performance, security, and the WordPress tooling you need — fully managed, so these checks stay green.

Get Started More tools