Write crawl rules. See what they do.
Generate a robots.txt file, load one from a website URL, or paste its text to test which paths it allows. Get the matching rule and line number for each result.
Your robots.txt file
Download and review the file before adding it to your site. Generating or testing a file here does not change your live robots.txt.
Your crawl test
| Path or URL | Crawl result | Rule and group |
|---|
Results use the displayed rules, which may be a loaded or edited copy. They do not verify access to the tested page, Google’s cached rules, indexing or server permissions.
How to use this robots.txt tool
- Choose Generate a file and add the paths you want to allow or disallow.
- Generate the file, then select Use in tester to check the homepage and representative paths.
- Review the results, download the file and place it at the root of the correct website origin, for example
https://example.com/robots.txt.
For an existing file, choose Test a file, then load it from a Website URL or paste its content. Enter the paths you want to check. The tester shows the winning rule, its line number and the selected crawler groups. It models Google’s documented robots.txt matching rules; other crawlers may behave differently.
Crawling and indexing are different
A blocked crawl does not guarantee that a URL disappears from Google Search. Robots.txt is also not access control: it cannot protect private documents. Use the appropriate indexing controls or authentication for those tasks.
How does the tester choose a crawler group?
It selects the most specific matching crawler product-token prefix. Repeated groups at that specificity are combined. A matching specific group takes priority over the wildcard group; the wildcard rules are not added to it. User-agent matching ignores case, while path matching preserves it.
Which path rule wins?
The longest matching rule wins. When matching Allow and Disallow rules have the same length, Allow wins. The tester supports * for a sequence of characters and a final $ for the end of the path and query. It compares Unicode and percent-encoded paths in a common encoded form.
What happens to an empty Disallow line?
An empty Allow or Disallow line has no matching path and is ignored. If no applicable rule blocks the tested path, the result is Allowed. A Disallow: / rule is very different: it is a broad block covering all paths in that group, subject to more specific exceptions.
Can I use the WordPress example on any site?
The example blocks /wp-admin/ and allows /wp-admin/admin-ajax.php. It is a starting point for review, not an audit of your site. Check existing rules, sitemap URLs and plugin behaviour before replacing your file.
Does this guarantee what Google will crawl?
No. This is a local rule simulator, not Google’s own parser or a live crawl. It supports up to 5,000 lines, 1,000 characters per rule path and 100 test URLs. Google’s fetch handling, redirects, cached files, HTTP errors and real crawler selection are outside this test. Non-root-relative rule paths are reported and ignored by this tool.
Are files or test URLs uploaded?
Generation, pasted-file reading and rule tests happen in your browser. Website URL loading sends the entered site address to Cordell Digital’s WordPress server, which retrieves its public robots.txt file. Pasted text and test paths are not uploaded. This tool does not save a history, although ordinary hosting logs may record requests.
Matching reference: Google’s robots.txt specification. Review the real file after deployment using your search-engine webmaster tools.
Need help with crawl access?
Cordell Digital can review crawl rules alongside your website’s indexing and technical SEO setup.
Talk to Cordell Digital ↗