robots.txt

One stray line and Google stops indexing you entirely.

A leftover Disallow: / from staging, or a /robots.txt that's secretly an HTML 404 so every rule you wrote is ignored — robots.txt fails silently. Canopy reads it as a document, groups and directives and sitemap line, and grades what a crawler would actually obey.

• Parsed as a real document
• AI-crawler stance graded too
• Silent failures caught
• Free, no account