Agent readiness checklist
For CrawlerDesk public demo (demo.crawlerdesk.com): what CrawlerDesk serves, and how to check the rest.
1. CrawlerDesk discovery files
| File | CrawlerDesk | State | Public URL |
|---|---|---|---|
llms.txt |
live | served · last publish 2026-09-27T03:51:00Z | demo.crawlerdesk.com/llms.txt |
llms-full.txt |
live | served · last publish 2026-09-27T03:51:00Z | demo.crawlerdesk.com/llms-full.txt |
Read live from CrawlerDesk — the same facts as the public status page. One body is served to every user-agent.
2. Run a full agent-readiness scan
Cloudflare's isitagentready.com — scan demo.crawlerdesk.com checks a public site for llms.txt, MCP, agent skills and other agent-friendly standards. The scan is Cloudflare's, not CrawlerDesk's: it reads whatever the scanned origin serves.
3. Weekly AI-crawler digest
Read-only demo digest (last closed week): open the digest. Counts are real known-AI-crawler hits on the demo's files — nothing is seeded; a quiet week shows as empty.
4. Keep your own origin honest
- Publish
llms.txtat your site root (CrawlerDesk serves the body you edit in the dashboard). - Declare crawler policy in
robots.txt/ Content Signals — a declaration, not a block. - Watch which known AI crawlers actually fetch those files.
This checklist does not certify any site. CrawlerDesk does not enforce crawler policy on traffic that does not reach it.