About the AEOReadinessAudit crawler

Every request sends this user agent:

Mozilla/5.0 (compatible; AEOReadinessAudit/0.1; +https://www.aeoreadi.com/bot)

• The product page URL that a user submitted, once per audit.
• /robots.txt and /llms.txt, to report which crawlers your rules allow.
• Sometimes a headless-browser render of the same page, to check which content depends on JavaScript. Images, fonts and media are not downloaded.

If you return a 429 we wait and retry at most twice, respecting Retry-After up to 10 seconds. Audits are started by people, one page at a time; there is no background crawling.

Each fetch is a one-off request made on behalf of a person, so generic rules (User-agent: *) don't apply to it, in the same way they don't apply to a browser. Rules that name our crawler do: add User-agent: AEOReadinessAudit with Disallow: / and we won't fetch your pages. We also read your robots.txt to report on other crawlers (OAI-SearchBot, PerplexityBot, Googlebot and others) and never impersonate them.

If your bot protection blocks audits of your own pages, add a rule that skips bot challenges for requests whose user agent contains AEOReadinessAudit. In Cloudflare, that's a WAF custom rule with the action Skip. User agents can be spoofed, so scope the rule to your product paths if you prefer.

You don't have to allow it: the audit form also accepts pasted page source.

Blocking the user agent at your firewall stops all audits of your site. To ask us to exclude your domain or report a problem, email padrondamon@gmail.com.