/bot · robots.txt aware
Crawler politeness
This page is our full politeness and boundary policy. It is published so you can audit it, block us, or get in touch — transparency is the whole point.
Identity
- Documented identity:
- afr-im-bot/1.0 (+https://afr.im/bot)
- Shipped user-agent string (this build):
- afrScanner/0.1 (+https://afr.im/bot; AI-friendliness research crawler) · scanner-0.1.0
Both identifiers resolve to this page. If you block us in robots.txt, the next scan honours it — see the declined mechanism below.
Request budget
- ≤ 8 public GETs per site
- Fixed probe list below; retries and challenge fallbacks stay inside a hard cap.
- Concurrency ≤ 5 sites
- Never more than five sites at a time, ≥2s apart inside one host.
- 1 full scan per site / 24h
- Deduplicated by the scan queue — submissions and re-scan requests merge into one round.
- crawl-delay honoured
- Your robots.txt crawl-delay is respected (the shipped scanner caps it at 3s, with a 150ms default gap).
- 15s timeout, 1 retry
- Per request, on network error only — we do not hammer a slow origin.
Probe list (fixed, in order)
- GET /robots.txt A1 evidence source, crawl-delay and Sitemap declarations
- GET /llms.txt B1
- HEAD /llms-full.txt B1
- GET / A2 / A3 / B2 / B3 / B5 — one request, many checks
- GET /sitemap.xml B4
- GET /agents.md C2
- HEAD /.well-known/openapi.* C1 — two extensions probed
- GET docs subdomain (optional) C1 / C3 — only when heuristics hit
Scope: read-only, public only
- GET requests to public URLs only — no authentication bypass, no forms, no injection, zero writes.
- No personal data is stored: HTML is reduced to parsed signals plus an excerpt of at most 2KB.
- One JSON snapshot per site is committed publicly, so everything we saw can be checked.
Declined mechanism
If your robots.txt disallows everything for "*" and does not exempt our bot, we skip probes 4–8 and keep only robots.txt itself plus passive data (Tranco rank, category). Checks A2–C4 are recorded as declined / unknown — neither rewarded nor punished — and your report page states explicitly that the rating is based solely on your public robots declaration.
You do not have to talk to us to be left alone: robots.txt is enough.
What we keep
- Signal values and HTTP status codes that the engine scored.
- An excerpt of at most 2KB per response, used as the evidence you can expand in every report.
- The engine version, scanner version and timestamp of the scan.
Opt out or say hello
Blocking: add our user-agent to robots.txt — it takes effect on the next scan. Questions, false positives or removal requests: open an issue on AFR-web, or send a correction PR to AFR-data.