Our Crawler: A Stable Identity, and It Respects Your Blocks.
So you can recognise and allow our reader — the same courtesy a search engine's crawler docs extend. This page documents the bot's identity for interoperability; it is not a description of how our analysis is performed.
How Our Reader Behaves
How We Identify
Our reader sends a stable, published user-agent so you can recognise it in your logs and allow it deliberately. The identity does not change scan to scan.
What We Fetch
- Your homepage, once, for a single-page read.
- Your
robots.txtandsitemap.xml, to stay inside what you allow. - Public DNS and certificate records, which don't touch your server's pages.
We Respect Blocks
If your robots.txt disallows us, or your security layer refuses the request, we stop. We report the refusal honestly and never work around it — no retries, no disguises, no second identity.
Why a Stable UA
A published, unchanging identity is what makes the honest-block story verifiable by you: you can see exactly who asked, and see that we stopped when told to. It is a prerequisite for verifiable bot authentication.