EvaristBot

Every request Evarist sends to a customer's site carries this exact User-Agent, verbatim:

Mozilla/5.0 (compatible; EvaristBot/1.0; +https://evarist.ai/bot)

The product token is EvaristBot. Match it in a robots.txt group, a WAF or CDN bot rule, or your access logs. The version number identifies crawler behaviour, not a release; it changes only when what we do changes.

We read robots.txt first and obey it before fetching anything. Your homepage renders next, in a real browser, so client-rendered sites read correctly and not just their initial HTML. Then we read sitemap.xml to plan the rest of the crawl.

Three other kinds of request carry the same User-Agent but aren't the crawl: an account holder authoring or checking a no-code tag, an account holder running an install check, and our tag-health monitor automatically re-checking a tag that looks broken. Each re-fetches a single page already known from your site's model, never a newly discovered one, so robots.txt, covered next, governs the crawl and doesn't gate these.

• Before crawling, we read robots.txt and obey it, including a group addressed specifically to EvaristBot. This governs the crawl. The owner-triggered, one-off fetches described above aren't autonomous crawling, so this rule doesn't apply to them.
• Normally only one crawl runs against a site at a time.
• Concurrent requests stay low: typically single digits, low double digits at most during a brief overlap between a scheduled crawl and a refresh.
• A first crawl covers up to a couple hundred pages; later refreshes are much smaller.
• We refresh roughly every couple of weeks, and stop entirely for sites that go quiet.

Match the EvaristBot token in your WAF's or CDN's bot rules. Cloudflare, Akamai and similar platforms let you allow (or block) a request by its user-agent string. Some platforms, Akamai's custom bot categories for instance, also let you define your own named bot category this way. A pre-built "verified bot" directory, where a platform has one, is typically populated by the bot operator applying to be listed, not something a site owner adds a bot to from their own dashboard, so it isn't where you'll find EvaristBot; the custom rule above is the one to use.

A user-agent identifies, it doesn't authenticate. Anyone can send this header, so treat a UA rule as "let this crawler through", never as proof of who is calling. Scope it to what you'd let any well-behaved crawler do, which is reading pages you already publish, and don't let it bypass authentication or reach anything you wouldn't serve to the open web. If you need a rule that can't be spoofed, email support@evarist.ai and we'll give you our current egress addresses to pair it with.

We honour a robots.txt rule that disallows us:

User-agent: EvaristBot
Disallow: /

A group naming EvaristBot replaces the * group for us entirely; it doesn't add to it. Scope the rules under your EvaristBot group completely; we won't fall back to whatever the * group also allows or disallows.