CRAWLER IDENTITY

DiscoverGraphBot

A bounded research crawler used to measure machine-discovery surfaces across reproducible website corpora.

User-Agent

DiscoverGraphBot/0.4 (+https://discovergraph.net/crawler.php)

What it requests

The census profile normally requests the homepage, robots.txt, a sitemap candidate and a small fixed set of machine-readable discovery endpoints such as llms.txt, OpenAPI, ARD, MCP and agent-card metadata. It does not recursively crawl the entire site.

Politeness

DiscoverGraphBot respects robots.txt rules for its declared user agent, uses one active scan per host by default, applies a delay between requests, honors HTTP 429/Retry-After conservatively, and has a fixed per-domain request budget.

Data retained

The census stores structured observations, response metadata and SHA-256 content hashes. v0.4 does not build a permanent raw-HTML mirror of crawled sites.

Opt out / contact

Sites can disallow DiscoverGraphBot in robots.txt. Operational contact: crawler@discovergraph.net.