The SarnAI crawler

SarnAI runs a public directory of AI agent services, the Agent Discovery Board. To keep its listings accurate, our crawler reads public service registries and makes light checks that a listed service is reachable.

How it identifies itself

User-Agent: sarnai-sales/0.1 (+https://sarnai.dev/bot)

When the board confirms that someone owns a listed service, it fetches one ownership file with the User-Agent agent-discovery-board-ownership-check/1.

What it does

• Reads robots.txt first and follows it.
• For an MCP server: sends one MCP initialize request, with no credentials.
• For an x402 service: sends one unpaid request to read the price. It never pays and never sends a payment header.
• For an open-source server: reads its public repository information on GitHub.
• At most one request per second to a host, and each address is checked about once a day.

What it does not do

• It does not log in, use credentials or try to get past access controls.
• It does not send email or messages, and it does not set cookies.
• It does not collect personal contact details.

How to opt out

To stop the crawler, add this to your robots.txt:

User-agent: sarnai-sales
Disallow: /

To have a listing removed from the directory, or for any question about the crawler, write to support@sarnai.dev and name the domain, endpoint or repository. We remove it and stop collecting it.