Knowing who operates a bot is not the same as knowing whether it will behave responsibly. Just as a driver's license confirms identity and competence but not future conduct, bot verification establishes accountability without guaranteeing ongoing compliance. Even identified bots can disregard site preferences, flood origins with traffic, or work around controls—sometimes through malice, but often through misconfiguration or engineering mistakes.
Cloudflare's Verified Bot program asks two fundamental questions: "Are you who you say you are?" and "Can we trust you?" Bots meeting the program's criteria must identify themselves accurately and securely using Web Bot Auth, do only what they claim to do, and avoid abusive behavior. With identity verification handled through Web Bot Auth and BotBase's open catalog, the focus now shifts to the second requirement: ensuring verified bots cause no harm.
Today Cloudflare is introducing a monitoring system that tracks whether Verified bot behavior remains trustworthy over time, answering the question "Can we trust you?" with continuous re-verification rather than a single judgment. The company de-listed Perplexity's bot from its Verified list in 2025 after the bot attempted to access resources on websites that had opted out of scraping and subsequently tried to circumvent controls when blocked.
Do not engage in abusive behavior
The distinction between risk and trust matters here. Risk reflects the probability that a specific request or action at a given moment may cause harm—it fluctuates constantly. Trust accumulates through repeated, consistent behavior over extended periods.
Cloudflare developed a measurement system to track how frequently crawlers respect site preferences. Rather than rendering permanent verdicts, the system produces an explainable trust band—a range showing how consistently a bot honors site preferences and which behaviors shaped that assessment. The initial metrics evaluated are whether the bot follows instructions and whether it responds to direct feedback. These represent basic courtesy toward the operators providing the information bots consume.
Cloudflare plans to publish these scores publicly on Radar and expand the measurement methods over time. This approach gives responsible bot operators a way to demonstrate sustained trustworthiness. The principle is straightforward: verify once, then verify continuously.
When in Rome, do as the Romans do
The robots.txt file remains the primary mechanism for website owners to communicate their preferences. Cloudflare's system checks whether a bot's observed traffic aligns with published preferences over time.
A robots.txt file instructs all bots or specific agents to crawl only designated sections of a website. Isolated breaches and one-time exceptions may not signal a problem. What matters is detecting repeated patterns of disregarding site preferences. The expectation is straightforward: responsible bots should consult and honor the preferences websites publish, consistent with RFC 9309.
A robots.txt establishes minimum rules for what bots should not do. Nothing unstated should be assumed as permitted. For instance, a robots.txt need not explicitly forbid access when a site communicates through other signals that continued visits are unwelcome.
What you do not wish for yourself, do not impose on others
Bots can stay within permitted paths while still imposing substantial load on target websites. HTTP 429 ("Too Many Requests") responses provide the clearest signal for pushback. A 429 directs the visitor to reduce its request rate, sometimes specifying how long to wait before retrying.
Cloudflare's system observes whether bot traffic adjusts after receiving that signal. Does the bot reduce speed, pause, or continue at the same rate? This behavior feeds into the bot's trust band over time: consistently slowing down or pausing raises trust, while repeatedly ignoring the signal lowers it.
Building trust over time
Monitoring these signals represents only the beginning of holding verified automation accountable. By linking bot identity to evidence of ongoing behavior, Cloudflare can give site owners greater confidence that allowed bots respect their wishes, give bot operators a way to demonstrate they merit trust, and provide website owners context about the automated traffic reaching them.
This system works alongside Bot Preference Sync and Bot Management to create an accountability loop for automated traffic. Bot Preference Sync helps website owners publish their preferences; observing how bots interact with those preferences reveals whether Verified bots consistently honor them. Bot Management lets website owners determine how that traffic is handled. Together, these tools ensure trust is earned through transparent, accountable behavior rather than assumed from identity alone.
Cloudflare is already using this system internally to evaluate how consistently Verified bots honor website preferences. Since trust develops over time, the company assesses sustained patterns rather than isolated incidents. This initial phase builds historical data and refines how observations become useful guidance. Cloudflare plans to make the system available to all customers, who will be able to set the trust threshold for handling Verified bot traffic. The goal is to keep website preferences meaningful and ensure trust is earned through sustained behavior.
Cloudflare customers can already use Bot Management to control automated traffic and Bot Preference Sync to align published preferences with configured policies. The growing Verified bot population and related trends are visible on Radar. Those interested in contributing to the future of bot identity and accountability can explore open positions at Cloudflare.