Cloudflare's Agentic Internet: Readable, Discoverable, Callable, Payable Protocols

Agents don’t render CSS. They don’t click ads. They don’t scroll. But they have a paying human on the other end, and when you block the agent, you block your customer.

Cloudflare is building open protocols to solve the collision between agents and publishers. The proposal centers on four primitives: readable (structured content), discoverable (agent-friendly sitemaps), callable (function endpoints), and payable (micropayment rails). This is infrastructure plumbing for a web where agents are first-class visitors, not just scrapers to block.

The Coordination Problem

Publishers built business models around HTML rendering and ad impressions. Agents bypass both. The current response is to block user agents that look like crawlers, but that breaks when your customer’s personal agent tries to fetch flight prices or compare product specs on their behalf.

The coordination problem has two sides:

• Publishers need a way to serve agents without cannibalizing ad revenue or giving away content to training runs.
• Agents need structured data, clear permissions, and a way to pay for access without pretending to be a browser.

Cloudflare’s approach is to extend existing web protocols rather than invent a parallel agent-only network.

The Four Primitives

Readable

Agents need structured content, not HTML soup. The readable primitive extends existing content negotiation to let agents request machine-friendly formats.

What it looks like:

• An agent sends Accept: application/json+ld or a similar structured MIME type.
• The server responds with JSON-LD, schema.org markup, or a custom structured format.
• Publishers control which fields are exposed and at what granularity.

Difference from existing approaches:

• robots.txt is binary (allow or deny).
• Open Graph and schema.org are embedded in HTML, requiring parsing.
• Readable content is a first-class response format, not a side effect of rendering.

Discoverable

Agents need to know what’s available before they request it. The discoverable primitive extends sitemap.xml with agent-specific metadata.

New metadata fields:

• Access tier: free, authenticated, paid.
• Rate limits: requests per minute, burst allowance.
• Capabilities: search, filter, sort, pagination.
• Data freshness: update frequency, cache TTL.

Example sitemap extension:

<url>
  <loc>https://example.com/api/flights</loc>
  <agent:access>paid</agent:access>
  <agent:rate-limit>10/min</agent:rate-limit>
  <agent:capability>search,filter</agent:capability>
  <agent:freshness>5m</agent:freshness>
</url>

This lets an agent’s orchestration layer decide whether to call the endpoint, cache the result, or route to a cheaper alternative.

Callable

Agents need to invoke functions, not scrape pages. The callable primitive exposes publisher functions as HTTP endpoints with typed inputs and outputs.

What it looks like:

• Publisher defines a function schema (OpenAPI, JSON Schema, or similar).
• Agent sends a POST request with typed parameters.
• Server returns a typed response or error.

Difference from REST APIs:

• Callable endpoints are explicitly marked for agent use in the discoverable sitemap.
• They include cost metadata (tokens, credits, dollars per call).
• They support agent-specific authentication (API keys, OAuth for agents, not humans).

Example callable endpoint:

{
  "endpoint": "/api/search-flights",
  "method": "POST",
  "input_schema": {
    "origin": "string",
    "destination": "string",
    "date": "ISO8601"
  },
  "output_schema": {
    "flights": "array<Flight>"
  },
  "cost": {
    "credits": 1,
    "usd": 0.01
  }
}

Payable

Agents need to pay for access without setting up a merchant account for every site. The payable primitive integrates micropayment rails into the protocol layer.

Where the wallet lives:

• Option 1: Agent harness holds credentials and signs payment requests.
• Option 2: Model provider (OpenAI, Anthropic) acts as payment intermediary.
• Option 3: Client-side wallet (browser extension, mobile app) authorizes payments.

Cloudflare’s proposal leans toward option 1, where the orchestration layer manages a payment token that gets attached to HTTP requests.

Payment flow:

1. Agent discovers a paid endpoint in the sitemap.
2. Orchestration layer checks budget and rate limits.
3. Agent sends request with Authorization: Bearer <payment-token>.
4. Publisher validates token, serves content, and settles via Cloudflare’s payment network.

Settlement options:

• Stripe Connect for fiat.
• Lightning Network for Bitcoin micropayments.
• Stablecoin transfers on L2 chains.

Architecture: How It Fits Together

┌─────────────┐
│   Agent     │
│  Harness    │
│             │
│ • Discovery │──┐
│ • Budget    │  │
│ • Wallet    │  │
└─────────────┘  │
                 │
                 ▼
         ┌───────────────┐
         │  Cloudflare   │
         │   Workers     │
         │               │
         │ • Route       │
         │ • Auth        │
         │ • Meter       │
         └───────────────┘
                 │
                 ▼
         ┌───────────────┐
         │  Publisher    │
         │   Origin      │
         │               │
         │ • Sitemap+    │
         │ • Callable    │
         │ • Payable     │
         └───────────────┘

Key integration points:

• Discovery: Agent fetches extended sitemap on first visit, caches locally.
• Routing: Cloudflare Workers intercept agent requests, validate payment tokens, enforce rate limits.
• Metering: Workers log usage and settle payments asynchronously.
• Fallback: If payment fails, Workers return 402 Payment Required with retry instructions.

Trade-Offs and Failure Modes

Component	Benefit	Risk
Readable	Structured data, no parsing	Publishers must maintain parallel formats
Discoverable	Agent can plan before calling	Sitemap bloat, stale metadata
Callable	Type-safe function invocation	Schema drift, versioning complexity
Payable	Micropayments without accounts	Payment token theft, settlement lag

Likely failure modes:

• Schema drift: Publisher updates callable endpoint but forgets to update sitemap. Agent sends old parameters, gets 400 Bad Request.
• Payment token theft: Attacker intercepts token, drains agent’s budget. Mitigation: short-lived tokens, rate limits, anomaly detection.
• Rate limit collisions: Multiple agents from the same IP hit the same endpoint. Publisher can’t distinguish between one agent making 100 requests and 100 agents making one request each.
• Stale discovery cache: Agent caches sitemap for 24 hours, misses new endpoints or price changes. Mitigation: cache TTL in sitemap metadata.

Security Boundaries

Agent authentication:

• Publishers issue API keys or OAuth tokens to agent harnesses, not end users.
• Tokens are scoped to specific endpoints and rate limits.
• Cloudflare Workers validate tokens before forwarding requests to origin.

Payment fraud:

• Payment tokens are signed by the agent’s wallet provider.
• Cloudflare verifies signatures and checks token balance before serving content.
• Publishers can set minimum payment thresholds to avoid dust attacks.

Data exfiltration:

• Readable content is rate-limited and metered.
• Publishers can mark fields as “preview only” (free) or “full access” (paid).
• Agents that exceed rate limits get throttled or banned.

Observability

What to instrument:

• Discovery latency: Time to fetch and parse extended sitemap.
• Callable success rate: 2xx vs 4xx vs 5xx responses.
• Payment settlement time: Lag between request and payment confirmation.
• Rate limit violations: How often agents hit limits, which endpoints.

Where to log:

• Cloudflare Workers log every agent request with payment status.
• Publishers log callable invocations with input/output schemas.
• Agent harnesses log discovery cache hits and payment approvals.

Deployment Shape

For publishers:

1. Add agent metadata to sitemap.xml.
2. Implement callable endpoints with typed schemas.
3. Integrate Cloudflare Workers for payment validation.
4. Set rate limits and pricing in Workers KV.

For agent developers:

1. Fetch extended sitemap on first visit.
2. Cache discovery metadata locally.
3. Attach payment tokens to requests for paid endpoints.
4. Handle 402 Payment Required and retry with valid token.

For Cloudflare:

• Workers act as the protocol enforcement layer.
• KV stores sitemap metadata and rate limit state.
• Durable Objects handle payment settlement and dispute resolution.

Technical Verdict

Use this when:

• You’re building an agent that needs to access paywalled or rate-limited content.
• You’re a publisher who wants to serve agents without breaking your business model.
• You need a standard way to discover, call, and pay for web functions.

Avoid this when:

• You control both the agent and the publisher (just use a private API).
• Your agent only needs public, unstructured content (scraping still works).
• You can’t afford the complexity of payment rails and schema versioning.

Open questions:

• Will publishers adopt extended sitemaps, or will they stick with robots.txt and blocking?
• Can micropayment rails scale to millions of agent requests per second?
• Who handles disputes when an agent pays but the publisher returns stale data?

Cloudflare is betting that open protocols beat proprietary agent networks. If they’re right, this becomes the plumbing layer for the agentic web. If they’re wrong, we get another spec that nobody implements.

Source Links

• Building an open Agentic Internet: readable, discoverable, callable, and payable (Cloudflare Blog, August 6, 2026)