Fireworks AI Breach at Notion: What Third-Party LLM Vendor Compromise Teaches About Agent Trust Boundaries

Notion disclosed a security incident at Fireworks AI, one of their third-party inference providers, that exposed unattributable chat sessions from fewer than 1% of workspaces. The breach is a clean example of vendor supply-chain risk in agentic systems: when your orchestrator routes user context to an external LLM API, you inherit that vendor’s security posture and blast radius.

The incident matters because it exposes a trust boundary most teams don’t instrument. Prompt injection and sandbox escapes get attention, but the data flow to inference providers is often invisible to observability stacks and incident response playbooks.

What Happened

Fireworks AI suffered a security breach. Notion used Fireworks as one of several inference providers for Notion AI chat sessions. The exposed data included:

Chat transcripts (user prompts and model responses)
No attribution metadata (no workspace IDs, user IDs, or company identifiers)
No tool call results (function outputs, database queries, or API responses)
No other workspace data

Notion’s investigation found that the sessions were unattributable, meaning an attacker could read the chat content but couldn’t tie it back to a specific workspace or user without additional context clues in the conversation itself.

The Trust Surface You Don’t See

When an agent calls an external LLM API, the request typically includes:

User prompt (often containing PII, business logic, or sensitive context)
System prompt (instructions, tool schemas, examples)
Conversation history (multi-turn context)
Tool call results (database rows, API responses, file contents)
Metadata (session IDs, user IDs, workspace identifiers)

Most orchestrators send all of this to the inference provider in plaintext. The provider logs it for debugging, rate limiting, abuse detection, and model improvement. If the provider is breached, all of that context is exposed.

Notion’s architecture stripped attribution metadata before sending requests to Fireworks, which limited the blast radius. But the chat transcripts themselves still leaked, and any sensitive information in the prompts or responses went with them.

Architectural Patterns for Containment

Here are the plumbing decisions that determine blast radius when an inference provider is compromised:

1. Credential Isolation

Each external service (LLM API, vector database, tool endpoint) should use scoped credentials that can’t pivot to other services.

# Bad: single API key with broad access
llm_client = LLMClient(api_key=os.environ["MASTER_API_KEY"])

# Good: scoped token per session, revocable
session_token = credential_manager.issue_scoped_token(
    service="fireworks",
    scope=["inference"],
    ttl_seconds=3600,
    metadata={"session_id": session_id}
)
llm_client = LLMClient(api_key=session_token)

If Fireworks is breached and an attacker extracts API keys from their database, scoped tokens limit what they can do with those keys.

2. Data Minimization at the Boundary

Strip attribution metadata and sensitive context before sending requests to external providers.

Data Type	Send to Provider?	Why
User prompt	Yes	Required for inference
System prompt	Yes	Required for behavior
Tool schemas	Yes	Required for function calling
Tool call results	No	Can be summarized or redacted
User ID	No	Not needed for inference
Workspace ID	No	Not needed for inference
Session ID	Maybe	Only if needed for debugging

Notion’s decision to send unattributable sessions meant the breach couldn’t be traced back to specific workspaces without manual correlation. That’s a meaningful reduction in risk.

3. Observability for Anomaly Detection

You need telemetry that can detect when an external LLM API starts behaving strangely:

Response latency spikes (possible exfiltration overhead)
Unexpected error rates (possible credential rotation or blocking)
Unusual request patterns (possible replay attacks)
Model output drift (possible model swap or poisoning)

@observe(span_name="llm_inference")
def call_external_llm(prompt: str, provider: str):
    start = time.time()
    response = llm_client.complete(prompt)
    latency = time.time() - start
    
    metrics.record("llm.latency", latency, {"provider": provider})
    metrics.record("llm.tokens", response.usage.total_tokens, {"provider": provider})
    
    if latency > P99_LATENCY * 2:
        alerts.trigger("llm_latency_anomaly", {"provider": provider, "latency": latency})
    
    return response

Most teams instrument their own services but treat external LLM APIs as black boxes. That’s a blind spot.

4. Multi-Provider Routing with Failover

If you route requests across multiple inference providers, you can fail over when one shows signs of compromise.

providers = [
    {"name": "fireworks", "weight": 0.5, "healthy": True},
    {"name": "openai", "weight": 0.3, "healthy": True},
    {"name": "anthropic", "weight": 0.2, "healthy": True},
]

def route_request(prompt: str):
    healthy_providers = [p for p in providers if p["healthy"]]
    if not healthy_providers:
        raise NoHealthyProvidersError()
    
    provider = weighted_random_choice(healthy_providers)
    
    try:
        return call_external_llm(prompt, provider["name"])
    except Exception as e:
        mark_unhealthy(provider["name"])
        return route_request(prompt)  # retry with different provider

This requires you to normalize tool schemas and system prompts across providers, which is non-trivial but pays off when one provider has an incident.

What Tool Call Results Reveal

Notion’s disclosure notes that tool call results were not included in the exposed sessions. That’s significant because tool outputs often contain the most sensitive data:

Database query results (customer records, financial data)
API responses (third-party service data, authentication tokens)
File contents (documents, spreadsheets, code)

If your orchestrator sends tool call results back to the inference provider for the next turn of reasoning, you’re expanding the attack surface. Consider these alternatives:

Summarize tool outputs: Send only the fields the model needs, not the full response.
Redact sensitive fields: Strip PII, credentials, and business-critical data before sending.
Local reasoning: Run the next reasoning step locally instead of sending tool outputs to the external provider.

# Bad: send full tool output to external LLM
tool_result = database.query("SELECT * FROM customers WHERE id = 123")
next_prompt = f"Here's the data: {tool_result}. What should I do next?"
response = llm_client.complete(next_prompt)

# Good: summarize and redact
tool_result = database.query("SELECT name, email FROM customers WHERE id = 123")
summary = f"Found customer record with name and email."
next_prompt = f"{summary}. What should I do next?"
response = llm_client.complete(next_prompt)

Incident Response for Vendor Breaches

When a third-party inference provider is breached, your incident response playbook should include:

Identify affected sessions: Which requests went to the compromised provider during the breach window?
Assess data exposure: What sensitive data was in those requests (prompts, tool outputs, metadata)?
Revoke credentials: Rotate API keys and tokens that the provider had access to.
Notify affected users: If attribution metadata was exposed, notify the specific users. If not, decide whether to notify all users as a precaution.
Fail over to alternate providers: Route traffic away from the compromised provider until they publish a post-mortem and remediation plan.

Most teams don’t have step 1 instrumented. If you don’t log which provider handled each request, you can’t scope the incident.

The Broader Supply-Chain Problem

This incident is part of a larger pattern: agentic systems have more supply-chain dependencies than traditional software.

Inference providers: OpenAI, Anthropic, Fireworks, Together, Replicate
Vector databases: Pinecone, Weaviate, Qdrant, Chroma
Tool endpoints: Slack API, GitHub API, Stripe API, internal microservices
Observability platforms: LangSmith, Helicone, Weights & Biases

Each dependency is a potential breach point. Each one sees some subset of your user data. Each one has its own security posture, incident response capability, and legal liability.

The traditional approach (vendor security questionnaires, SOC 2 audits, contract terms) doesn’t move fast enough for the AI supply chain. You need runtime defenses:

Scoped credentials that limit blast radius
Data minimization at every boundary
Observability that detects anomalies in real time
Failover routing that doesn’t depend on vendor uptime or trustworthiness

Technical Verdict

Use third-party inference providers when:

You need model variety or cost optimization across multiple providers
You can strip attribution metadata and sensitive context before sending requests
You have observability and failover routing in place
Your threat model accepts the risk of vendor breach (with containment)

Avoid third-party inference providers when:

Your data is subject to strict regulatory controls (HIPAA, GDPR, financial services)
You can’t instrument the boundary or detect anomalies
You don’t have the engineering capacity to build multi-provider routing
Your users expect zero third-party data sharing

The Fireworks breach at Notion is a clean example of supply-chain risk in agentic systems. The good news is that Notion’s architecture (unattributable sessions, no tool call results) limited the blast radius. The bad news is that most teams don’t have those defenses in place, and the next breach might not be so contained.

Agentic AI, minus the noise.