Most agent benchmarks test whether a model can write a function or answer a question. Supabase Evals tests whether an agent can build a working application that spans database schemas, Row Level Security policies, Edge Functions, and authentication flows. The difference matters because integration work is where agents fail in production.
Supabase open-sourced this benchmark on August 1, 2026. It runs coding agents (Claude Code, Codex, OpenCode) against real Supabase tasks and scores how well they perform. The tasks are grounded in support tickets, bug reports, and GitHub issues, not synthetic puzzles.
Why Integration Evals Are Different
Traditional code generation benchmarks measure isolated function correctness. You give an agent a problem, it returns code, you run unit tests. Pass or fail.
Integration evals measure whether an agent can coordinate across multiple surfaces:
- Create a Postgres schema with foreign keys and constraints
- Define RLS policies that match the schema
- Deploy an Edge Function that queries the database
- Wire authentication so the function respects user context
- Debug when any of these steps break
An agent might generate syntactically correct SQL but fail to create a working app because it misunderstands how RLS policies interact with Edge Function execution context. That failure mode does not show up in HumanEval.
How Supabase Structures the Benchmark
Supabase defines scenarios across three dimensions:
- Product areas: Database, Auth, Edge Functions, Storage
- Cross-cutting topics: SDK usage, observability, error handling
- Builder journey stages: Scaffolding a new app, debugging a broken feature, fixing a security policy
Each scenario is the smallest reproducible task that touches at least one dimension. For example:
- “Build a schema for a multi-tenant SaaS app with RLS”
- “Debug why an Edge Function returns 401 for authenticated users”
- “Fix a broken RLS policy that leaks data across tenants”
The benchmark suite prioritizes breadth. It covers the real user journey with a small set of diverse scenarios. The regression suite prioritizes depth. It tracks specific known failure modes without influencing published scores.
The Eval Harness and Scoring
The harness runs each scenario against a real Supabase project. Agents interact through:
- Supabase CLI
- Client libraries (JavaScript, Python)
- MCP server
- Dashboard API
The harness does not mock responses. If an agent creates a table, that table exists in a live Postgres instance. If it deploys an Edge Function, that function runs in Deno Deploy.
Scoring is not binary. Supabase measures:
- Task completion: Did the agent finish the scenario?
- Correctness: Does the result match the spec?
- Partial success: Did the agent get the schema right but break auth, or vice versa?
Partial success scoring is critical. An agent might create a working database schema but deploy an Edge Function with the wrong environment variables. That is a different failure mode than an agent that cannot create the schema at all.
Failure Modes That Emerge
Supabase identified several patterns where agents struggle:
Context switching across tools: Agents that perform well with the CLI often fail when they need to use the dashboard API or client library in the same task. They lose state or make incorrect assumptions about how the tools interact.
RLS policy logic: Agents frequently generate policies that are syntactically valid but semantically wrong. For example, a policy that checks auth.uid() = user_id when the column is actually named owner_id.
Error recovery: When an Edge Function deployment fails, agents often retry the exact same code instead of reading the error message and adjusting. They lack a feedback loop between tool output and next action.
Non-determinism in tool selection: Given the same task, an agent might choose the CLI on one run and the client library on another. This makes debugging harder because the failure surface changes.
Architecture of the Eval System
The eval harness is a Python application that orchestrates agent runs and collects results. Here is the flow:
- Scenario loader: Reads scenario definitions from YAML files
- Project provisioner: Spins up a fresh Supabase project for each run
- Agent executor: Invokes the agent with the scenario prompt and tool access
- Result validator: Runs assertions against the live project state
- Scorer: Computes task completion, correctness, and partial success metrics
- Teardown: Destroys the project and logs results
The harness exposes a standard interface so Supabase can plug in different agents without rewriting the orchestration logic.
class AgentHarness:
def run_scenario(self, scenario: Scenario, agent: Agent) -> Result:
project = self.provision_project()
try:
agent_output = agent.execute(
prompt=scenario.prompt,
tools=self.get_tools(project),
max_steps=scenario.max_steps
)
validation = self.validate(project, scenario.assertions)
score = self.score(validation, agent_output)
return Result(score=score, logs=agent_output.logs)
finally:
self.teardown_project(project)The validate step runs SQL queries, HTTP requests, and CLI commands to check project state. For example, a schema validation might query information_schema.tables to verify that the agent created the right columns with the right constraints.
Benchmark vs. Regression Suites
| Dimension | Benchmark Suite | Regression Suite |
|---|---|---|
| Purpose | Measure breadth across product areas | Track specific known failures |
| Run frequency | Weekly or on major releases | Daily or on every commit |
| Scenario count | Small (10-20 scenarios) | Large (50+ scenarios) |
| Model coverage | Multiple models and harness configs | Single reference model |
| Published | Yes, on Supabase site | No, internal only |
| Stability | Stable, rarely changes | Evolves as new bugs are found |
The separation lets Supabase experiment with new scenarios in the regression suite without skewing benchmark scores. When a regression scenario proves stable and representative, it can graduate to the benchmark suite.
What This Means for Agent Builders
If you are building agents that integrate with third-party platforms, Supabase Evals is a template for how to test them. The key insight is that you cannot evaluate integration work with unit tests. You need:
- Live environments: Agents must interact with real APIs, not mocks
- Multi-step scenarios: Tasks that require coordinating across multiple tools
- Partial success scoring: Binary pass/fail hides important failure modes
- Regression tracking: Known failures should not pollute benchmark scores
The hardest part is defining scenarios. Supabase grounds theirs in support tickets and bug reports. That ensures the benchmark tests real problems, not hypothetical ones.
Observability and Debugging
The harness logs every tool call, API response, and agent decision. This is critical for debugging non-deterministic failures. When an agent fails a scenario, you need to see:
- Which tool it chose at each step
- What the tool returned
- How the agent interpreted the response
- Why it chose the next action
Supabase stores these logs in a structured format so they can query them later. For example, they can ask “how often do agents retry a failed Edge Function deployment without reading the error message?” and get a quantitative answer.
Security Boundaries
Each scenario runs in an isolated Supabase project. This prevents agents from interfering with each other or leaking data across runs. The harness provisions projects on demand and destroys them after the scenario completes.
The agent has full access to the project it is working on, but no access to the harness itself or other projects. This mirrors how agents will run in production: they control the resources they are building, but not the orchestration layer.
Technical Verdict
Use Supabase Evals as a model when:
- You are building agents that integrate with multi-surface platforms (database, auth, functions, storage)
- You need to measure whether agents can complete real tasks, not just write isolated code
- You want to track regression in agent performance as your platform or models change
- You need observability into why agents fail at integration work
Avoid this approach when:
- Your agent only interacts with a single API or tool (unit tests are simpler)
- You cannot afford to spin up live environments for each eval run
- Your tasks are deterministic and do not require multi-step orchestration
- You are optimizing for speed over realism (mocked evals run faster)
The benchmark is open source. If you are building agents that interact with Supabase, you can run it yourself to see where your agent struggles. If you are building agents for a different platform, you can fork the harness and adapt the scenarios.