Testing agents is not like testing functions. Functions take input, return output, and you assert on the result. Agents take input, call tools, maintain state, retry failures, and produce output along a path you cannot predict. When you build an eval harness for agents, you learn this the hard way.
Debashish Ghosal built AgentEval Forge to validate agent workflows across multiple orchestration frameworks. The project started as a scoring system and turned into a reality check on what it means to test non-deterministic execution. The harness supports five agent surfaces (subprocess, Python import, HTTP, LangGraph, PydanticAI), implements 28 scoring metrics, and enforces security boundaries to prevent ground-truth leakage. The real lessons came from watching agents break the clean assumptions.
Why Agent Evaluation Is Not Model Evaluation
Model evaluation is a closed loop. You send a prompt, get a completion, score the output. The model does not call external tools, does not retry, does not maintain state across turns. You can batch thousands of examples through a pipeline and aggregate metrics.
Agent evaluation is an open loop. The agent decides which tools to call, in what order, with what arguments. It can retry failed calls, adjust its plan mid-execution, or terminate early. Two runs with identical input can produce different valid outputs. You are not scoring an answer. You are scoring a run.
This changes the testing contract:
- Multiple valid paths: An agent might solve a task by calling tool A then tool B, or by calling tool C directly. Both paths can be correct.
- State pollution: If an agent maintains context across steps, test isolation becomes a problem. One test can leave state that affects the next.
- Retry logic: Agents retry failed tool calls. Your harness needs to distinguish between acceptable retries and infinite loops.
- Security boundaries: Agents should not see the expected answer or scoring thresholds during execution. Leaking ground truth into the agent’s context invalidates the eval.
Architecture of AgentEval Forge
The harness is built around five core components:
Scenario Pack Engine
Loads test cases from YAML or JSON. Each scenario defines input, allowed tools, disallowed tools, budget constraints, and expected outcomes. The engine validates the pack schema and surfaces parsing errors before execution.
Runner
Invokes the agent through one of five adapters. The adapter contract is thin: it receives a restricted payload (scenario input, tool allowlist, tool denylist, budget) and returns a trace. The agent never sees the expected answer.
Scoring Layer
Applies 17 deterministic checks (tool usage, budget compliance, output format) and 11 LLM-as-judge metrics (correctness, safety, coherence). Deterministic checks run first. LLM-as-judge metrics run only if deterministic checks pass, to save tokens.
Regression Engine
Compares current run against baseline. Flags regressions when scores drop below threshold or when tool usage patterns change unexpectedly.
Adversarial Generator
Mutates scenarios to generate edge cases: missing tools, corrupted input, budget exhaustion, conflicting constraints. Helps surface failure modes before production.
Adapter Contract and Invocation Payload
Each adapter implements the same contract:
class AgentAdapter:
def invoke(self, payload: InvocationPayload) -> AgentTrace:
pass
@dataclass
class InvocationPayload:
scenario_input: dict
allowed_tools: list[str]
disallowed_tools: list[str]
budget: Budget
@dataclass
class AgentTrace:
output: Any
tool_calls: list[ToolCall]
state_snapshots: list[StateSnapshot]
errors: list[Error]
duration_ms: intThe payload does not include the expected answer. The agent cannot see scoring thresholds. This prevents the agent from gaming the eval by pattern-matching against ground truth.
The trace captures everything: tool calls, state snapshots, errors, duration. The scoring layer uses the trace to compute metrics without re-running the agent.
State Isolation and Test Pollution
Agents that maintain state across steps can pollute subsequent tests. If test A leaves a variable in memory and test B assumes a clean slate, test B fails for the wrong reason.
AgentEval Forge enforces isolation at the adapter level:
- Subprocess adapter: Spawns a new process for each test. Process exit guarantees clean state.
- Python import adapter: Reloads the agent module before each test. Clears global state.
- HTTP adapter: Sends a reset signal to the agent endpoint before each test. Relies on the agent to honor the reset.
- LangGraph adapter: Creates a new graph instance for each test. Does not reuse checkpoints.
- PydanticAI adapter: Instantiates a new agent object for each test. Clears conversation history.
The HTTP adapter is the weakest link. If the agent endpoint does not implement proper reset logic, state leaks across tests. The harness logs a warning but cannot enforce isolation.
Retry Logic and Failure Modes
Agents retry failed tool calls. The harness needs to distinguish between acceptable retries and runaway loops.
The budget constraint includes a retry limit:
@dataclass
class Budget:
max_tool_calls: int
max_retries_per_tool: int
max_duration_ms: intThe runner tracks retries per tool. If an agent retries the same tool more than max_retries_per_tool times, the runner terminates the run and flags it as a failure.
This surfaces two common failure modes:
- Infinite retry loop: Agent retries a failing tool without adjusting arguments. Usually caused by poor error handling in the agent’s retry logic.
- Retry storm: Agent retries multiple tools simultaneously, exhausting the budget. Usually caused by missing backoff logic.
The trace includes retry counts per tool. The scoring layer penalizes excessive retries even if the agent eventually succeeds.
Scoring: Deterministic Checks vs. LLM-as-Judge
The scoring layer runs in two phases:
Phase 1: Deterministic Checks
Fast, cheap, no LLM calls. Checks tool usage, budget compliance, output format, security violations. If any check fails, the run is marked invalid and Phase 2 is skipped.
Phase 2: LLM-as-Judge
Slow, expensive, requires LLM. Evaluates correctness, safety, coherence, relevance. Only runs if Phase 1 passes.
| Metric Type | Examples | Cost per Run | Latency | False Positive Rate |
|---|---|---|---|---|
| Deterministic | Tool usage, budget, format | $0 | <10ms | ~0% |
| LLM-as-Judge | Correctness, safety, coherence | $0.01-$0.10 | 500-2000ms | 5-15% |
Deterministic checks catch 60-70% of failures. LLM-as-judge catches the rest but introduces latency and cost. The two-phase design keeps eval runs fast and cheap for the common case.
Security Model: Preventing Ground-Truth Leakage
The agent should not see the expected answer during execution. If the agent can pattern-match against ground truth, the eval is invalid.
AgentEval Forge enforces this at three layers:
Adapter layer: The invocation payload does not include the expected answer. The adapter cannot leak it.
Sandbox mode: When enabled, the runner intercepts tool calls and redacts any arguments that match the expected answer. This catches cases where the agent tries to exfiltrate ground truth through tool arguments.
Audit trail: The runner logs every tool call, every state snapshot, every error. The audit trail is append-only and tamper-evident. If ground truth leaks, the audit trail shows where.
API keys and secrets are sanitized before logging. The runner replaces them with placeholders.
Regression Detection: Comparing Runs
The regression engine compares the current run against a baseline. It flags regressions when:
- Score drops below threshold: Current run scores lower than baseline by more than the configured delta.
- Tool usage changes unexpectedly: Current run calls different tools or calls them in a different order.
- Budget usage increases: Current run uses more tool calls or takes longer than baseline.
The engine stores baselines in a versioned store. Each baseline is tagged with a commit hash, timestamp, and agent version. You can compare against the most recent baseline or against a specific version.
This catches two common regression patterns:
- Capability regression: Agent stops solving tasks it used to solve. Usually caused by prompt changes or model updates.
- Efficiency regression: Agent still solves tasks but uses more resources. Usually caused by inefficient tool selection or missing caching.
Adversarial Case Generation
The adversarial generator mutates scenarios to create edge cases:
- Missing tools: Removes a required tool from the allowlist. Tests whether the agent fails gracefully or retries indefinitely.
- Corrupted input: Injects malformed data into the scenario input. Tests input validation.
- Budget exhaustion: Sets budget limits below what the agent needs. Tests whether the agent respects limits.
- Conflicting constraints: Adds contradictory requirements to the scenario. Tests whether the agent detects conflicts.
The generator uses a mutation strategy inspired by fuzzing: it applies small, random changes to valid scenarios and checks whether the agent handles them correctly.
This surfaces failure modes that do not appear in hand-written test cases. It also helps build a corpus of adversarial examples for future testing.
What Broke When Real Agents Ran
The harness was designed for clean, well-behaved agents. Real agents broke several assumptions:
Assumption 1: Agents respect budget limits
Reality: Some agents ignore budget limits and keep calling tools until they hit an external timeout. The runner now enforces hard limits and terminates runs that exceed the budget.
Assumption 2: Agents produce structured output
Reality: Some agents return unstructured text instead of JSON. The scoring layer now includes a fallback parser that extracts structured data from text.
Assumption 3: Agents fail fast
Reality: Some agents retry indefinitely without backoff. The runner now tracks retry counts per tool and terminates runs that exceed the retry limit.
Assumption 4: Agents do not leak state
Reality: Some agents cache tool results in global state and reuse them across tests. The HTTP adapter now sends a reset signal before each test, but enforcement depends on the agent.
These breakages forced changes to the runner, the adapter contract, and the scoring layer. The harness is now more defensive and less trusting.
Technical Verdict
Use AgentEval Forge when:
- You are deploying agents in production and need regression detection.
- You are testing agents across multiple orchestration frameworks (LangGraph, PydanticAI, custom).
- You need to enforce security boundaries and prevent ground-truth leakage.
- You want to generate adversarial test cases automatically.
Avoid it when:
- You are testing simple LLM calls without tool use. Use a model eval framework instead.
- You need real-time eval feedback. The two-phase scoring design trades latency for cost.
- Your agents do not respect budget limits or retry constraints. The harness will terminate runs, but it cannot fix broken agent logic.
The real value is not the scoring metrics. It is the adapter contract, the state isolation primitives, and the adversarial generator. These are the pieces you need when deterministic test assumptions meet non-deterministic agent execution.