Motorway, a UK car marketplace, ran into the classic agent evaluation problem: their AI agent worked fine in staging but produced incorrect results in 1 out of every 8 production queries. Worse, they only discovered issues hours after users hit them. AWS and Motorway built an evaluation pipeline that dropped the error rate to 1 in 50 and cut detection time to minutes. The architecture combines the Strands Agents SDK for execution tracing with Amazon Bedrock AgentCore for managed observability.
This is not a framework paper. It is a production deployment with real traffic, real failure modes, and concrete numbers.
The Evaluation Gap
Most agent evaluation happens in two places: pre-deployment synthetic tests and post-incident log analysis. The gap is production runtime. You need to know when an agent hallucinates a tool call, skips a reasoning step, or returns a plausible-sounding wrong answer while the user is still waiting.
Traditional monitoring catches crashes and timeouts. It does not catch semantic failures: the agent completed its loop, called the right tools, and returned garbage. You need instrumentation that exposes intermediate reasoning steps and a system that evaluates them in real time.
Architecture: Strands SDK + AgentCore
The pipeline has three layers:
Instrumentation layer (Strands SDK)
The SDK wraps agent execution to emit structured traces. Every tool call, reasoning step, and state transition becomes an event. The agent does not know it is being traced. You add the SDK to your agent runtime, configure trace destinations, and the agent continues to execute normally.
Observability layer (AgentCore)
AgentCore is AWS’s managed service for agent operations. It ingests traces from Strands, stores them in a queryable format, runs evaluation rules, and routes alerts. You define evaluation logic as code (Python functions that take a trace and return pass/fail), and AgentCore schedules them against incoming traces.
Evaluation layer (custom rules)
Motorway wrote evaluation functions that check:
- Tool call validity (did the agent call a tool that exists with valid parameters?)
- Reasoning coherence (does the final answer follow from the tool results?)
- Output format compliance (does the response match the expected schema?)
Each function runs asynchronously after the agent completes. If a rule fails, AgentCore fires an alert and logs the trace for review.
Instrumentation Without Breaking the Loop
The Strands SDK uses decorators and context managers to capture execution flow. Here is the pattern:
from strands import trace_agent, trace_tool
@trace_agent(name="vehicle_query_agent")
def run_agent(query: str) -> str:
reasoning = generate_plan(query)
tool_results = []
for step in reasoning.steps:
result = execute_tool(step.tool, step.params)
tool_results.append(result)
return synthesize_answer(tool_results)
@trace_tool(name="search_inventory")
def search_inventory(make: str, model: str, max_price: int):
# Tool implementation
return resultsThe decorators emit events to a local buffer. The SDK batches them and sends them to AgentCore asynchronously. The agent does not wait for trace delivery. If AgentCore is unreachable, traces queue locally and retry.
The key design choice: traces are append-only. The SDK never modifies agent state or execution flow. This prevents the observer effect where instrumentation changes behavior.
What AgentCore Actually Handles
AgentCore is not a black box. It provides:
- Trace storage: S3-backed storage with Athena query interface. Traces are partitioned by date and agent ID.
- Metric aggregation: Pre-built dashboards for latency, error rate, tool call distribution. You can define custom metrics with SQL.
- Evaluation scheduling: You upload Python functions as Lambda layers. AgentCore invokes them for each trace based on sampling rules (every trace, 10% sample, or conditional triggers).
- Alert routing: Integration with SNS, PagerDuty, Slack. You define alert rules in YAML.
You still write:
- The evaluation logic itself
- Custom metrics beyond the built-in set
- Integration with your incident management workflow
AgentCore does not interpret your traces semantically. It runs the code you give it.
Versioning and Regression Testing
Agents change constantly. The tool set expands, prompts get tuned, and the underlying model upgrades. Your evaluation pipeline needs to handle drift without breaking.
Motorway’s approach:
Version traces at capture time
Every trace includes metadata: agent version, model ID, prompt hash, tool schema version. When you query traces, you filter by version to compare behavior across changes.
Pin evaluation rules to agent versions
Evaluation functions are versioned separately. When you deploy agent v2, you can run both v1 and v2 evaluation rules against the same traces to detect regressions. If v2 passes its own rules but fails v1 rules, you know you changed behavior intentionally.
Regression test suite
Motorway maintains a golden set of 500 production traces. Before deploying a new agent version, they run it against the golden set and compare outputs. If more than 5% of traces change, they review manually.
This is expensive. Each regression run costs about $2 in model inference. They run it on every PR that touches agent code.
Failure Modes and Trade-offs
| Risk | Mitigation | Cost |
|---|---|---|
| Evaluation rule false positives | Shadow mode: log failures without alerting for 7 days, tune thresholds | Engineer time to review logs |
| Trace volume overwhelms storage | Sample 10% of traces, always capture failures | Miss rare edge cases in unsampled traces |
| Evaluation latency blocks deployment | Run evals async, deploy first, alert later | 2-5 minute detection delay |
| Model changes break eval assumptions | Version eval rules, run old + new rules in parallel | 2x evaluation compute cost during transition |
| AgentCore outage loses traces | Local trace buffer with 1-hour retention, replay on reconnect | Disk space on agent hosts |
The biggest operational surprise: evaluation rule maintenance. Motorway updates rules every 2-3 weeks as they learn new failure patterns. The rules are code, so they need testing, review, and deployment discipline.
Production Numbers
Motorway’s agent handles vehicle search queries. Before the eval pipeline:
- 12.5% incorrect results (1 in 8 queries)
- 2-4 hour detection time (manual user reports)
- 1-2 incidents per week
After:
- 2% incorrect results (1 in 50 queries)
- 3-8 minute detection time (automated alerts)
- 0.5 incidents per week
The error rate did not drop because of better evaluation. It dropped because fast feedback let them iterate on prompts and tool design. The eval pipeline is a forcing function for quality.
Cost: $0.03 per agent invocation for tracing and evaluation. At 10,000 queries per day, that is $300/day or $9,000/month. Motorway considers this cheap compared to the cost of incorrect inventory recommendations.
Technical Verdict
Use this architecture when:
- Your agent runs in production with real user traffic
- Semantic failures (wrong but plausible answers) are more common than crashes
- You need to detect issues in minutes, not hours
- You have engineering capacity to write and maintain evaluation rules
Avoid it when:
- Your agent is still in research/prototype phase (overkill)
- You have fewer than 100 agent invocations per day (manual review is cheaper)
- Your agent’s correctness is easily verified by deterministic checks (use unit tests)
- You cannot tolerate the 2-5 minute async evaluation delay (need synchronous validation)
The Strands SDK is open source. AgentCore is AWS-only. If you are not on AWS, you can replicate the architecture with Langfuse or Helicone for observability and your own Lambda-equivalent for evaluation scheduling. The hard part is not the infrastructure. It is writing evaluation rules that catch real failures without drowning you in false positives.