Georgia State University’s TReNDS research center built an agent that investigates production errors in under 60 seconds. The previous workflow required 15 to 30 minutes of manual log parsing, code repository searches, and hypothesis testing. The new pipeline runs on Amazon Bedrock and the open-source Strands Agents SDK, automating the entire root-cause analysis loop.
This is not a chatbot wrapper. The agent ingests error logs, generates hypotheses, queries code repositories, retrieves external knowledge, and iterates until it identifies the failure point. The architecture exposes useful patterns for anyone building agentic debugging tools: hypothesis-driven loops, tool boundary design, state management across multi-step investigations, and failure mode handling.
The Problem: Manual Root-Cause Analysis Does Not Scale
Production errors arrive faster than engineers can investigate them. A typical debugging session involves:
- Parsing stack traces and error messages
- Searching code repositories for relevant functions
- Correlating logs across services
- Consulting documentation or Stack Overflow
- Testing hypotheses by tracing execution paths
Each step requires context switching. Engineers lose time reconstructing mental models of distributed systems. TReNDS needed a system that could automate this loop without requiring human intervention for every error.
Architecture: Hypothesis-Testing Loop with Tool Orchestration
The TReNDS agent uses a structured investigation pipeline:
- Error ingestion: Production monitoring systems push error logs to the agent via API or message queue.
- Hypothesis generation: The agent uses a foundation model (via Bedrock) to generate initial hypotheses based on the error message, stack trace, and recent code changes.
- Tool invocation: The agent calls tools to test each hypothesis:
- Code repository search (GitHub, GitLab, or internal VCS)
- Log aggregation queries (CloudWatch, Datadog, or similar)
- External knowledge retrieval (documentation, Stack Overflow, internal wikis)
- Re-planning: If a hypothesis fails, the agent generates a new one and repeats the loop.
- Termination: The agent stops when it identifies a root cause or exhausts a predefined iteration budget.
The Strands Agents SDK handles orchestration. It provides a framework for defining tools, managing agent state, and controlling the hypothesis loop.
Tool Boundary Design
Each tool has a clear input/output contract:
| Tool | Input | Output | Failure Mode |
|---|---|---|---|
| Code search | Function name, file path, or error signature | Relevant code snippets with line numbers | Returns empty if code not indexed |
| Log query | Time range, service name, error pattern | Matching log entries with timestamps | Timeout if log volume exceeds threshold |
| Knowledge retrieval | Error message, stack trace keywords | Documentation links, similar issues | Returns generic results if no exact match |
The agent does not share state across tool calls. Each invocation receives only the data it needs. This prevents tools from leaking sensitive information (like production credentials) into the agent’s context.
State Persistence Across Investigations
The agent maintains a working memory that persists across hypothesis iterations:
- Error context: Original error message, stack trace, and metadata
- Hypothesis history: List of tested hypotheses and their outcomes
- Tool results: Cached responses from previous tool calls
Bedrock’s agent runtime stores this state in a session object. Each investigation gets a unique session ID. If the agent crashes mid-investigation, it can resume from the last checkpoint.
The SDK does not spawn isolated contexts for each hypothesis branch. All hypotheses share the same session memory. This reduces latency but introduces a risk: if the agent generates a bad hypothesis early, it may bias later iterations. TReNDS mitigates this by limiting the number of hypotheses the agent can test (typically five to seven per investigation).
Code Example: Defining a Tool in Strands Agents SDK
The SDK uses a decorator pattern to register tools:
from strands_agents import tool
@tool(
name="search_code_repository",
description="Search the codebase for functions or files matching a query",
parameters={
"query": {"type": "string", "description": "Search term (function name, file path, or error signature)"},
"max_results": {"type": "integer", "default": 10}
}
)
def search_code_repository(query: str, max_results: int = 10):
# Integration with GitHub API, GitLab, or internal VCS
results = vcs_client.search(query, limit=max_results)
# Return structured data for the agent to parse
return {
"matches": [
{
"file": r.file_path,
"line": r.line_number,
"snippet": r.code_snippet
}
for r in results
]
}The agent calls this tool when it needs to locate code related to an error. The SDK handles serialization, error handling, and retry logic.
Observability: Tracking Agent Decisions
TReNDS logs every hypothesis, tool call, and decision point. The observability stack includes:
- Bedrock CloudWatch metrics: Latency, token usage, and error rates per investigation
- Custom event logs: Hypothesis generation, tool invocation results, and termination reasons
- Trace IDs: Each investigation gets a unique ID that links logs, metrics, and agent state
Engineers can replay investigations by querying the trace ID. This is critical for debugging agent failures. If the agent misdiagnoses an error, engineers can inspect the hypothesis history and tool results to identify where the logic broke down.
Failure Modes and Human Override Paths
The agent fails in predictable ways:
- Hypothesis exhaustion: The agent tests all hypotheses without finding a root cause. TReNDS sets a hard limit of seven hypotheses per investigation. If the agent hits this limit, it escalates to a human engineer.
- Tool timeout: A tool call exceeds the timeout threshold (typically 10 seconds). The agent logs the failure and moves to the next hypothesis.
- Hallucinated root cause: The agent identifies a plausible but incorrect root cause. This is the hardest failure mode to detect automatically.
For hallucinated diagnoses, TReNDS relies on human feedback. Engineers can mark an investigation as incorrect via a web UI. The system stores this feedback and uses it to fine-tune the agent’s hypothesis generation logic.
The override path is simple: engineers can pause an investigation, inspect the agent’s work, and manually correct the diagnosis. The agent does not lock engineers out of the debugging process.
Deployment Shape
The TReNDS agent runs as a containerized service on AWS ECS. The deployment includes:
- Agent runtime: Strands Agents SDK + Bedrock API client
- Tool adapters: Lightweight containers that wrap external APIs (GitHub, CloudWatch, etc.)
- State store: DynamoDB table for session persistence
- Event queue: SQS queue for incoming error logs
The agent scales horizontally. Each container handles one investigation at a time. If error volume spikes, ECS spins up additional containers.
Security Boundaries
The agent has read-only access to code repositories and log aggregation systems. It cannot modify code, deploy changes, or access production databases.
Tool adapters enforce least-privilege access. The code search adapter can query GitHub but cannot push commits. The log query adapter can read CloudWatch logs but cannot delete or modify them.
Bedrock API calls use IAM roles with scoped permissions. The agent cannot invoke arbitrary AWS services. It can only call Bedrock’s InvokeModel and InvokeAgent APIs.
Performance: 15-30 Minutes to 60 Seconds
TReNDS measured investigation time before and after deploying the agent:
- Manual debugging: 15 to 30 minutes per error (median: 22 minutes)
- Automated agent: Under 60 seconds per error (median: 42 seconds)
The speedup comes from parallelizing tool calls and eliminating context switching. The agent does not need to reconstruct mental models or search documentation manually.
Technical Verdict
Use this approach when:
- You have high error volume and limited engineering time for manual debugging.
- Your codebase is well-indexed and searchable (the agent depends on accurate code search results).
- You can tolerate occasional misdiagnoses and have a human review process in place.
- You need to reduce mean time to resolution (MTTR) for production incidents.
Avoid this approach when:
- Your errors require deep domain knowledge that cannot be encoded in tools (e.g., hardware failures, network topology issues).
- Your codebase changes too frequently for the agent to maintain accurate context.
- You cannot afford false positives (e.g., safety-critical systems where a misdiagnosis could cause harm).
- Your team is too small to maintain the agent infrastructure (tool adapters, observability, feedback loops).
The TReNDS agent works because it automates a well-defined, repetitive task. It does not replace engineers. It gives them time to focus on problems that require human judgment.