When Replit’s coding agent deleted a production database, the incident report explained what happened but not how to prevent it from happening again. News coverage captures the outcome. Incident catalogs collect the stories. But neither rebuilds the execution path or tests whether the blamed root cause actually matters.
ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation) is a multi-agent framework that turns incident reports into reproducible experiments. It reconstructs the original task environment, runs the agent with and without the suspected cause, and measures whether the harm reappears. Across 108 attempted replications, the framework reproduced the reported harm in 89% of cases.
The Reproducibility Gap
Most agent observability stops at logs. You get timestamps, tool calls, and maybe token counts. You do not get enough context to replay the decision tree or test whether a proposed fix actually works.
What is missing:
- Task state snapshots. The file tree, database schema, or API responses the agent saw at decision time.
- Deterministic replay. LLM sampling is stochastic, so you cannot rerun the same prompt and expect the same tool sequence.
- Control conditions. Without a matched baseline (same task, minus the hazard), you cannot isolate the cause.
ROGUE-AI addresses all three by building matched pairs: one replication with the blamed cause present, one without. If the agent fails in the first case and succeeds in the second, the cause is validated.
Event Capture Architecture
The framework captures five primitives per agent run:
| Primitive | What It Stores | Why It Matters |
|---|---|---|
| Task description | Original user prompt or goal | Defines success criteria |
| Tool inventory | Available functions and schemas | Bounds the action space |
| Environment snapshot | File system, database state, API mocks | Makes replay deterministic |
| Execution trace | Tool calls, arguments, return values | Shows the decision path |
| Hazard annotation | The suspected cause (e.g., shared DB) | Enables control condition |
Sensitive credentials are replaced with placeholder tokens. API responses are recorded once and replayed from disk. File operations target a sandboxed copy of the original tree.
Replay Without Secrets
Audit logs must preserve enough context to reproduce behavior without leaking API keys or database passwords. ROGUE-AI uses a token substitution layer:
class SecureReplayContext:
def __init__(self, original_trace):
self.token_map = {}
self.sanitized_trace = self._replace_secrets(original_trace)
def _replace_secrets(self, trace):
for event in trace:
if event['type'] == 'tool_call':
for key in ['api_key', 'password', 'token']:
if key in event['args']:
placeholder = f"REDACTED_{hash(event['args'][key])[:8]}"
self.token_map[placeholder] = event['args'][key]
event['args'][key] = placeholder
return trace
def replay(self, agent):
# Inject token_map into agent's credential resolver
agent.credential_resolver.load_placeholders(self.token_map)
for event in self.sanitized_trace:
agent.execute_tool(event['tool'], event['args'])
The credential resolver intercepts placeholder tokens at runtime and swaps in real credentials only inside the execution sandbox. The audit log never contains plaintext secrets.
Matched Control Design
To validate a root cause, ROGUE-AI builds two environments:
- Replication environment: Includes the suspected hazard (e.g., a database shared with production).
- Control environment: Identical except the hazard is removed (e.g., database points to a staging instance).
Both environments run the same agent with the same task prompt. The framework measures harm (files deleted, payments sent, data exposed) in each case.
Results across five models (GPT-4, Claude 3, Gemini, DeepSeek):
- Agents made the reported mistake in 18% to 33% of replication runs.
- Control runs showed harm in less than 1% of cases.
- 62 of 72 real-world incidents involved no attacker. The agent did what the task asked but missed an ordinary hazard.
Failure Modes and Observability
The framework identified three common patterns:
1. Shared Resource Confusion
The agent receives a task like “deploy the new feature” and correctly identifies the deployment script. But the script points to a production database because the environment variable was not overridden. The agent runs the script, and production data is deleted.
Observability gap: Logs show the tool call (run_script deploy.sh) but not the environment context (which database the script targets).
ROGUE-AI capture: Snapshots the environment variables and database connection strings at task start. Replay injects a staging database URL in the control condition.
2. Implicit Permissions
The agent has file system access and is asked to “clean up old logs.” It identifies /var/log/app as the target directory but does not check whether /var/log/app is a symlink to /var/log/system. Deleting the directory removes system logs.
Observability gap: Tool schemas define parameters (path, recursive) but not permission boundaries or symlink resolution.
ROGUE-AI capture: Records the file system graph, including symlinks and mount points. Replay runs in a chroot jail where symlinks are rewritten to point to safe targets.
3. Cascading Tool Calls
The agent is asked to “send a payment reminder.” It calls get_customer_email, then send_email. But get_customer_email returns a test account in staging and a real customer in production. The agent does not check which environment it is running in.
Observability gap: Each tool call succeeds individually. The harm emerges from the sequence.
ROGUE-AI capture: Traces the full call graph and annotates each tool with its environment context. Replay injects a mock email service in the control condition.
Defense Evaluation
The paper tested two interventions:
- Hazard-aware warnings: After each tool output, inject a message like “Warning: This database is shared with the live site. Deleting data will affect production.”
- Generic warnings: Same length, but generic: “Warning: This operation may have side effects. Proceed with caution.”
Hazard-aware warnings reduced harm for all five models. Generic warnings left most of the harm in place. This suggests that agents can use explicit hazard annotations if the orchestration layer provides them.
Deployment Shape
ROGUE-AI is designed as a post-incident audit tool, not a runtime monitor. The replay infrastructure is too slow for production (each replication takes minutes to hours). But the framework can run as part of a release pipeline:
- Incident ingestion: Parse incident reports from news, bug trackers, or support tickets.
- Environment synthesis: Rebuild the task context from logs, screenshots, or user descriptions.
- Batch replay: Run all replications overnight before each model release.
- Regression detection: Flag any incident that reappears in the new model.
The framework outputs a pass/fail report for each incident, plus a diff showing which tool calls changed between the old and new model.
Trade-Offs: Full Replay vs. Symbolic Trace
Full execution replay is expensive. You need to snapshot the entire environment, mock all external APIs, and rerun the agent from scratch. Symbolic trace reconstruction is cheaper: you replay only the tool call sequence, not the LLM reasoning.
| Approach | Cost | Fidelity | Use Case |
|---|---|---|---|
| Full replay | High (minutes per run) | Exact | Root cause validation |
| Symbolic trace | Low (seconds per run) | Approximate | Regression testing |
| Hybrid | Medium | Good enough | CI/CD pipelines |
Hybrid replay runs the tool sequence symbolically but invokes the LLM only at decision points where the trace diverges. This cuts replay time by 80% while preserving most of the fidelity.
Technical Verdict
Use ROGUE-AI when:
- You need to validate incident reports before publishing a postmortem.
- You are building a release pipeline for agentic systems and want regression tests based on real failures.
- You are debugging a production incident and need to isolate the root cause from confounding factors.
Avoid it when:
- You need real-time monitoring. Replay is too slow for production traffic.
- Your agent does not interact with external systems. If there are no tool calls, there is nothing to replay.
- You cannot snapshot the environment. If the task depends on live APIs or ephemeral state, replay will not be deterministic.
The framework works best for agents with well-defined tool boundaries and reproducible environments. It struggles with agents that depend on real-time data or human-in-the-loop decisions.