Moving agent orchestration rules from embedded code into a standalone CLI sounds like a clean separation of concerns. In practice, it exposes every assumption you made about state boundaries, failure modes, and trust perimeters. Francisco Grandón’s ag_ops v2.1 refactor documents four specific bugs that surfaced when a 1,031-line test guardrail became a 1,783-line governance engine, and the design pattern that emerged: synthetic scars.
What Broke in v1
The original ag_ops CLI enforced test integrity for multi-agent sessions by hash-checking test files and blocking modifications unless an agent explicitly called renegotiate --reason "...". The design assumed prose rationales would create friction. It did not.
Bug 1: Honor-system rationales
Any language model can generate a plausible 200-millisecond excuse. The --reason flag became a formality, not a gate. Comment feedback from @max_quimby highlighted the core issue: if an agent with shell access can unlock a Merkle-sealed test tree by passing a string argument, the security boundary is imaginary.
Bug 2: Exit codes without context
Witnessing a non-zero exit proves a command failed, not that the failure was informative or tied to the right invariant. The CLI could detect failure but not classify it. An agent might retry the wrong operation or escalate to a human without enough signal.
Bug 3: Single-file state collapse
As the tool gained automated tests, a changelog, and stricter contracts, the single-file Gist structure became a liability. State that belonged in separate modules (validation logic, audit logs, policy definitions) lived in one 1,031-line script. Refactoring required untangling implicit dependencies.
Bug 4: No failure memory
The CLI had no mechanism to remember past failure modes or encode defensive checks for bugs that had already been fixed. Every session started from zero institutional knowledge.
Synthetic Scars: Pre-Emptive Failure Handling
Randal L. Schwartz’s “synthetic scars” pattern addresses bug 4 directly. The idea: encode past production failures as permanent guardrails in the codebase, even if the original trigger is gone. Instead of waiting for a bug to recur, you install the fix prophylactically.
In ag_ops v2.1, synthetic scars manifest as:
- Explicit policy objects that encode why a rule exists, not just what it enforces
- Failure taxonomies that classify exit codes, timeouts, and permission errors into actionable categories
- Audit trails that persist across sessions, so an agent can see that a particular renegotiation pattern failed three weeks ago
The refactor grew the codebase from 1,031 to 1,783 lines (39,144 to ~68,000 bytes) because each scar requires metadata: the failure mode it prevents, the date it was added, and the test case that would catch regression.
Architecture: Rule Engine vs. Runtime Boundary
The v2.1 design splits orchestration into three layers:
| Layer | Responsibility | State Location | Failure Mode |
|---|---|---|---|
| CLI interface | Parse commands, validate syntax, return structured errors | Stateless (per-invocation) | Malformed arguments, missing flags |
| Policy engine | Evaluate rules, check hashes, enforce renegotiation contracts | JSON policy file + in-memory cache | Stale policy, hash mismatch, permission conflict |
| Agent runtime | Execute tool calls, manage conversation state, retry logic | LLM context + session log | Hallucinated commands, infinite retry loops, context overflow |
The critical boundary is between the policy engine and the agent runtime. The policy engine must be read-only from the agent’s perspective. If an agent can modify its own governance rules, the entire trust model collapses.
Enforcement Mechanism
v2.1 uses filesystem permissions and a separate policy-update CLI:
# Agent-facing CLI (read-only policy access)
def validate_test_modification(file_path, current_hash, reason):
policy = load_policy_readonly() # Fails if policy file is writable by agent user
if not policy.allows_renegotiation(reason):
return {"allowed": False, "scar": policy.get_scar_for_reason(reason)}
# ... hash validation, audit log append
# Operator-facing CLI (policy mutation)
def add_synthetic_scar(failure_mode, mitigation, test_case):
policy = load_policy_writable() # Requires elevated permissions
policy.scars.append(SyntheticScar(failure_mode, mitigation, test_case, date.today()))
policy.save_atomic()The agent user cannot write to the policy file. Policy updates require a separate credential or manual file edit. This creates a hard boundary: agents can request renegotiation, but they cannot redefine what “valid renegotiation” means.
What Changed in the Refactor
Before (v1):
- 1,031 lines, single Python file
- Prose rationales accepted at face value
- Exit codes logged but not classified
- No cross-session memory
After (v2.1):
- 1,783 lines, modular structure (CLI, policy engine, audit logger, test harness)
- Rationales matched against a failure taxonomy
- Exit codes mapped to retry strategies (transient vs. permanent failure)
- Synthetic scars persist in a versioned policy file
The growth is not bloat. Every additional line encodes a specific failure mode that occurred in production or was surfaced in comment threads.
Observability Gaps
The refactor introduced new failure modes:
Policy drift
If the policy file and the CLI version get out of sync (e.g., an operator manually edits the JSON), the agent sees cryptic validation errors. v2.1 adds a schema version field and rejects stale policies, but this creates a deployment dependency: you cannot upgrade the CLI without also migrating the policy file.
Audit log growth
Every renegotiation attempt appends to a JSON log. In a high-frequency multi-agent environment, this log can grow to hundreds of megabytes. The current design has no rotation or compaction strategy.
Scar proliferation
As the scar count grows, policy evaluation slows. Each renegotiation request must check against every scar. v2.1 uses a simple linear scan. At 100+ scars, this becomes a bottleneck.
When This Pattern Fits
Use synthetic scars when:
- You run agents in production and have a backlog of known failure modes
- Your orchestration rules change frequently enough that hardcoding them in the agent runtime is painful
- You need a human-readable audit trail of why certain guardrails exist
- You can enforce a read-only policy boundary via filesystem permissions or a separate credential
Avoid this pattern when:
- Your agent workload is exploratory or one-off (the overhead of maintaining a policy file is not justified)
- You cannot guarantee the policy file is immutable from the agent’s perspective
- Your failure modes are too diverse to classify (you will end up with a 10,000-line taxonomy)
- You need sub-millisecond policy evaluation (the current design does not optimize for latency)
Technical Verdict
The ag_ops v2.1 refactor shows that externalizing orchestration rules is not just a code organization exercise. It forces you to define the trust boundary between agent and governance layer, and to encode institutional knowledge (past bugs, edge cases, security assumptions) in a machine-readable format.
The synthetic scars pattern works when you have enough production history to justify the maintenance burden. If you are still discovering your failure modes, keep the rules embedded and extract them later. If you already know what breaks, encode it now before you forget why the guardrail exists.
The 752-line growth from v1 to v2.1 is the cost of making implicit assumptions explicit. That cost is worth paying if you plan to run the same orchestration rules across multiple agents, sessions, or teams. It is not worth paying if you are still figuring out what the rules should be.
Source Links
- Primary article: What Broke After I Moved My Agent’s Rules into a CLI
- ag_ops repository (referenced in article)