Every team building multi-agent systems hits the same wall. An agent fails. You open the logs. Was it a bad tool boundary, missing context, or broken orchestration? Or did the LLM just hallucinate? The first is architectural debt you can refactor. The second is inherent probabilistic noise you can only mitigate.
A new ArXiv paper (2605.27320v1) formalizes this distinction and gives it names: Agentic Technical Debt (deterministic, fixable architecture) and Stochastic Tax (recurring probabilistic variance). The framework provides a measurement model, simulation tooling, and dashboard structure to separate the two failure modes.
The Core Distinction
Agentic Technical Debt is a stock. It accumulates when you:
- Design tool boundaries that leak state
- Omit necessary context from agent memory
- Build orchestration flows that skip validation steps
- Hardcode business logic inside prompt templates
- Ignore governance policies for tool access
This is the stuff you can fix. Refactor the orchestration graph. Add a context retrieval step. Tighten the tool schema. The debt is deterministic. Given the same inputs, the same architectural flaw will trigger the same failure.
Stochastic Tax is a flow. It recurs every time you invoke an agent because LLMs are probabilistic. Even with perfect architecture, you pay:
- Hallucination risk on every tool call
- Prompt drift when context windows shift
- Model variance across API versions
- Latency jitter from inference load
- Token cost volatility
You cannot eliminate stochastic tax. You can only measure it, budget for it, and decide whether the probabilistic burden is worth the automation gain.
The two constructs interact. High technical debt amplifies stochastic tax (a broken context window makes hallucinations worse). But stochastic tax remains positive even when debt is zero. A perfectly architected agent still hallucinates.
Instrumentation Strategy
To measure these separately, you need telemetry at three layers:
1. Orchestration Layer
Track deterministic failures:
- Tool call rejections (schema mismatch, auth failure)
- Context retrieval misses (empty memory, stale cache)
- Workflow timeouts (missing handoff, circular dependency)
- Validation errors (output schema violation, constraint breach)
These are architectural. If you see the same tool call fail with the same error across multiple runs, that is debt.
2. Model Layer
Track probabilistic failures:
- Hallucinated tool names or parameters
- Inconsistent outputs for identical prompts
- Confidence score variance (if your model exposes it)
- Retry counts for non-deterministic errors
These are stochastic. If the same prompt sometimes works and sometimes fails, that is tax.
3. Business Layer
Track outcome variance:
- Task completion rate (did the agent finish?)
- Human-in-the-loop intervention rate (did someone have to fix it?)
- Downstream error propagation (did a bad output break the next step?)
- Cost per successful task (tokens, retries, manual cleanup)
This is where debt and tax compound. A high intervention rate might mean bad architecture (debt) or noisy outputs (tax). You need the first two layers to decompose it.
Dashboard Structure
The paper proposes splitting observability into two panels.
Agentic Technical Debt Dashboard
| Metric | What It Measures | Signal |
|---|---|---|
| Tool schema violation rate | Malformed calls per 1k invocations | Bad tool boundaries |
| Context miss rate | Retrieval failures per agent run | Insufficient memory design |
| Workflow timeout rate | Orchestration deadlocks per task | Missing handoff logic |
| Validation failure rate | Output schema breaches per step | Weak output constraints |
| Governance override rate | Policy violations per tool call | Missing access controls |
These metrics should trend toward zero as you refactor. If they plateau, you have unresolved architectural debt.
Stochastic Tax Dashboard
| Metric | What It Measures | Signal |
|---|---|---|
| Hallucination rate | Invalid outputs per prompt | Model noise floor |
| Prompt consistency score | Output variance for identical inputs | Inference stability |
| Retry rate | Non-deterministic failures per task | Probabilistic overhead |
| Token cost variance | Spend deviation per task type | Model pricing volatility |
| Latency jitter (p99) | Inference time variance | API load sensitivity |
These metrics have a floor. You cannot drive hallucination rate to zero. You can only decide whether the tax is acceptable for the task.
Implementation Pattern
Here is a minimal instrumentation wrapper for a tool-calling agent:
import time
from collections import defaultdict
from typing import Any, Dict
class AgentTelemetry:
def __init__(self):
self.debt_counters = defaultdict(int)
self.tax_counters = defaultdict(int)
self.task_outcomes = []
def record_tool_call(self, tool_name: str, params: Dict,
result: Any, error: str = None):
# Deterministic failures = debt
if error and "schema" in error.lower():
self.debt_counters["schema_violations"] += 1
elif error and "auth" in error.lower():
self.debt_counters["auth_failures"] += 1
# Probabilistic failures = tax
elif error and "hallucination" in error.lower():
self.tax_counters["hallucinations"] += 1
def record_context_retrieval(self, query: str,
results: list, latency_ms: float):
if not results:
self.debt_counters["context_misses"] += 1
# Track latency variance (stochastic)
self.tax_counters["retrieval_latency_samples"].append(latency_ms)
def record_task_outcome(self, task_id: str, success: bool,
retries: int, human_intervention: bool):
self.task_outcomes.append({
"task_id": task_id,
"success": success,
"retries": retries, # Stochastic signal
"human_intervention": human_intervention # Compound signal
})
def get_debt_score(self) -> float:
"""Deterministic failure rate"""
total_calls = sum(self.debt_counters.values())
if total_calls == 0:
return 0.0
return total_calls / len(self.task_outcomes)
def get_tax_score(self) -> float:
"""Probabilistic overhead rate"""
total_retries = sum(t["retries"] for t in self.task_outcomes)
if not self.task_outcomes:
return 0.0
return total_retries / len(self.task_outcomes)This pattern separates deterministic errors (schema violations, auth failures, context misses) from probabilistic overhead (hallucinations, retries, latency variance). You can extend it to track workflow timeouts, validation failures, and governance overrides.
When Debt Amplifies Tax
The paper includes an accounts-payable simulation showing how architectural debt compounds stochastic tax. The scenario:
- Agent extracts invoice data, calls a payment API, logs the transaction
- Debt scenario: Missing context window means the agent cannot see prior invoices, so it hallucinates duplicate payments
- Tax scenario: Even with full context, the LLM occasionally misreads invoice amounts
In the simulation, fixing the context window (reducing debt) cuts the hallucination rate by 60%. But the remaining 40% is irreducible stochastic tax. You cannot eliminate it without swapping the model or adding a deterministic validation step.
The key insight: refactoring architecture reduces both debt and tax. But at some point, you hit a floor where only probabilistic mitigation (retries, voting, human review) helps.
Deployment Decision Tree
The framework changes how you respond to failures:
High debt, low tax: Refactor orchestration. Add context. Tighten tool schemas. The architecture is broken.
Low debt, high tax: Tune prompts. Swap models. Add validation layers. The architecture is fine, but the LLM is noisy.
High debt, high tax: Fix architecture first. Debt amplifies tax, so you cannot measure the true tax floor until you clean up the orchestration.
Low debt, low tax: Ship it. Monitor for drift.
Observability Boundaries
The paper does not prescribe specific tooling, but the instrumentation pattern maps cleanly to existing observability stacks:
- Orchestration layer: OpenTelemetry spans for tool calls, context retrieval, workflow steps
- Model layer: LangSmith, Weights & Biases, or custom LLM loggers for prompt/output pairs
- Business layer: Application metrics (task success rate, intervention rate, cost per task)
The key is tagging each event with a failure mode (debt vs. tax) so you can split dashboards and set different SLOs.
Likely Failure Modes
Misattributing tax as debt: You see high retry rates and assume the orchestration is broken. You refactor. Nothing improves. The real issue is model variance. Always check if the failure is deterministic (same input, same error) before refactoring.
Ignoring debt because tax is high: You accept high hallucination rates as “just how LLMs work” and skip architectural fixes. Debt compounds tax. Fix the context window first.
Dashboarding without decomposition: You track overall task success rate but do not split debt and tax. You cannot tell whether to refactor or retune.
Over-instrumenting: You log every token, every latency sample, every retry. Your telemetry pipeline costs more than the agent. Start with the five debt metrics and five tax metrics in the tables above.
Technical Verdict
Use this framework when you are running multi-agent systems in production and need to decide whether to refactor orchestration or accept probabilistic overhead. It is especially useful for teams with both engineering and business stakeholders, because it separates fixable architecture (debt) from inherent model noise (tax).
Skip it if you are still in the prototype phase. Early-stage agent systems have so much architectural churn that formal debt measurement is premature. Wait until you have stable orchestration and repeatable workflows.
The framework does not solve the measurement problem for you. You still need to instrument your agent runtime, tag failures correctly, and build the dashboards. But it gives you a vocabulary and a decomposition strategy that makes the instrumentation decisions clearer.