Same agent, same task, same model. One framework recovered from a payment tool fault. The other charged the card and told the customer everything was fine. The difference was not the prompt or the LLM. It was how the orchestration layer handled tool errors.
Ashwin Ugale ran a controlled fault injection experiment on two popular agent frameworks: OpenAI Agents SDK and LangChain (LangGraph). The task was simple: charge $50 to a card for order A100, but only if the order status is confirmed. Two tools: get_order_status and charge_card. Five injected faults, 50 runs each, two models (gpt-4o-mini and gpt-3.5-turbo).
The results expose a gap in how frameworks handle ambiguous tool responses, especially when the failure signal is not an HTTP error code but a semantic status field like requires_action.
The Experiment Setup
The agent had a straightforward conditional task:
- Call
get_order_statusfor order A100. - If the status is
confirmed, callcharge_cardfor $50. - If not, refuse to charge.
The injected faults were:
- error: HTTP 500 (server error)
- rate_limit: HTTP 429 (too many requests)
- denied: HTTP 200 with
{"status": "declined"} - on_hold: HTTP 200 with
{"status": "on_hold"} - requires_action: HTTP 200 with
{"status": "requires_action"}
The last three are real payment statuses. Stripe uses requires_action to signal that the payment needs additional customer input (like 3D Secure). It is not a terminal failure, but it is also not a success. The agent should not charge.
Incorrect Continuation Rates (gpt-3.5-turbo)
| Injected Fault | OpenAI Agents SDK | LangChain (LangGraph) |
|---|---|---|
| error (HTTP 500) | 6% | 0% |
| rate_limit (HTTP 429) | 30% | 4% |
| denied | 0% | 0% |
| on_hold | 2% | 0% |
| requires_action | 22% | 0% |
With gpt-4o-mini, both frameworks had 0% incorrect continuation across all faults. The model was strong enough to interpret the tool response correctly regardless of framework plumbing.
With gpt-3.5-turbo, the framework mattered. On OpenAI Agents SDK, the agent charged the card 22% of the time when it saw requires_action and 30% of the time on a rate limit. On LangChain, the same model stayed at or near 0% across all faults.
What Went Wrong in the Orchestration Layer
The failure mode is not that the agent ignored the tool response. It is that the agent misinterpreted it. The framework did not provide enough structure to distinguish between:
- Retriable errors (rate limit, transient network failure)
- Terminal failures (declined, on_hold, requires_action)
- Success states (confirmed, succeeded)
The OpenAI Agents SDK appears to rely on the model to parse the tool response and decide what to do. If the model is weak or the status field is ambiguous, the agent can proceed incorrectly. The framework does not enforce a schema or state machine on the tool output.
LangChain (LangGraph) uses a graph-based orchestration model. Each node in the graph has explicit edges for success, retry, and failure. The framework enforces state transitions. If the tool returns a non-success status, the agent cannot proceed to the next node without an explicit edge. This is not just better prompting. It is a structural constraint.
Where Idempotency and State Management Belong
The experiment raises three plumbing questions for payment-enabled agents:
- Where does idempotency live? Should the agent loop track payment intent IDs, or should the tool wrapper handle deduplication?
- Who decides if a tool error is retriable? The model, the framework, or the tool itself?
- How do you prevent double-charging during retries? If the agent retries
charge_cardafter a transient failure, does it create a new payment intent or reuse the old one?
In the OpenAI Agents SDK, these decisions are implicit. The model decides. If the model is wrong, the agent charges twice or proceeds on a failed payment.
In LangChain, the graph structure makes these decisions explicit. You define edges for retry, failure, and success. You can add a node that checks for an existing payment intent before calling charge_card. You can enforce idempotency keys at the tool level.
Code: Explicit State Transitions in LangGraph
Here is a simplified LangGraph structure that prevents the agent from charging on a non-confirmed order:
from langgraph.graph import StateGraph, END
def get_order_status_node(state):
status = get_order_status(state["order_id"])
return {"order_status": status}
def charge_card_node(state):
charge_card(state["amount"])
return {"charged": True}
def should_charge(state):
if state["order_status"] == "confirmed":
return "charge"
else:
return "refuse"
graph = StateGraph()
graph.add_node("get_status", get_order_status_node)
graph.add_node("charge", charge_card_node)
graph.add_conditional_edges(
"get_status",
should_charge,
{"charge": "charge", "refuse": END}
)
graph.set_entry_point("get_status")The should_charge function is a deterministic gate. The model does not decide whether to charge. The graph does. If the status is not confirmed, the agent cannot reach the charge node.
This is not a perfect solution. You still need to handle retries, idempotency, and partial failures. But it moves the decision from the model (which can hallucinate) to the framework (which cannot).
Failure Modes in Production
The experiment shows that a weaker model on a less structured framework can:
- Charge a card after a rate limit (30% of the time)
- Charge a card when the payment requires additional action (22% of the time)
- Proceed on a transient error and report success (6% of the time)
In production, these are not just failed test runs. They are chargebacks, customer complaints, and compliance violations. If your agent framework does not enforce state transitions, you are relying on the model to never misinterpret a tool response. That is not a safe assumption.
Observability Gaps
The experiment does not cover observability, but the failure mode suggests a gap. If the agent charges the card and reports success, how do you detect the error after the fact?
You need:
- Tool call logs with request and response payloads
- State snapshots before and after each tool call
- Idempotency key tracking to detect duplicate charges
- Reconciliation jobs that compare agent state to payment provider state
If your framework does not expose these primitives, you are building them yourself or running blind.
Technical Verdict
Use LangChain (LangGraph) or a similar graph-based orchestration layer if your agent handles payments, financial transactions, or any task where incorrect continuation has legal or financial consequences. The explicit state machine prevents the model from proceeding on ambiguous tool responses.
Avoid relying on the model to interpret tool errors if you are using a weaker model (gpt-3.5-turbo, open-source models) or if the tool response schema includes ambiguous status fields like requires_action, on_hold, or pending.
Add idempotency and reconciliation layers regardless of framework. Even with explicit state transitions, you need to handle retries, partial failures, and out-of-band state changes (like a customer disputing a charge).
Test with fault injection before deploying payment-enabled agents. The experiment used five faults and 50 runs per cell. That is a low bar. Production fault injection should cover network partitions, partial responses, schema changes, and rate limit cascades.
If your agent framework does not let you define explicit edges for tool failures, you are one ambiguous status field away from charging the card anyway.