mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

Dev Tools

Your Agent Retried the Tool Call and Charged the Card Twice: A Zero-Cost Resilience Layer in Python

Build idempotency keys, circuit breakers, and dead-letter queues in 310 lines of standard-library Python to prevent duplicate charges during agent retries.

Source: dev.to
Your Agent Retried the Tool Call and Charged the Card Twice: A Zero-Cost Resilience Layer in Python

The agent timed out, retried the tool call, and charged the customer twice. Both attempts succeeded. The logs show two clean executions separated by three seconds. No error flags, no warnings. Just a duplicate side effect that nobody caught until the customer complained.

This is the failure mode that “just add retries” creates. The request reached the payment API, the charge went through, and the response got lost on the way back. Your agent saw a timeout and did exactly what you told it to do: try again. The second charge also succeeded. Now you have a refund to issue and a trust problem to solve.

Retries are not optional in production agent systems. Network calls fail, APIs hiccup, and timeouts happen. But naive retry logic turns transient failures into permanent damage when tool calls have side effects. You need idempotency keys, circuit breakers, jittered backoff, and a place to park work that cannot be retried safely.

This article walks through a 310-line resilience layer built with Python’s standard library. No Redis, no database, no external dependencies. It shows how to prevent duplicate operations, avoid retry storms, and handle failures that cannot be fixed by trying again.

The Two Bugs That Naive Retries Create

Lost response: The tool call succeeded on the server side, but the response never made it back to the agent. Your code sees a timeout. The truth is a success plus a mystery. Retrying now executes the side effect twice: two charges, two emails, two Slack messages.

Retry storm: A dependency starts failing for everyone at once. Every worker waits exactly two seconds before retrying. They all come back at the same moment, hammer the recovering service, and turn a short wobble into a sustained outage. Fixed backoff is an accidental synchronization mechanism.

Both are fixable with patterns older than LLMs, but only if you design the retry layer and the tool call as one unit.

Architecture: Four Layers, One Flow

The resilience layer wraps every tool call with four mechanisms:

LayerPurposeFailure Mode It Prevents
Idempotency keyDeduplicates retries of the same logical operationDuplicate charges, emails, API mutations
Jittered backoffSpreads retry attempts over timeRetry storms, thundering herd
Circuit breakerStops calling a failing dependencyCascading failures, wasted retries
Dead-letter queueParks work that cannot be retried safelySilent data loss, infinite retry loops

The flow looks like this:

  1. Generate an idempotency key for the operation (or reuse one if this is a retry).
  2. Check the circuit breaker. If it is open, fail fast and send the work to the dead-letter queue.
  3. Execute the tool call with the idempotency key attached.
  4. If it succeeds, return the result and close the circuit.
  5. If it fails, apply jittered backoff and retry (up to a limit).
  6. If all retries fail, send the work to the dead-letter queue.

The idempotency key is the critical piece. It travels with the operation through every retry attempt, so the downstream service can recognize duplicates and return the cached result instead of executing the side effect again.

Idempotency Keys: Tracking Operations Across Retries

An idempotency key is a unique identifier for a logical operation. When you retry a tool call, you send the same key. The service checks if it has already processed that key. If yes, it returns the cached result. If no, it executes the operation and caches the result under that key.

Without a database, you store idempotency keys in memory. This works for single-process agents and testing. For multi-process or distributed systems, you need shared storage (Redis, DynamoDB, or a database).

import hashlib
import json
from typing import Any, Dict

class IdempotencyTracker:
    def __init__(self):
        self.seen: Dict[str, Any] = {}
    
    def generate_key(self, tool_name: str, args: dict) -> str:
        """Generate a deterministic key from tool name and arguments."""
        payload = json.dumps({"tool": tool_name, "args": args}, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()[:16]
    
    def check(self, key: str) -> tuple[bool, Any]:
        """Returns (already_seen, cached_result)."""
        if key in self.seen:
            return True, self.seen[key]
        return False, None
    
    def record(self, key: str, result: Any):
        """Cache the result for this operation."""
        self.seen[key] = result

The key is generated from the tool name and arguments. If you retry with the same inputs, you get the same key. The tracker checks if it has seen this key before. If yes, it returns the cached result without calling the tool again.

This prevents the duplicate charge scenario. The first attempt generates key abc123, calls the payment API, and caches the result. The second attempt generates the same key, sees it in the cache, and returns the cached result without calling the API.

Jittered Backoff: Breaking Synchronization

Fixed backoff (wait 2 seconds, then 4, then 8) creates retry storms. Every worker that sees a failure at the same time will retry at the same time. They synchronize their retries and overwhelm the recovering service.

Jittered backoff adds randomness to each wait interval. Workers spread their retries over time instead of clustering them.

import random
import time

def jittered_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 60.0) -> float:
    """Exponential backoff with full jitter."""
    delay = min(base_delay * (2 ** attempt), max_delay)
    return random.uniform(0, delay)

# Example: first retry waits 0-2 seconds, second waits 0-4, third waits 0-8
for attempt in range(5):
    wait = jittered_backoff(attempt)
    print(f"Attempt {attempt}: wait {wait:.2f}s")
    time.sleep(wait)

Full jitter (randomizing the entire interval) is better than partial jitter (randomizing a fraction). It spreads load more evenly and prevents clusters from forming.

Circuit Breaker: Failing Fast When a Dependency Is Down

A circuit breaker stops calling a failing dependency after a threshold of consecutive failures. Instead of wasting time on retries that will fail, it fails fast and gives the dependency time to recover.

The circuit has three states:

  • Closed: Normal operation. Requests go through.
  • Open: Too many failures. All requests fail immediately without calling the dependency.
  • Half-open: Testing if the dependency has recovered. One request goes through. If it succeeds, close the circuit. If it fails, open it again.
from enum import Enum
from dataclasses import dataclass
import time

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    timeout: float = 60.0  # seconds to wait before trying half-open
    
    def __post_init__(self):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.last_failure_time = None
    
    def call(self, func, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.timeout:
                self.state = CircuitState.HALF_OPEN
            else:
                raise Exception("Circuit breaker is open")
        
        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise
    
    def on_success(self):
        self.failure_count = 0
        self.state = CircuitState.CLOSED
    
    def on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN

When the circuit is open, the agent does not waste time retrying. It fails fast and sends the work to the dead-letter queue. This prevents cascading failures and gives the dependency breathing room.

Dead-Letter Queue: Parking Work That Cannot Be Retried

Some failures cannot be fixed by retrying. The API key is invalid, the input is malformed, or the circuit breaker is open. Retrying wastes resources and delays the inevitable.

A dead-letter queue (DLQ) is a holding area for work that failed after all retries. It prevents silent data loss and gives you a place to inspect failures, fix the root cause, and replay the work later.

from dataclasses import dataclass
from typing import Any, List
import json

@dataclass
class DeadLetter:
    tool_name: str
    args: dict
    error: str
    timestamp: float
    attempts: int

class DeadLetterQueue:
    def __init__(self):
        self.queue: List[DeadLetter] = []
    
    def add(self, tool_name: str, args: dict, error: str, attempts: int):
        letter = DeadLetter(
            tool_name=tool_name,
            args=args,
            error=str(error),
            timestamp=time.time(),
            attempts=attempts
        )
        self.queue.append(letter)
    
    def drain(self) -> List[DeadLetter]:
        """Return all dead letters and clear the queue."""
        letters = self.queue.copy()
        self.queue.clear()
        return letters
    
    def save_to_disk(self, path: str):
        """Persist dead letters for later inspection."""
        with open(path, "w") as f:
            for letter in self.queue:
                f.write(json.dumps(letter.__dict__) + "\n")

The DLQ stores the tool name, arguments, error message, timestamp, and number of attempts. You can drain it periodically, inspect the failures, and decide whether to replay them (after fixing the root cause) or discard them.

Putting It Together: The Resilient Tool Call Wrapper

The complete resilience layer wraps every tool call with idempotency tracking, jittered backoff, circuit breaking, and dead-letter handling.

class ResilientToolExecutor:
    def __init__(self, max_retries: int = 3):
        self.idempotency = IdempotencyTracker()
        self.circuit_breaker = CircuitBreaker()
        self.dlq = DeadLetterQueue()
        self.max_retries = max_retries
    
    def execute(self, tool_name: str, tool_func, args: dict):
        # Generate idempotency key
        key = self.idempotency.generate_key(tool_name, args)
        
        # Check if we've already done this operation
        already_done, cached_result = self.idempotency.check(key)
        if already_done:
            return cached_result
        
        # Try with retries
        for attempt in range(self.max_retries):
            try:
                result = self.circuit_breaker.call(tool_func, **args)
                self.idempotency.record(key, result)
                return result
            except Exception as e:
                if attempt < self.max_retries - 1:
                    wait = jittered_backoff(attempt)
                    time.sleep(wait)
                else:
                    # All retries failed, send to DLQ
                    self.dlq.add(tool_name, args, str(e), attempt + 1)
                    raise

This wrapper handles the entire retry flow. It checks for duplicate operations, applies backoff between retries, respects the circuit breaker, and parks failed work in the DLQ.

Testing with a Virtual Clock

Testing retry logic with real time is slow and flaky. A virtual clock lets you control time in tests, making them fast and deterministic.

class VirtualClock:
    def __init__(self):
        self.current_time = 0.0
    
    def time(self) -> float:
        return self.current_time
    
    def sleep(self, duration: float):
        self.current_time += duration
    
    def advance(self, duration: float):
        self.current_time += duration

# Inject the virtual clock into your retry logic
def jittered_backoff_testable(attempt: int, clock, base_delay: float = 1.0):
    delay = min(base_delay * (2 ** attempt), 60.0)
    jittered = delay * 0.5  # deterministic jitter for testing
    clock.sleep(jittered)
    return jittered

With a virtual clock, you can test that a retry sequence takes 15 seconds without waiting 15 real seconds. You can verify that the circuit breaker opens after five failures without running five slow API calls.

Observability: What to Log

Retry logic is invisible until it breaks. Log enough to reconstruct what happened:

  • Idempotency key: So you can trace retries of the same operation.
  • Attempt number: So you know how many times it retried.
  • Wait duration: So you can see if backoff is working.
  • Circuit state: So you know when the breaker opened or closed.
  • DLQ additions: So you know what work got parked.
import logging

logger = logging.getLogger(__name__)

def execute_with_logging(self, tool_name: str, tool_func, args: dict):
    key = self.idempotency.generate_key(tool_name, args)
    logger.info(f"Executing {tool_name} with key {key}")
    
    already_done, cached_result = self.idempotency.check(key)
    if already_done:
        logger.info(f"Idempotency hit for {key}, returning cached result")
        return cached_result
    
    for attempt in range(self.max_retries):
        try:
            logger.info(f"Attempt {attempt + 1} for {tool_name}, circuit: {self.circuit_breaker.state.value}")
            result = self.circuit_breaker.call(tool_func, **args)
            self.idempotency.record(key, result)
            logger.info(f"Success for {tool_name} on attempt {attempt + 1}")
            return result
        except Exception as e:
            logger.warning(f"Attempt {attempt + 1} failed for {tool_name}: {e}")
            if attempt < self.max_retries - 1:
                wait = jittered_backoff(attempt)
                logger.info(f"Waiting {wait:.2f}s before retry")
                time.sleep(wait)
            else:
                logger.error(f"All retries exhausted for {tool_name}, sending to DLQ")
                self.dlq.add(tool_name, args, str(e), attempt + 1)
                raise

Structured logs (JSON) are better than plain text. They let you query by idempotency key, tool name, or circuit state.

Technical Verdict

This 310-line resilience layer is production-ready for single-process agents and local testing. It prevents duplicate charges, retry storms, and cascading failures without adding external dependencies. The performance overhead is minimal: one dictionary lookup per tool call (idempotency check), one state check (circuit breaker), and one random number generation (jitter). In benchmarks with a virtual clock, the wrapper adds less than 1ms of latency per call when the circuit is closed and idempotency cache is warm.

Use in-memory idempotency when: You run a single agent process, your tool calls complete in seconds (not minutes), and process restarts are rare. This covers most development workflows, integration tests, and small-scale automations.

Switch to Redis when: You run multiple agent processes on the same machine

Tags

agentic-ai orchestration infrastructure

Primary Source

dev.to ↗