Moving an agent from a demo notebook to a production fleet exposes a gap. MLOps gave us experiment tracking, model versioning, and pipeline orchestration. Agent systems need the same primitives, but the execution model is different. An agent makes tool calls, holds conversational state, and can fail mid-task. ZenML’s new open-source project, Kitaru, addresses this by applying MLOps patterns to agent harnesses: replayable executions, versioned state snapshots, and fleet-level observability.
Hamza Tahir from ZenML walked through the production challenges on the Practical AI podcast. The conversation focused on what “durable” means for agent systems and how harnesses differ from traditional CI/CD or Kubernetes controllers.
What Agent Harnesses Actually Do
An agent harness is the runtime wrapper that manages an agent’s lifecycle. It handles:
- State snapshots at each tool call or reasoning step
- Retry logic when a tool fails or returns malformed data
- Rollback primitives to revert to a known-good state
- Observability hooks for tracing multi-step reasoning and tool call graphs
This is not a CI/CD pipeline. Pipelines assume deterministic steps and linear dependencies. Agents branch, loop, and make decisions based on runtime context. A harness needs to capture that non-linear execution and make it replayable.
Replay vs. Experiment Tracking
In MLOps, experiment tracking logs hyperparameters, metrics, and artifacts. You can reproduce a training run by re-executing the same code with the same data.
For agents, replay means:
- Capturing the exact sequence of tool calls, LLM responses, and state transitions
- Storing the full context window at each step
- Versioning the tool definitions and their implementations
- Recording external API responses so you can replay without hitting live endpoints
Kitaru implements this with state snapshots tied to execution IDs. Each snapshot includes the agent’s memory, the tool call history, and the LLM’s reasoning trace. You can load a snapshot and resume from any point, or replay the entire execution to debug a failure.
Harness Primitives for Failure Isolation
Agent failures are not binary. A tool might return partial data, the LLM might hallucinate a malformed JSON payload, or a rate limit might block a single step in a 20-step task. The harness needs primitives to handle these cases without killing the entire execution.
| Primitive | Purpose | Example Use Case |
|---|---|---|
| Checkpoint | Save state before a risky operation | Before calling an external API with unknown latency |
| Retry with backoff | Re-attempt a failed tool call with exponential delay | Rate-limited API returns 429 |
| Fallback tool | Substitute a different tool if the primary fails | Use a local search if vector DB is down |
| Partial rollback | Revert to the last checkpoint without restarting | LLM returns invalid JSON, retry the reasoning step |
| Circuit breaker | Stop calling a failing tool after N consecutive failures | Third-party service is down, avoid wasting tokens |
These primitives are exposed through the harness API. You define them declaratively in the agent’s configuration, and the harness enforces them at runtime.
Observability Signals for Agent Fleets
A single-model inference endpoint emits latency, throughput, and error rate. An agent fleet needs different signals:
- Tool call graph: Which tools were invoked, in what order, and how many times
- Reasoning trace: The LLM’s chain of thought at each decision point
- Cost per task: Token usage, API calls, and compute time aggregated by task type
- State divergence: How often agents hit unexpected states or infinite loops
- Tool success rate: Per-tool failure rates, broken down by error type
Kitaru integrates with OpenTelemetry to export these signals. You can visualize tool call graphs in Jaeger, track token costs in Prometheus, and set alerts on state divergence thresholds.
Example: Tracing a Multi-Step Agent
from kitaru import AgentHarness, Checkpoint, RetryPolicy
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
harness = AgentHarness(
agent=my_agent,
checkpoints=[
Checkpoint(before="call_external_api"),
Checkpoint(after="parse_response")
],
retry_policy=RetryPolicy(
max_attempts=3,
backoff_factor=2,
fallback_tool="local_search"
)
)
with tracer.start_as_current_span("agent_task") as span:
result = harness.run(task="Summarize Q2 earnings")
span.set_attribute("tool_calls", len(result.tool_history))
span.set_attribute("tokens_used", result.total_tokens)
span.set_attribute("cost_usd", result.cost)This emits a trace with nested spans for each tool call, including retry attempts and fallback invocations. You can correlate failures across the fleet and identify which tools are bottlenecks.
Deployment Shape and State Management
Agent harnesses run as long-lived processes, not ephemeral functions. They need:
- Persistent state storage for snapshots (S3, Postgres, or a versioned object store)
- Message queues for task distribution (SQS, RabbitMQ, or Kafka)
- Horizontal scaling with sticky sessions so the same agent instance handles retries
- Health checks that verify tool connectivity and LLM availability
Kitaru supports multiple deployment shapes:
- Single-tenant: One harness per customer, isolated state, dedicated compute
- Multi-tenant: Shared harness pool, namespaced state, cost allocation by tenant
- Hybrid: Critical agents run single-tenant, batch tasks run multi-tenant
State is versioned using content-addressable storage. Each snapshot gets a hash, and the harness stores a pointer to the current head. Rollbacks update the pointer without deleting history.
Likely Failure Modes
| Failure Mode | Symptom | Mitigation |
|---|---|---|
| State bloat | Snapshots grow unbounded as context window fills | Prune old snapshots, compress memory, or use sliding windows |
| Replay divergence | Replayed execution produces different results | Pin tool versions, mock external APIs, or accept non-determinism |
| Checkpoint overhead | Frequent snapshots slow down execution | Checkpoint only before expensive operations, or use async writes |
| Tool version skew | Agent calls a tool that changed since the snapshot | Version tools in the harness config, reject mismatched versions |
| Infinite loops | Agent gets stuck in a reasoning cycle | Set max iterations per task, or detect repeated states |
The biggest risk is treating agents like stateless functions. If you deploy without harnesses, you lose replay, observability, and failure isolation. You end up debugging production issues by reading logs and guessing at state.
Technical Verdict
Use Kitaru and agent harnesses when:
- You need to debug multi-step agent failures in production
- You want to replay executions without hitting live APIs
- You run agent fleets and need cost and performance metrics per task
- You need rollback and retry primitives that respect agent state
Avoid or defer when:
- Your agents are single-shot (one tool call, no state)
- You are still prototyping and execution replay is not a priority
- Your infrastructure does not support persistent state storage
- You need sub-100ms latency and cannot afford checkpoint overhead
The MLOps-to-agents translation is not just conceptual. Harnesses, state snapshots, and fleet observability are production requirements. Kitaru provides the plumbing. The rest is deciding which primitives matter for your workload and how much replay fidelity you need.