Three steps into a multi-page research task, the agent lost everything. Not a crash. Not a thrown exception. The function returned, context reset, and the pricing data it had just collected vanished.
This is the classic stateless execution problem. Most agent runtimes were never built to hold state across browser sessions that run for twenty minutes. You hit it eventually, usually at the worst moment.
The two standard workarounds are both annoying. Stuffing state into the prompt works until token costs spiral. An external state store solves the problem but now you are maintaining another service, handling serialization, and debugging race conditions when multiple agent steps try to update the same record.
A developer recently published detailed performance metrics from a production stateful agent build using TensorLake sandboxes. The implementation exposes the actual latency penalties, state recovery behavior, and failure modes you encounter when you move agent state inside containerized environments instead of keeping it in-memory or in Redis.
The State Persistence Problem
Most agent frameworks treat execution environments as disposable. You spin up a container, run a function, return a result, and tear it down. This works fine for single-step tasks like “summarize this document” or “generate a SQL query.”
It breaks when agents need to:
- Navigate multi-page workflows where each step depends on data from the previous one
- Keep browser sessions alive across multiple tool calls
- Resume from a checkpoint after a network timeout or rate limit
- Maintain file handles, database connections, or authenticated sessions
The naive solution is to serialize everything to an external store after each step. But now you are paying serialization overhead, network round-trips, and deserialization costs on every agent action. For a research agent that makes 30 tool calls in a single task, that overhead compounds fast.
TensorLake’s Suspend/Resume Model
TensorLake sandboxes introduce named instances with suspend() and resume() primitives that preserve the full VM state, not just files. Running processes, open browser sessions, and in-memory data structures all survive across suspend/resume cycles.
The setup is minimal:
from tensorlake.sandbox import Sandbox
sandbox = Sandbox.create(
name="research-agent",
cpus=2.0,
memory_mb=4096,
secret_names=["OPENAI_API_KEY"],
image="tensorlake/ubuntu-vnc",
)The tensorlake/ubuntu-vnc image provides a full desktop environment with VNC access, which matters when your agent needs to interact with browser UIs or desktop applications that do not expose clean APIs.
State persistence happens at the VM level. When you call sandbox.suspend(), the entire memory snapshot gets written to object storage. When you call sandbox.resume(), that snapshot gets loaded back into a new VM instance. The developer claims sub-second resume times, which is the key metric that determines whether this approach is viable for interactive agent workflows.
Latency Breakdown
The published implementation measured latency across three phases:
- Cold start: Spinning up a new sandbox from scratch
- Warm execution: Running agent steps inside an active sandbox
- Resume overhead: Restoring a suspended sandbox
| Phase | Latency | Notes |
|---|---|---|
| Cold start | 8-12 seconds | Includes image pull, VM boot, dependency install |
| Warm execution | 200-500ms per tool call | Network overhead, no state serialization |
| Resume from suspend | <1 second | Snapshot restore, claimed by vendor |
| Full state serialization (Redis baseline) | 1.2-3.5 seconds | For comparison, external store approach |
The warm execution latency is competitive with in-memory orchestration. The resume overhead is the critical number. If it stays under one second, you can suspend after every agent step without killing interactivity. If it drifts to five seconds, the user experience degrades fast.
The developer did not publish percentile data (p50, p95, p99), which matters when you are trying to set SLOs. Resume latency variance determines whether you can safely suspend after every step or need to batch multiple steps before suspending.
State Recovery and Failure Modes
The interesting failure mode is partial state corruption. If a sandbox crashes mid-step, you have three options:
- Resume from last suspend: You lose all work since the last checkpoint
- Replay from logs: Requires deterministic tool execution, which is rare
- Fail and restart: User starts over
The developer’s implementation suspends after every successful tool call, which minimizes data loss but maximizes suspend/resume overhead. The trade-off depends on your task duration and failure rate.
For a 20-minute research task with 30 tool calls, suspending after every call adds 30 seconds of overhead (30 calls × 1 second resume). That is acceptable. If resume latency drifts to 3 seconds, you are adding 90 seconds, which starts to hurt.
The other failure mode is snapshot corruption. If the suspend operation fails halfway through, you lose the entire session. TensorLake does not document their snapshot durability guarantees, which is a gap. You need to know whether snapshots are replicated, what the failure rate looks like, and whether you get any consistency guarantees.
Observability Hooks
Debugging stateful agents inside sandboxes is harder than debugging local code. The developer mentions VNC access, which helps when you need to see what the browser is doing, but it does not solve the logging problem.
Key observability gaps:
- No structured logs by default: You need to instrument your agent code to ship logs to an external sink
- No built-in tracing: If a tool call hangs, you cannot see which subprocess is blocking
- Snapshot visibility: You cannot inspect snapshot contents without resuming the sandbox
The workaround is to add explicit logging at every state transition:
import logging
logger = logging.getLogger(__name__)
def research_step(sandbox, query):
logger.info(f"Starting research step: {query}")
result = sandbox.run_command(f"python research.py '{query}'")
logger.info(f"Step completed, suspending sandbox")
sandbox.suspend()
return resultThis gives you a breadcrumb trail, but it does not help when the sandbox itself hangs or crashes. You need external health checks and timeout logic to detect stuck sandboxes.
Deployment Shape
The architecture looks like this:
- Orchestrator: Runs your agent loop, decides when to suspend/resume
- Sandbox instances: One per active agent session, named by session ID
- Snapshot storage: Object storage backend (S3, GCS, etc.)
- State metadata: Tracks which sandbox belongs to which session
The orchestrator needs to handle:
- Session routing (which sandbox serves which user request)
- Timeout detection (kill sandboxes that hang)
- Cleanup (delete old snapshots)
The developer does not publish their orchestrator code, but the pattern is straightforward. You maintain a mapping from session ID to sandbox name, and route requests accordingly.
The cost model is different from serverless functions. You pay for:
- Sandbox uptime (billed per second)
- Snapshot storage (billed per GB-month)
- Network egress (if your agent downloads large files)
Suspending aggressively minimizes uptime costs but increases snapshot storage costs. The break-even point depends on your task duration and idle time between steps.
Security Boundaries
Running untrusted code inside sandboxes requires careful boundary enforcement. The developer’s implementation uses the tensorlake/ubuntu-vnc image, which includes a full desktop environment. This is convenient for development but expands the attack surface.
Key security questions:
- Network isolation: Can the sandbox reach internal services?
- Secrets management: How do API keys get injected?
- Resource limits: Can a runaway agent consume unbounded CPU or memory?
TensorLake supports secret injection via environment variables, which is better than hardcoding keys but still exposes them to any code running inside the sandbox. If your agent executes untrusted Python code (e.g., generated by an LLM), that code can exfiltrate secrets.
The safer approach is to run sensitive operations outside the sandbox and expose them via a controlled API. The agent calls the API, the orchestrator validates the request, and the operation runs in a separate, more restricted environment.
When to Use This Pattern
Sandbox-based stateful agents make sense when:
- Your agent needs to interact with desktop UIs or browsers
- Task duration exceeds 5 minutes
- You need to pause and resume across process restarts
- You want isolation between agent sessions
Avoid this pattern when:
- Your agent makes 100+ tool calls per task (suspend overhead compounds)
- You need sub-100ms latency (cold starts and resumes add seconds)
- Your tasks are fully deterministic (replay from logs is cheaper)
- You can keep all state in-memory (simpler, faster, cheaper)
The latency data suggests this approach is viable for research agents, data collection tasks, and long-running workflows where the user expects multi-second response times. It is not viable for interactive chatbots or real-time decision systems where every second counts.
Technical Verdict
Sandbox-based state persistence trades latency for simplicity. You avoid the complexity of external state stores, serialization logic, and race conditions. You pay with cold start overhead and resume latency.
The published numbers (sub-second resume, 200-500ms warm execution) are competitive if they hold at scale. The missing data is percentile latency, snapshot durability, and failure recovery behavior under load.
Use this pattern when your agent needs full VM state (browser sessions, file handles, running processes) and your task duration justifies the overhead. Avoid it when you can serialize state to Redis in under 100ms or when your agent makes dozens of tool calls per second.
The real test is whether resume latency stays under one second at p95 when you are running 100 concurrent agent sessions. That data is not public yet.