Running 50 concurrent Playwright sessions inside an agent workflow sounds like a throughput win. In practice, it triggers CAPTCHA walls, eats CPU, and creates a retry death spiral that drags your entire pipeline down.
The fix is not better stealth headers or residential proxies. It is concurrency caps, overflow queuing, and bounded retries. A setup with 5 active sessions and 5 queued tasks often beats 50 tabs running wild.
The Self-Inflicted Load Problem
Browser automation agents fail in a predictable pattern:
- One worker hits a CAPTCHA or rate limit
- It retries immediately
- Ten more workers hit the same domain
- They all retry at once
- Old sessions hang, consuming memory and CPU
- The agent pipeline backs up behind stuck browser contexts
- Downstream LLM steps process junk or duplicates
This looks like a target-site problem in logs. Most of the time, it is your own concurrency model.
Why High Browser Concurrency Backfires
Every Playwright browser or context adds pressure:
- CPU contention (rendering, JS execution, event loop)
- Memory pressure (each context holds DOM state, network cache, cookies)
- Bursty traffic to the same domain (looks bot-like)
- Retry amplification (failures cascade across workers)
Two bad things happen at once: you look more suspicious to the target, and your own browser stack gets less stable.
Concurrency Cap and Queue Architecture
The working pattern is simple:
- Cap active browser sessions (5 to 10 is a good starting point)
- Queue overflow tasks in memory or Redis
- Pace requests per domain (1 to 3 seconds between hits)
- Bound retries (3 max, with exponential backoff)
- Kill stuck sessions fast (30-second timeout)
In-Memory Queue (Node.js Example)
class BrowserPool {
constructor(maxActive = 5, maxQueued = 10) {
this.maxActive = maxActive;
this.maxQueued = maxQueued;
this.active = new Set();
this.queue = [];
this.domainLastHit = new Map();
}
async acquire(domain, task) {
if (this.queue.length >= this.maxQueued) {
throw new Error('Queue full');
}
// Wait for slot
while (this.active.size >= this.maxActive) {
await new Promise(resolve => setTimeout(resolve, 100));
}
// Domain pacing
const lastHit = this.domainLastHit.get(domain) || 0;
const elapsed = Date.now() - lastHit;
if (elapsed < 2000) {
await new Promise(resolve => setTimeout(resolve, 2000 - elapsed));
}
const session = { id: crypto.randomUUID(), domain, task };
this.active.add(session);
this.domainLastHit.set(domain, Date.now());
return session;
}
release(session) {
this.active.delete(session);
}
async execute(domain, task, retries = 3) {
let attempt = 0;
while (attempt < retries) {
const session = await this.acquire(domain, task);
try {
const result = await Promise.race([
task(),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('Timeout')), 30000)
)
]);
this.release(session);
return result;
} catch (err) {
this.release(session);
attempt++;
if (attempt >= retries) throw err;
await new Promise(resolve =>
setTimeout(resolve, Math.pow(2, attempt) * 1000)
);
}
}
}
}Redis-Backed Queue (Python with BullMQ-style pattern)
For distributed agents or multi-container deployments, Redis provides durable queuing:
import redis
import time
import uuid
class RedisBrowserQueue:
def __init__(self, redis_url, max_active=5):
self.redis = redis.from_url(redis_url)
self.max_active = max_active
self.active_key = "browser:active"
self.queue_key = "browser:queue"
self.domain_key_prefix = "browser:domain:"
def enqueue(self, domain, task_data):
queue_size = self.redis.llen(self.queue_key)
if queue_size >= 50:
raise Exception("Queue full")
task = {
"id": str(uuid.uuid4()),
"domain": domain,
"data": task_data,
"enqueued_at": time.time()
}
self.redis.rpush(self.queue_key, json.dumps(task))
return task["id"]
def acquire(self):
while self.redis.scard(self.active_key) >= self.max_active:
time.sleep(0.1)
task_json = self.redis.lpop(self.queue_key)
if not task_json:
return None
task = json.loads(task_json)
self.redis.sadd(self.active_key, task["id"])
# Domain pacing
domain_key = f"{self.domain_key_prefix}{task['domain']}"
last_hit = self.redis.get(domain_key)
if last_hit:
elapsed = time.time() - float(last_hit)
if elapsed < 2.0:
time.sleep(2.0 - elapsed)
self.redis.setex(domain_key, 10, time.time())
return task
def release(self, task_id):
self.redis.srem(self.active_key, task_id)Trade-offs: In-Memory vs. Redis Queue
| Dimension | In-Memory Queue | Redis Queue |
|---|---|---|
| Latency | 1-5 ms | 10-20 ms (network RTT) |
| Durability | Lost on crash | Survives restarts |
| Multi-worker | Single process only | Distributed agents |
| Observability | Logs only | Redis CLI, monitoring tools |
| Complexity | 50 lines of code | Redis dependency, serialization |
| Failure mode | Process dies, queue lost | Redis down, all agents stall |
Session Affinity and Partial Workflow Resumption
If an agent needs to resume a partially completed browser workflow (e.g., multi-step form, OAuth flow), you need session affinity:
- Store session state (cookies, localStorage, auth tokens) in Redis or a database
- Tag tasks with a
session_id - Route tasks with the same
session_idto the same worker - Serialize Playwright context state using
context.storageState()
// Save state
const state = await context.storageState();
await redis.set(`session:${sessionId}`, JSON.stringify(state));
// Restore state
const state = JSON.parse(await redis.get(`session:${sessionId}`));
const context = await browser.newContext({ storageState: state });This adds complexity but prevents re-authentication and session loss on retry.
Observability Hooks
You need visibility into queue depth, active sessions, and retry rates:
- Emit metrics on
enqueue,acquire,release,timeout,retry - Track per-domain hit rates and CAPTCHA rates
- Log session lifecycle (created, active, released, killed)
- Expose a
/metricsendpoint for Prometheus or Datadog
Example metrics:
browser_pool_active_sessions 5
browser_pool_queued_tasks 3
browser_pool_retries_total{domain="example.com"} 12
browser_pool_captchas_total{domain="example.com"} 2
browser_pool_timeouts_total 1Likely Failure Modes
Queue fills up: Set a max queue depth and reject new tasks with a 429 or backpressure signal. Do not let the queue grow unbounded.
Redis goes down: In-memory fallback or circuit breaker. If Redis is unavailable, either queue in-process or reject tasks.
Stuck sessions leak: Timeout enforcement is critical. Use Promise.race() with a hard timeout (30 seconds) and kill the browser context on expiry.
Domain-specific rate limits: Track per-domain pacing separately. A global cap is not enough if one domain allows 10 req/s and another allows 1 req/min.
CAPTCHA solver latency: If you integrate a CAPTCHA solver (2Captcha, Anti-Captcha), add a separate timeout and retry budget. Solver failures should not block the queue.
Technical Verdict
Use this approach when:
- Your agent opens more than 5 concurrent browser sessions
- You hit CAPTCHAs or rate limits regularly
- You run browser automation inside n8n, Make, Zapier, or custom orchestrators
- You need distributed queuing across multiple agent workers
Avoid or defer when:
- You run fewer than 3 concurrent sessions (overhead is not worth it)
- Your target sites have no rate limiting or bot detection
- You need sub-second latency (queuing adds 10-100 ms per task)
- You are prototyping and do not care about stability yet
The boring fix (cap, queue, pace, timeout) beats clever stealth every time. Start with 5 active sessions, 5 queued, and 2-second domain pacing. Tune from there based on CAPTCHA rates and CPU usage.