Screenpipe records your screen and audio continuously, stores everything locally, and exposes it as searchable memory for AI agents. No cloud sync. No external APIs for the recording loop. The interesting part is not the capture itself but the pipeline that turns raw pixels and audio into queryable context without overwhelming local storage or indexing throughput.
This is the input side of agent memory. Most agent frameworks assume context arrives pre-structured (support tickets, API responses, database rows). Screenpipe starts with unstructured desktop activity and builds retrieval primitives on top. The tradeoffs show up in OCR latency, embedding generation cost, and permission boundaries when multiple agents need access.
The Capture and Storage Pipeline
Screenpipe runs a continuous loop:
- Screen capture: periodic screenshots (configurable interval, typically 1-5 seconds)
- Audio capture: microphone and system audio streams
- Local processing: OCR on screenshots, speech-to-text on audio
- Indexing: full-text search index plus optional embeddings
- Storage: SQLite for metadata, filesystem for raw frames and audio chunks
The bottleneck is OCR and transcription. Running Tesseract or Whisper on every frame locally means CPU spikes and battery drain on laptops. Screenpipe mitigates this with:
- Frame diffing to skip duplicate screenshots
- Batched OCR processing during idle periods
- Configurable quality/speed tradeoffs (fast OCR vs. accurate OCR)
Storage grows predictably. At 1 screenshot per 2 seconds with 200KB per compressed frame, you accumulate roughly 8GB per 24-hour day before OCR text extraction. Audio at 16kHz mono adds another 1.4GB per day. The system prunes old recordings based on retention policies, but the default is local-only forever unless you delete manually.
Retrieval Primitives for Agents
Agents query Screenpipe through a local API (HTTP server on localhost). The retrieval layer exposes:
- Full-text search: SQLite FTS5 over OCR’d text and transcribed audio
- Time-range queries: “show me everything from 2pm to 4pm yesterday”
- Semantic search: optional CLIP embeddings for image similarity, sentence transformers for text
- Structured extraction: regex or LLM-based extraction of entities (URLs, email addresses, code snippets)
The API returns JSON with timestamps, content snippets, and pointers to raw frames. Agents can request the original screenshot or audio chunk if they need full fidelity.
Example query flow:
import requests
# Agent queries for recent Slack messages
response = requests.post("http://localhost:3030/search", json={
"query": "deployment failed",
"start_time": "2026-10-25T14:00:00Z",
"end_time": "2026-10-25T16:00:00Z",
"content_type": ["ocr", "audio"]
})
results = response.json()
for hit in results["hits"]:
print(f"{hit['timestamp']}: {hit['text']}")
# Agent can fetch raw frame: GET /frames/{hit['frame_id']}The agent does not need to know which app was active. It searches across all captured activity. This is powerful for workflow extraction (watching a user complete a task, then automating it) but risky for accidental exposure of sensitive data.
Local-Only Processing Tradeoffs
Running OCR and speech-to-text locally avoids cloud API costs and privacy leaks, but introduces accuracy and latency constraints.
| Component | Local Option | Cloud Option | Tradeoff |
|---|---|---|---|
| OCR | Tesseract, EasyOCR | Google Vision, AWS Textract | Local: 2-5s per frame, 85-90% accuracy. Cloud: <500ms, 95%+ accuracy, costs $1.50 per 1000 images |
| Speech-to-text | Whisper (base/small) | Deepgram, AssemblyAI | Local: real-time factor 0.5-1x on CPU, 0.1x on GPU. Cloud: <0.1x, costs $0.0048 per minute |
| Embeddings | sentence-transformers, CLIP | OpenAI, Cohere | Local: 50-200ms per text chunk on CPU. Cloud: <50ms, costs $0.0001 per 1K tokens |
Screenpipe defaults to local processing. You can plug in cloud APIs for OCR and transcription if you accept the privacy and cost tradeoffs. The embedding layer is optional because full-text search covers most agent queries. Semantic search helps when agents need conceptual similarity (“find all discussions about database migrations” when the word “migration” never appears).
Security Boundaries and Permission Scoping
Screenpipe records everything. Agents should not have blanket access to all recordings. The permission model is:
- Time-based scoping: agents request access to specific time ranges
- Content-type filtering: agents can be restricted to OCR-only (no audio) or audio-only (no screenshots)
- Redaction rules: regex-based filters to strip credit card numbers, API keys, passwords from OCR text before indexing
The current implementation relies on agent cooperation. There is no OS-level sandbox preventing an agent from querying the full database. For multi-agent workflows, you need an orchestration layer that enforces scoping:
# Example agent permission config
agents:
- name: "slack-summarizer"
allowed_time_range: "last_24_hours"
allowed_content: ["ocr"]
redaction: ["credit_card", "ssn"]
- name: "meeting-transcriber"
allowed_time_range: "last_7_days"
allowed_content: ["audio"]
redaction: []This config lives outside the agent code. The orchestrator (a separate process or sidecar) intercepts API calls and enforces policies. Without this, agents can exfiltrate recordings by querying the API directly.
State Synchronization Across Machines
Screenpipe is local-only by design. If you run agents on multiple machines (laptop, desktop, cloud VM), each has its own recording database. There is no built-in sync.
Options for multi-machine workflows:
- Centralized query proxy: agents on remote machines query a proxy that forwards requests to the local Screenpipe instance over SSH or VPN
- Periodic export: cron job exports recent recordings to S3, agents query the export
- Federated search: each machine runs Screenpipe, agents query all instances and merge results
The first option is simplest but requires the local machine to stay online. The second option introduces staleness (agents see recordings from 1 hour ago, not real-time). The third option is complex but supports offline work.
For most use cases, agents run on the same machine as Screenpipe. The sync problem only appears when you want a cloud-hosted agent to access your local recordings without uploading them.
Observability and Failure Modes
Screenpipe exposes metrics on:
- Capture rate: frames per second, audio buffer overruns
- Processing lag: time from capture to indexed (OCR/transcription backlog)
- Storage usage: disk space consumed, retention policy enforcement
- Query latency: p50/p95/p99 for search API
Common failure modes:
- OCR backlog: if OCR is slower than capture rate, the queue grows unbounded. Mitigation: drop frames or reduce capture frequency.
- Disk full: recordings fill the disk, capture stops. Mitigation: aggressive retention policies or external storage mount.
- Privacy leak: agent queries reveal sensitive data (passwords, PII). Mitigation: redaction rules and permission scoping.
- Indexing corruption: SQLite FTS5 index becomes inconsistent after crash. Mitigation: periodic index rebuilds from raw recordings.
The system does not auto-recover from disk-full or index corruption. You need external monitoring (disk space alerts, index health checks) and manual intervention.
Deployment Shape
Screenpipe runs as a local daemon (systemd service on Linux, launchd on macOS, Windows service). Agents connect to the HTTP API on localhost. The typical stack:
- Screenpipe daemon: Rust binary, 50-100MB memory footprint, 10-20% CPU during active capture
- SQLite database: 1-5GB for metadata and FTS5 index
- Raw recordings: 10-50GB per week depending on retention policy
- Agent runtime: separate process (Python, Node.js, Go) that queries the API
For production use, you add:
- Log aggregation: ship Screenpipe logs to Loki or Elasticsearch for debugging
- Metrics export: Prometheus exporter for capture rate, query latency, storage usage
- Backup: periodic snapshots of SQLite database and raw recordings to S3 or NAS
The daemon does not require root privileges unless you want system-wide audio capture (all apps, not just microphone). Screen capture works without elevated permissions on most OSes.
Technical Verdict
Use Screenpipe when:
- You need agents to access unstructured desktop activity (not just API data or database rows)
- Privacy constraints prevent cloud-based recording or indexing
- You can tolerate local processing latency (2-5 seconds for OCR, real-time for audio)
- You have disk space for 10-50GB of recordings per week
Avoid it when:
- Agents need real-time context (OCR lag makes it unsuitable for live assistance)
- You need cloud-hosted agents to access recordings (no built-in sync, requires custom proxy)
- You cannot enforce permission boundaries (agents can query the full database without OS-level sandboxing)
- You need high-accuracy OCR or transcription (local models lag cloud APIs by 5-10% accuracy)
The sweet spot is workflow automation where agents learn from past activity (SOP generation, task extraction) rather than real-time assistance. The local-first architecture makes it viable for privacy-sensitive environments (healthcare, legal, finance) where cloud recording is not an option.