OpenAI just published the first part of their Habitat storage scaling series, revealing how they evolved conversation storage from a Python library into a distributed system handling 22 million requests per second for over 1 billion ChatGPT users. This is rare public infrastructure disclosure from a company that typically keeps plumbing private.
The core problem: stateless LLM inference needs stateful conversation history. Every agent turn requires fetching prior context, appending new messages, and persisting the updated thread. At billion-user scale, this storage layer becomes the bottleneck between fast inference and durable memory.
The Python Library Era
Habitat started as an internal Python library wrapping basic storage primitives. Early ChatGPT conversations were simple key-value pairs: conversation ID mapped to a list of messages. The library handled serialization, caching, and basic retry logic.
This worked until it didn’t. As usage grew, three failure modes emerged:
- Hot partition problems: Popular conversations (shared links, viral threads) created uneven load distribution
- Cross-region latency: Users in Asia fetching conversations stored in US data centers saw 200ms+ round trips
- Schema migration pain: Adding new fields (function calls, tool outputs, vision data) required coordinating deploys across all services using the library
The library model also made observability impossible. When a conversation failed to load, engineers had no visibility into which storage backend failed, whether the issue was replication lag, or if the problem was client-side caching.
Architecture Evolution: From Library to Platform
OpenAI rebuilt Habitat as a standalone service with explicit contracts. The new architecture separates concerns:
Storage tier: Globally distributed key-value stores with multi-region replication. Conversations are sharded by user ID, not conversation ID, to keep all threads for a single user co-located. This reduces cross-shard queries when building context windows.
Caching tier: Multi-level caching with different TTLs based on conversation recency. Active conversations (last message within 1 hour) stay in hot cache. Older threads move to warm cache backed by SSDs. Archived conversations (30+ days inactive) live in cold storage with higher retrieval latency.
API layer: Versioned gRPC endpoints that handle schema evolution transparently. Clients request conversations using a version number. The API layer translates between storage schema and client-expected format.
Replication topology: Conversations replicate across three availability zones within a region, then asynchronously to other regions. Writes go to the nearest region. Reads prefer local replicas but fall back to remote if local data is stale beyond a threshold (currently 500ms).
Data Locality and Agent Memory
Agent systems complicate storage because context windows span multiple turns. A 10-turn conversation might reference files uploaded in turn 2, function calls from turn 5, and vision analysis from turn 8. Habitat handles this with structured message objects:
{
"conversation_id": "conv_abc123",
"messages": [
{
"turn": 1,
"role": "user",
"content": "Analyze this spreadsheet",
"attachments": [{"type": "file", "id": "file_xyz"}]
},
{
"turn": 2,
"role": "assistant",
"content": "I'll use the data analysis tool",
"tool_calls": [{"id": "call_123", "function": "analyze_csv"}]
},
{
"turn": 3,
"role": "tool",
"tool_call_id": "call_123",
"content": "{\"summary\": {...}}"
}
],
"metadata": {
"model": "gpt-4",
"created_at": 1725984000,
"last_active": 1725987600
}
}File references and tool outputs are stored separately in blob storage. The conversation object holds pointers. When building a context window, Habitat fetches the conversation skeleton first, then hydrates attachments and tool outputs in parallel.
This separation matters for cost and performance. Blob storage is cheaper but slower. Conversation metadata needs low-latency access. Splitting them lets Habitat optimize each independently.
Failure Modes and Observability
At 22M requests per second, every failure mode manifests constantly. OpenAI’s post highlights three categories:
Replication lag: Asynchronous cross-region replication means users might see stale conversation state if they switch regions mid-session. Habitat tracks replication lag per region and redirects reads to the primary region if lag exceeds 500ms.
Partial writes: A conversation update might succeed in the primary region but fail to replicate to secondaries. Habitat uses versioned writes with monotonic timestamps. Reads return the highest version available and trigger background reconciliation if versions diverge.
Cache inconsistency: Multi-level caching creates windows where hot cache shows old data while warm cache has the latest write. Habitat invalidates cache entries on write but uses probabilistic invalidation (bloom filters) to avoid thundering herds.
Observability primitives include:
- Per-conversation trace IDs that follow requests across storage tiers
- Replication lag histograms broken down by region pair
- Cache hit/miss rates segmented by conversation age
- Write amplification metrics (how many storage operations per API call)
Schema Evolution at Scale
Agent capabilities evolve faster than storage schemas. ChatGPT added function calling, then vision, then multi-modal outputs. Each required schema changes without breaking existing conversations.
Habitat uses a versioned schema approach:
| Schema Version | Introduced | Key Changes | Migration Strategy |
|---|---|---|---|
| v1 | 2022-11 | Basic text messages | N/A (initial) |
| v2 | 2023-03 | Function call support | Lazy migration on read |
| v3 | 2023-09 | Vision and image inputs | Dual-write for 30 days |
| v4 | 2024-05 | Multi-modal outputs | Background backfill |
| v5 | 2026-02 | Structured tool outputs | Opt-in per conversation |
Lazy migration means old conversations stay in v1 format until accessed. On read, Habitat upgrades the schema in-memory and writes back the upgraded version asynchronously. This avoids a big-bang migration of billions of conversations.
Dual-write handles breaking changes. For 30 days, writes go to both old and new schema. Reads prefer new schema but fall back to old if new is unavailable. After 30 days, old schema becomes read-only.
Request Flow and Latency Budget
A typical conversation retrieval follows this path:
- Client sends gRPC request to nearest API endpoint (target: <10ms)
- API layer checks hot cache (target: <1ms)
- On cache miss, query warm cache (target: <5ms)
- On warm cache miss, fetch from primary storage (target: <20ms)
- Hydrate attachments and tool outputs in parallel (target: <30ms)
- Return assembled conversation (total target: <50ms p95)
The 22M req/s figure includes both reads and writes. Reads dominate (80% of traffic) because every agent turn fetches conversation history. Writes happen only when users send messages or agents append tool outputs.
Latency budget allocation:
- 40% for network hops (client to API, API to storage)
- 30% for storage query execution
- 20% for attachment hydration
- 10% for serialization and protocol overhead
When Storage Becomes the Agent Bottleneck
Habitat’s architecture reveals a fundamental tension in agentic systems: inference is getting faster, but durable state management is not. GPT-4 can generate a response in 2 seconds. Fetching the conversation history to build the prompt takes 50ms. As context windows grow (128k, 1M tokens), the storage layer becomes the long pole.
Three specific bottlenecks emerge:
Context window assembly: Large conversations require fetching hundreds of messages plus attachments. Habitat parallelizes attachment retrieval but still hits storage IOPS limits.
Write amplification: Every agent turn writes to primary storage, replicates to secondaries, invalidates caches, and updates indexes. A single user message triggers 10+ storage operations.
Consistency vs. availability trade-offs: Strong consistency across regions would add 100ms+ to every write. Eventual consistency risks showing users stale conversation state. Habitat chooses availability and uses client-side reconciliation when conflicts occur.
Technical Verdict
Use Habitat’s patterns when:
- You’re building multi-turn agent systems where conversation history is critical state
- Your user base spans multiple geographic regions and you need local read latency
- Schema evolution is frequent (new agent capabilities, tool integrations, modality support)
- You can tolerate eventual consistency and need to prioritize availability
Avoid this architecture when:
- Your agent interactions are stateless or ephemeral (no conversation history needed)
- Strong consistency is required (financial transactions, collaborative editing)
- Your scale is <1M requests/day (the operational complexity isn’t worth it)
- You need sub-10ms p99 latency for all operations (the multi-tier caching adds variance)
The key insight from Habitat is that conversation storage is not a generic database problem. It’s a specialized workload with unique access patterns (recency-biased, user-scoped, schema-fluid) that benefits from purpose-built infrastructure. If you’re scaling agent systems beyond toy demos, you’ll eventually need something like this.