Token costs are the silent budget killer in production agent systems. A single RAG query can pull 20 chunks at 500 tokens each. Tool outputs dump entire log files. File uploads hit context limits before the agent even starts reasoning. The usual fix is to throw money at larger context windows or rewrite prompts by hand.
Context compression middleware sits between your retrieval layer and the LLM, shrinking tool outputs, logs, files, and RAG chunks while preserving the semantic content needed for correct answers. This is not prompt engineering. It is a deployable layer that intercepts context before tokenization.
The approach claims 60-95% token reduction with maintained answer quality across three deployment patterns: library, proxy, and MCP server. These patterns target tool outputs, logs, files, and RAG chunks rather than user prompts.
Where Compression Fits in the Pipeline
Most agent architectures have three handoff points where bloat accumulates:
- RAG retrieval to context assembly: Vector search returns chunks, embeddings metadata, source citations. The agent needs the answer, not the boilerplate.
- Tool output to next reasoning step: A file reader returns 10,000 lines. The agent needs the error message on line 47.
- Multi-turn memory to prompt construction: Conversation history grows linearly. The agent needs the decision points, not the small talk.
Compression middleware intercepts at these boundaries. You get three deployment patterns:
- Library: Import a function, pass raw context, get compressed context back. Fits inline in your orchestrator.
- Proxy: HTTP endpoint that accepts context payloads and returns compressed versions. Sits between your agent runtime and external tools.
- MCP server: Model Context Protocol integration that exposes compression as a tool the agent can call directly.
The library pattern gives you control. The proxy pattern decouples compression from your agent code. The MCP server pattern lets the agent decide when to compress, which is useful when you have mixed workloads (some queries need full context, others do not).
How Compression Preserves Semantic Fidelity
The core challenge is not shrinking text. It is shrinking text without breaking the agent’s ability to answer questions. A naive approach strips whitespace and truncates. That works until the truncated section contains the error code you need.
Semantic compression uses a smaller model to rewrite context into dense summaries that preserve task-relevant details. The process:
- Chunk the input into logical units (paragraphs, log entries, JSON objects).
- Score each chunk for relevance to the current query or task.
- Rewrite high-scoring chunks into compressed forms that retain key entities, relationships, and data points.
- Drop low-scoring chunks entirely or replace them with one-line summaries.
The scoring step is critical. If you compress everything uniformly, you lose the signal. If you compress nothing, you pay full token costs. The middleware needs to know what the agent is trying to do.
This is where the library pattern shines. You pass the user query alongside the raw context. The proxy and MCP patterns require the agent to include the query in the compression request. If your orchestrator does not pass task context to tools, compression becomes a blind operation.
Failure Modes and Detection
Compression fails in three predictable ways:
- Over-compression removes critical details: The agent hallucinates because the compressed context no longer contains the fact it needs.
- Under-compression wastes tokens: You pay for a compression layer that does not compress enough to justify the latency.
- Compression introduces errors: The rewrite model misinterprets technical terms, inverts logic, or drops numeric precision.
You cannot catch these failures with unit tests. You need runtime validation:
- Answer consistency checks: Run the same query against compressed and uncompressed context. Compare outputs. If they diverge, log the diff and the compression ratio.
- Token budget alerts: Track actual token savings per request. If compression consistently underperforms the target ratio, the scoring model is miscalibrated.
- Semantic drift metrics: Embed the original and compressed context. Measure cosine distance. Large distances indicate the compression model is rewriting too aggressively.
The MCP server pattern makes validation easier because the agent can request both compressed and uncompressed versions, compare them, and decide which to use. This doubles the token cost for validation queries, but it gives you a ground truth signal.
Deployment Shapes
| Pattern | Latency Overhead | Coupling | Observability |
|---|---|---|---|
| Library | In-process (reported 50-200ms) | Tight (imports) | Full (same trace) |
| Proxy | Network + queue (reported 100-500ms) | Loose (HTTP) | Separate service |
| MCP Server | Agent-controlled (reported 200-800ms) | Medium (protocol) | Agent-controlled |
The library pattern is fastest but locks you into the compression library’s language and dependencies. The proxy pattern adds network hops but lets you scale compression independently. The MCP server pattern gives the agent control but requires the agent to understand when compression is useful.
If you run multiple agents in different languages (Python orchestrator, TypeScript UI agent, Go data agent), the proxy pattern is the only practical choice. If you have a monolithic Python agent, the library pattern is simpler.
Related Infrastructure Considerations
Compression middleware sees everything: user queries, tool outputs, RAG chunks, conversation history. This creates operational concerns common to agent infrastructure:
Data handling: If the compression service logs inputs for debugging, you are writing sensitive data to disk. Run compression in the same security boundary as your LLM calls. If the LLM is in a VPC, the compression proxy should be too. Do not log raw inputs. Log compression ratios, token counts, and error rates.
Input validation: Validate compressed output before passing it to the LLM. If compression introduces new prompt markers (triple backticks, system tags), strip them. The MCP server pattern is the riskiest because the agent controls when compression happens.
Observability: You need metrics to operate compression in production. Track compression ratio per request, compression latency, answer consistency rate, and token cost savings. The library pattern lets you emit these metrics inline. The proxy pattern requires structured logging and a metrics collector. The MCP server pattern depends on the agent runtime’s observability stack.
If you run compression as a proxy, add a /health endpoint that returns current compression ratio and latency percentiles. This lets your orchestrator decide whether to use compression or bypass it when the service is slow.
Technical Verdict
Use context compression middleware when:
- Your agent makes 100+ LLM calls per day and token costs are a line item in your budget.
- RAG retrieval or tool outputs regularly exceed 50% of your context window.
- You have already optimized prompts and chunking strategies but still hit token limits.
- You can tolerate added latency (reported 50-800ms depending on deployment pattern) in exchange for token cost reduction.
Avoid it when:
- Your queries are short and context fits comfortably in the window.
- Latency is more important than cost (compression adds processing time per request).
- Your agent needs exact text matches or numeric precision that compression might distort.
- You cannot validate answer consistency between compressed and uncompressed contexts.
Start with the library pattern for proof of concept. Measure token savings and answer consistency over 1,000 queries. If savings meet your cost targets and consistency stays acceptable for your use case, deploy the proxy pattern for production scale. Reserve the MCP server pattern for agents that need dynamic control over compression.