Rapid-MLX: How 0.08s Cached TTFT and 17 Tool Parsers Make Local Inference 4× Faster Than Ollama

Rapid-MLX claims 4.2× faster inference than Ollama on Apple Silicon, with 0.08s cached time-to-first-token and 100% tool calling success. It ships 17 tool parsers, prompt caching, and a drop-in OpenAI API replacement. The project hit trending #7 on GitHub Python with 3,355 stars, positioning itself as the runtime layer beneath coding agents like Claude Code, Cursor, and Aider.

The speed claims are bold. The architecture is specific. Here’s what’s under the hood.

Why Speed Matters for Local Agents

Most agentic workflows involve multi-turn conversations with tool calls. Each turn waits for the model to emit a tool invocation, the orchestrator executes it, and the model resumes. Latency compounds. A 2-second TTFT becomes 10 seconds across five turns. A 0.08s cached TTFT becomes 0.4 seconds.

Rapid-MLX targets the cached case: when the model has already seen the system prompt, conversation history, or tool definitions. This is the common path in agent loops. The first turn pays full inference cost. Subsequent turns hit the cache.

The 4.2× claim compares Rapid-MLX to Ollama on the same hardware (M-series Macs) with the same model weights. The difference is in the inference engine and cache strategy, not the model itself.

Prompt Caching Architecture

Rapid-MLX uses MLX’s KV cache to store intermediate attention states. When a prompt prefix matches a cached entry, the engine skips recomputation and jumps to the new tokens.

Key mechanics:

Prefix matching: The cache key is the token sequence hash. If the first N tokens match, the engine reuses the KV states for those tokens.
Cache eviction: LRU policy with configurable size limits. Default is 2GB.
Cache granularity: Per-request, not per-model. Each conversation thread gets its own cache entry.

This is not novel. OpenAI and Anthropic offer prompt caching in their APIs. The difference is local control: you own the cache, you tune the eviction policy, and you don’t pay per-token for cache hits.

The 0.08s cached TTFT assumes a warm cache and a prompt that reuses 80% of the previous turn’s tokens. In practice, this means:

System prompts are cached across all turns.
Tool definitions are cached unless you add new tools mid-conversation.
Conversation history is cached incrementally.

Cold-start TTFT is higher (1-2s for a 7B model on M1 Max), but the cached path dominates in agent loops.

17 Tool Parsers: Why Diversity Matters

Rapid-MLX ships 17 tool-calling parsers, each targeting a different model family’s output format. This is not a bug. It’s a design choice driven by the reality of local inference.

Tool calling is not standardized. OpenAI uses a JSON schema with function_call objects. Anthropic uses XML tags. Open-weight models (Qwen, DeepSeek, Llama) use ad-hoc formats: JSON in code blocks, YAML, plain text with delimiters, or custom tokens.

When you run a model locally, you don’t get a clean API response. You get raw tokens. The orchestrator must parse those tokens into structured tool calls. If the parser fails, the agent loop breaks.

Rapid-MLX’s parser registry includes:

OpenAI-style JSON: {"name": "search", "arguments": {"query": "..."}}
Anthropic XML: <tool_use><tool_name>search</tool_name><parameters>...</parameters></tool_use>
Code block JSON: Markdown fences around JSON objects
YAML blocks: For models trained on YAML examples
Plain text with delimiters: TOOL: search ARGS: query="..."
Custom tokens: Special tokens like <|tool_call|> for fine-tuned models

Each parser is a regex or state machine that extracts tool name and arguments from the model’s output. The engine tries parsers in priority order until one succeeds.

The 100% tool calling success rate claim means: given a model that can produce tool calls in any of these formats, Rapid-MLX will parse it correctly. This does not mean every model will produce valid tool calls. It means the parser won’t be the bottleneck.

Reasoning Separation: Splitting Thought from Action

Reasoning separation is a runtime optimization that splits model output into two phases: reasoning and tool invocation.

Standard flow:

Model generates tokens: “I need to search for X. Let me call the search tool with query Y.”
Parser extracts tool call from the full output.
Orchestrator executes tool.
Model resumes with tool result.

Reasoning separation flow:

Model generates reasoning tokens: “I need to search for X.”
Engine detects reasoning phase (no tool syntax yet).
Model generates tool call tokens: <tool_use>search</tool_use>
Parser extracts tool call.
Orchestrator executes tool.
Model resumes with tool result, skipping reasoning re-generation.

The optimization: reasoning tokens are cached separately from tool call tokens. When the model resumes after a tool execution, it doesn’t regenerate the reasoning. It jumps straight to the next action.

This saves tokens and latency in multi-tool workflows. If an agent needs to call three tools in sequence, it reasons once and executes three times, instead of reasoning three times.

The trade-off: reasoning separation requires the model to emit structured output. If the model mixes reasoning and tool syntax in unpredictable ways, the parser can’t split cleanly. This works best with models fine-tuned for tool use (Qwen2.5, DeepSeek-R1, Llama-3.3-Instruct).

Drop-In OpenAI API Compatibility

Rapid-MLX exposes a FastAPI server that implements the OpenAI /v1/chat/completions endpoint. This means any tool that speaks OpenAI’s API can point at Rapid-MLX without code changes.

Supported features:

Streaming responses (stream=true)
Tool definitions (tools array in request)
Tool choice (tool_choice parameter)
System messages
Temperature, top-p, max tokens

Not supported:

Function calling (deprecated OpenAI feature)
Logprobs
Embeddings endpoint (separate MLX library)

The API compatibility layer translates OpenAI-style tool definitions into the model’s native format. For example, if you send an OpenAI tool schema to a Qwen model, Rapid-MLX converts it to Qwen’s expected format before inference.

This is the integration point for coding agents. Claude Code, Cursor, and Aider all support custom OpenAI-compatible endpoints. You configure the base URL, and the agent uses your local model instead of hitting OpenAI’s API.

Architecture: Inference Engine and Orchestration Boundary

Rapid-MLX is an inference engine, not an orchestration framework. It handles:

Model loading and quantization
Token generation with KV caching
Tool call parsing
API server

It does not handle:

Multi-agent coordination
State persistence across sessions
Tool execution (you provide the tool implementations)
Retry logic or error recovery

The boundary is clear. Rapid-MLX sits between your orchestrator (LangChain, LlamaIndex, custom code) and the model weights. It replaces the OpenAI API call with a local inference call.

Your orchestrator still owns:

Tool definitions and implementations
Conversation state management
Error handling and retries
Multi-turn loop control

This is the right boundary. Inference engines should not dictate orchestration patterns. They should be fast, reliable, and compatible.

Performance Comparison: Rapid-MLX vs. Ollama

Metric	Rapid-MLX	Ollama	Notes
Cached TTFT	0.08s	0.34s	Warm cache, 80% prefix match
Cold TTFT	1.2s	1.8s	First turn, no cache
Tokens/sec (7B)	45	38	M1 Max, 32GB RAM
Tool call parse rate	100%	~85%	Across 17 model families
Memory overhead	2GB cache	1GB cache	Default settings
API compatibility	OpenAI + Anthropic	OpenAI	Rapid-MLX adds Anthropic XML

The 4.2× claim comes from the cached TTFT comparison (0.08s vs. 0.34s). This is the best-case scenario. In cold-start or long-context cases, the gap narrows to 1.5×.

Ollama’s strength is simplicity: one command to install, one command to run. Rapid-MLX’s strength is speed and tool-calling robustness. If you’re building an agent that makes 20 tool calls per session, the cached TTFT difference adds up.

Deployment Shape and Failure Modes

Rapid-MLX runs as a local HTTP server. Typical deployment:

Install via Homebrew or pip.
Download model weights (GGUF or MLX format).
Start server: rapid-mlx serve --model qwen2.5-7b
Point your agent at http://localhost:8080/v1/chat/completions.

Failure modes:

Out of memory: M1 Macs with 8GB RAM can’t run 13B models. The server crashes or swaps to disk, killing performance.
Cache thrashing: If your agent switches contexts frequently (different system prompts, different tool sets), the cache hit rate drops. You pay cold-start cost every time.
Parser mismatch: If you use a model that outputs tool calls in a format not covered by the 17 parsers, the agent loop breaks. You’ll need to add a custom parser.
Quantization artifacts: Rapid-MLX supports 4-bit and 8-bit quantization. Lower precision means faster inference but higher error rates in tool argument extraction.

Observability is minimal. The server logs token counts and latency per request, but there’s no built-in tracing or metrics export. You’ll need to wrap the API calls with your own instrumentation.

Security Boundaries

Rapid-MLX runs on your machine. The model weights, cache, and conversation state never leave your hardware. This is the primary security benefit: no data exfiltration to third-party APIs.

Risks:

Prompt injection: If your agent accepts user input and passes it to the model, an attacker can inject tool calls. Rapid-MLX does not sanitize inputs. You must validate tool arguments before execution.
Tool execution sandbox: Rapid-MLX parses tool calls but does not execute them. Your orchestrator executes tools. If you don’t sandbox tool execution (e.g., running shell commands, file I/O), the model can trigger arbitrary code.
Model supply chain: You download model weights from Hugging Face or other mirrors. If the weights are poisoned, the model can output malicious tool calls. Verify checksums.

The inference engine is not the security boundary. Your orchestrator is. Rapid-MLX gives you speed and compatibility. You own input validation, output sanitization, and tool execution safety.

When to Use Rapid-MLX

Use Rapid-MLX when:

You’re building an agent that makes frequent tool calls (10+ per session).
You need sub-100ms cached TTFT for responsive UX.
You want to run coding agents (Claude Code, Cursor, Aider) on local models.
You have an M-series Mac with 16GB+ RAM.
You need to support multiple model families with different tool-calling formats.

Avoid Rapid-MLX when:

You need embeddings or fine-tuning (use separate MLX libraries).
You’re running on non-Apple hardware (MLX is Apple Silicon only).
You need built-in orchestration (use LangChain or LlamaIndex on top).
You need production-grade observability (add your own tracing).
You’re okay with slower inference and want simpler setup (use Ollama).

Technical Verdict

Rapid-MLX is a well-scoped inference engine. It does one thing (fast local inference with tool calling) and does it well. The 17-parser strategy is pragmatic, not elegant, but it solves a real problem: open-weight models don’t agree on tool-calling formats.

The cached TTFT advantage is real and measurable. If your agent makes 20 tool calls per session, you save 5 seconds compared to Ollama. That’s the difference between a responsive coding assistant and a sluggish one.

The API compatibility layer is the right integration point. You can swap Rapid-MLX in without rewriting your orchestrator. The lack of built-in orchestration is a feature, not a bug. Inference engines should not dictate workflow patterns.

The main risk is cache thrashing. If your agent switches contexts frequently, the speed advantage disappears. Profile your cache hit rate before committing to Rapid-MLX in production.

For local agentic workflows on Apple Silicon, Rapid-MLX is the fastest option available. It won’t replace cloud APIs for high-scale deployments, but for developer tools and side projects, it’s the right tool.