Running agents on local models sounds simple until you hit the plumbing: which model fits your hardware, how do you download and tune it, and how do you wire it into the agent framework you already use? Magnitude is a TypeScript inference server that solves the hardware-to-model-to-agent chain. It profiles your machine, recommends models that actually fit in memory, downloads them, and exposes a unified API that works with Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline.
This is not a model registry or a cloud service. It is a local server that runs on your VPS, GPU cluster, or laptop and handles the entire lifecycle from hardware detection to inference.
How Hardware Profiling Works
Magnitude starts by profiling your system to determine what models you can actually run. The CLI collects:
- Available RAM and VRAM
- CPU architecture and core count
- GPU presence and memory capacity
- Disk space for model storage
It then cross-references this against a model database that includes quantization levels, memory footprints, and inference speed benchmarks. The recommendation engine filters out models that will not fit or will run too slowly for interactive agent use.
For example, on a machine with 16 GB RAM and no GPU, Magnitude might recommend Llama 3.2 3B quantized to Q4_K_M. On a machine with 24 GB VRAM, it might suggest Qwen 2.5 Coder 14B at Q5_K_M. The profiler does not guess. It measures, then filters.
Model Download and Runtime Management
Once you select a model, Magnitude handles the download, verification, and tuning. Models are pulled from Hugging Face or other registries, checksummed, and stored in a local cache. The server manages multiple models and switches between them without restarting.
The hermes model command lets you swap models at runtime:
hermes model llama-3.2-3b-instructNo code changes. No config file edits. The server reloads the new model and continues serving requests. This is useful when you want to test a faster model for simple tasks or a larger model for complex reasoning.
Magnitude also handles quantization automatically. If you select a model that is too large for your hardware, it will download a quantized version that fits. You can override this and specify a quantization level manually if you want to trade accuracy for speed.
Agent Integration Layer
The integration layer is where Magnitude earns its keep. Instead of forcing you to rewrite your agent to call a new API, it exposes an OpenAI-compatible endpoint. Any agent that can talk to OpenAI can talk to Magnitude.
Here is what the integration looks like for a typical agent:
import { OpenAI } from "openai";
const client = new OpenAI({
baseURL: "http://localhost:8080/v1",
apiKey: "not-needed-for-local",
});
const response = await client.chat.completions.create({
model: "llama-3.2-3b-instruct",
messages: [{ role: "user", content: "Explain async/await" }],
});The agent does not know it is talking to a local model. It sends the same JSON payloads it would send to OpenAI, and Magnitude translates them into llama.cpp or vLLM calls under the hood.
For agents that do not use the OpenAI SDK, Magnitude also exposes raw HTTP endpoints. You can POST to /v1/chat/completions with a standard OpenAI payload and get a standard response.
Supported Agents and Protocols
Magnitude works with:
- Pi: A coding agent that uses OpenAI-compatible APIs
- OpenCode: An open-source code generation agent
- Hermes: A general-purpose agent framework
- OpenClaw: A web automation agent
- Codex: GitHub Copilot-style code completion
- Claude Code: Anthropic-compatible agents
- Oh My Pi: A Pi variant with extended tooling
- Cline: A CLI-based agent runner
Each agent has slightly different expectations for streaming, tool calls, and context windows. Magnitude normalizes these differences. If an agent expects streaming tokens, Magnitude streams them. If it expects a single JSON response, Magnitude buffers and returns one.
Tool calls are passed through as-is. Magnitude does not interpret or execute them. It forwards the tool call JSON to the agent, waits for the agent to execute it, and then sends the result back to the model for the next turn.
Deployment Shapes
Magnitude runs in three primary configurations:
| Deployment | Use Case | Trade-offs |
|---|---|---|
| Local laptop | Development and testing | Limited to smaller models, no GPU acceleration, but zero latency and full privacy |
| $5 VPS | Lightweight production agents | CPU-only inference, slower but predictable cost, good for batch or async workflows |
| GPU cluster | High-throughput production | Fast inference, supports large models, requires infrastructure and cost management |
You can also run Magnitude in a serverless environment, but cold starts are brutal. A 7B model takes 10-15 seconds to load into memory, which makes it unsuitable for synchronous request/response patterns. Serverless works if you can keep the instance warm or if your agent can tolerate startup delays.
State Management and Observability
Magnitude does not manage conversation state. That is the agent’s job. The server is stateless. Each request includes the full conversation history, and Magnitude returns a single completion. If you want to persist state, you do it in the agent layer.
Observability is minimal but functional. Magnitude logs:
- Model load times
- Inference latency per request
- Token throughput
- Memory usage
Logs are written to stdout in JSON format, so you can pipe them into your existing observability stack. There is no built-in dashboard or metrics endpoint. If you want Prometheus metrics, you write a scraper.
Security Boundaries
Magnitude runs on localhost by default. If you expose it to the network, you are responsible for authentication and rate limiting. The server does not include built-in auth because it assumes you are running it behind a reverse proxy or within a private network.
The OpenAI-compatible API accepts an apiKey parameter, but Magnitude ignores it. This is intentional. The server is designed for local use, where the security boundary is the network perimeter, not the API key.
If you need multi-tenancy or per-user rate limits, you add them in the proxy layer. Magnitude does not try to solve those problems.
Likely Failure Modes
Model does not fit in memory. The profiler is conservative, but it can still recommend a model that causes OOM crashes if other processes are consuming RAM. Monitor memory usage and downgrade to a smaller model if you see instability.
Inference is too slow. CPU-only inference on large models can take 30+ seconds per response. If your agent expects sub-second latency, you need a GPU or a smaller model.
Agent expects a feature Magnitude does not support. The OpenAI API surface is large. Magnitude implements the most common endpoints (chat completions, embeddings), but it does not support fine-tuning, moderation, or image generation. If your agent needs those, you will hit 404s.
Model switching breaks in-flight requests. If you swap models while a request is being processed, that request will fail. The server does not queue requests during model reloads. You need to drain traffic before switching.
Technical Verdict
Use Magnitude if:
- You want to run agents on local models without vendor lock-in
- You need hardware-aware model selection and do not want to manually benchmark every model
- You are already using an agent that speaks OpenAI-compatible APIs
- You want to test multiple models without rewriting integration code
Avoid Magnitude if:
- You need sub-second inference and do not have a GPU
- You require built-in authentication, rate limiting, or multi-tenancy
- Your agent depends on OpenAI features beyond chat completions (fine-tuning, moderation, DALL-E)
- You want a managed service with uptime guarantees and support
Magnitude is plumbing. It does one thing well: it profiles your hardware, picks a model that fits, and exposes it through a standard API. If you need more than that, you build it on top.