Harness Tuner: Eval-Driven Loops Turn Agent Tuning from Guesswork into Measurement

Most agent tuning happens in a loop: tweak the prompt, run it manually, check the output, repeat. Harness Tuner replaces that manual cycle with an automated optimization loop that plans experiments, measures performance on a training set, validates on a held-out set, and never touches the test set until you explicitly promote a candidate.

The tool targets three tuning dimensions: agent tools (which functions the agent can call), prompts (system and user instructions), and model selection (which LLM backend to use). Instead of guessing which change will improve performance, you define evals and let the loop measure.

How the Tuning Loop Works

Harness Tuner is an agentic optimizer. It reads task definitions from Kiln (a structured eval format), plans experiments, runs them, and tracks results. The loop follows a train/val/test discipline borrowed from ML pipelines:

Train set: The optimizer runs experiments here, measuring performance and iterating.
Validation set: Promising candidates are tested here to check for overfitting.
Test set: Locked until you explicitly promote a configuration for final evaluation.

The optimizer is itself an agent. It decides which dimension to tune next based on current performance metrics. If tool selection is the bottleneck, it experiments with different tool combinations. If the prompt is unclear, it generates variations and measures which one improves task completion.

Experiment Planning

The optimizer maintains a state machine:

Baseline measurement: Run the current configuration on the train set to establish a performance floor.
Hypothesis generation: The agent proposes experiments (e.g., “Try adding a web search tool” or “Simplify the system prompt”).
Execution: Run each experiment on the train set and log metrics.
Validation: If train performance improves, test on the validation set to confirm the gain is real.
Promotion: If validation passes, mark the configuration as a candidate for production.

The loop never reads the test set during optimization. This prevents the classic ML mistake of tuning to your test data.

Eval Harness Format

Harness Tuner consumes Kiln task definitions. A task is a JSON structure that includes:

Input: The data the agent receives.
Expected output: The correct answer or a scoring function.
Tools: Available functions the agent can call.
Prompt template: The system and user instructions.

Here’s a simplified example:

{
  "task_id": "summarize_article",
  "input": {
    "url": "https://example.com/article"
  },
  "expected_output": {
    "summary": "A concise summary of the article",
    "key_points": ["point1", "point2"]
  },
  "tools": ["fetch_url", "extract_text"],
  "prompt": "You are a summarization agent. Fetch the article and extract key points."
}

The optimizer runs this task with different configurations and compares outputs to the expected result. Scoring can be exact match, semantic similarity, or a custom function.

Handling Non-Determinism

LLM outputs are stochastic. Harness Tuner runs each experiment multiple times (configurable, default is 3) and aggregates results. It tracks:

Mean score: Average performance across runs.
Variance: How much the output fluctuates.
Pass rate: Percentage of runs that meet the success threshold.

High variance signals that a configuration is unstable. The optimizer penalizes configurations with high variance even if mean performance is good.

Tuning Dimensions

Tool Selection

The optimizer can add, remove, or swap tools. If an agent has access to fetch_url, search_web, and extract_text, the optimizer might test:

Removing search_web to see if it’s a distraction.
Adding summarize_text to improve output quality.
Swapping fetch_url for a cached version to reduce latency.

Each change is measured on the train set. If removing a tool improves performance (e.g., the agent was misusing it), the optimizer keeps the change.

Prompt Optimization

The optimizer generates prompt variations using an LLM. It might:

Simplify verbose instructions.
Add examples to clarify expected behavior.
Reorder instructions to emphasize critical steps.

Prompt changes are risky because they can degrade performance in unexpected ways. The optimizer runs multiple variations in parallel and picks the one with the best train score, then validates it.

Model Selection

The optimizer can swap the underlying LLM. If you start with GPT-4, it might test:

Claude 3.5 Sonnet for tasks that need long context.
GPT-4o-mini for tasks where speed matters more than quality.
A local model for tasks with sensitive data.

Model selection is measured on cost, latency, and accuracy. The optimizer balances these metrics based on weights you provide.

Overfitting Guardrails

Automated tuning can overfit to the train set. Harness Tuner includes several guardrails:

Validation gate: No configuration is promoted without passing validation.
Diversity penalty: The optimizer avoids making too many similar changes in a row.
Rollback on regression: If a change hurts validation performance, it’s automatically reverted.
Test set lockout: The test set is never used during optimization. You manually trigger test evaluation when you’re ready to ship.

The optimizer also logs every experiment. You can inspect the history to see which changes were tried and why they were accepted or rejected.

Deployment Shape

Harness Tuner runs as a standalone process. You point it at a Kiln task directory, configure the optimization budget (how many experiments to run), and let it work.

harness-tuner optimize \
  --task-dir ./tasks \
  --budget 50 \
  --train-split 0.7 \
  --val-split 0.2 \
  --test-split 0.1

The optimizer writes results to a SQLite database. You can query it to see experiment history, performance trends, and promoted configurations.

For production, you export the winning configuration as a JSON file and load it into your agent runtime. Harness Tuner doesn’t deploy agents; it just finds the best configuration.

Observability

The optimizer logs every experiment to a structured format. Each log entry includes:

Experiment ID: Unique identifier.
Configuration: Tools, prompt, and model used.
Metrics: Score, latency, cost, and variance.
Dataset split: Train, val, or test.
Timestamp: When the experiment ran.

You can export logs to your observability stack (e.g., Datadog, Grafana) or query them locally. The optimizer also exposes a simple web UI that shows:

Current best configuration.
Performance over time.
Experiment history.

Failure Modes

Failure Mode	Cause	Mitigation
Overfitting to train set	Optimizer runs too many experiments on the same data	Use a large train set and strict validation gate
Unstable configurations	High variance in LLM outputs	Increase number of runs per experiment and penalize variance
Expensive experiments	Testing large models or slow tools	Set cost and latency budgets; optimizer will skip expensive configs
Prompt degradation	Optimizer generates a worse prompt that still scores well on train	Manual review of promoted prompts before production
Tool misuse	Optimizer adds a tool that the agent uses incorrectly	Monitor tool call logs and add negative examples to the eval set

When to Use Harness Tuner

Use it when:

You have a structured eval set and want to systematically improve agent performance.
You’re tuning multiple dimensions (tools, prompts, models) and manual iteration is too slow.
You need to prevent overfitting and want a train/val/test discipline.
You want experiment tracking and reproducibility for agent configurations.

Avoid it when:

You don’t have a good eval set. The optimizer is only as good as your evals.
Your agent tasks are too diverse. The optimizer works best on a narrow task domain.
You need real-time tuning. The optimization loop is batch-oriented, not online.
Your agent is already good enough. Don’t optimize for the sake of optimizing.

Technical Verdict

Harness Tuner is a closed-loop optimizer for agent configurations. It automates the tedious work of testing tool combinations, prompt variations, and model choices. The train/val/test split prevents overfitting, and the experiment tracking gives you a clear history of what worked and why.

The main risk is garbage-in-garbage-out. If your eval set is bad, the optimizer will tune to the wrong target. Spend time building good evals before you turn on automated tuning.

For teams running production agents, this is a useful primitive. It turns agent tuning from a manual art into a measured process. Just don’t expect it to fix a fundamentally broken agent design.

Agentic AI, minus the noise.