System prompts are text. They go to the same model that would have answered without them. They change no weights and add no tools. The skeptical reading follows: a specialist prompt is a checklist, the model reads the checklist, and whatever the checklist buys is too small to justify maintaining hundreds of them.
Willian Pinho ran the experiment. He deleted the system prompt from 19 agent configurations and measured what broke. Thirteen survived above the noise floor. Two scored zero without their prompt. Four results say more about the test harness than the prompts.
Boris Cherny, who built Claude Code, ran the same kind of ablation and found the opposite. Anthropic deleted 80% of the system prompt for the Opus 5 release, then deleted the rest as an experiment. The model got slightly more intelligent without the prompts.
The difference is scope. Pinho’s agents are narrow specialists. Claude Code is a general-purpose coding assistant. The methodology matters because most production agent systems sit somewhere between those two extremes.
The Ablation Protocol
Pinho followed Charlie Hills’ Six-Month Audit checklist, a published protocol for finding instructions that newer models have outgrown. The checklist has seven delete checks and five add checks. The delete checks run against live guidance, never hardcoded copies.
The ablation test is simple:
- Run the agent configuration with its system prompt
- Run the same configuration with the system prompt deleted
- Measure the delta
The measurement is where it gets interesting. Pinho established a noise floor by running the same configuration multiple times with the prompt intact. If the delta between prompt and no-prompt is smaller than the variance between runs with the prompt, the prompt does not steer behavior in a measurable way.
What Gets Measured
The metrics depend on the agent’s job. For a code review agent, you might measure:
- Number of issues flagged
- False positive rate
- Coverage of known vulnerability patterns
- Time to first response
For a data extraction agent:
- Field extraction accuracy
- Schema compliance rate
- Handling of edge cases (nulls, malformed input)
- Token efficiency
Pinho does not publish his exact metrics, but the pattern is clear. You need a ground truth or a stable baseline. You need multiple runs to establish variance. You need the delta to exceed that variance by a margin you can defend.
Results Across 19 Configurations
| Outcome | Count | Interpretation |
|---|---|---|
| Survived above noise floor | 13 | Prompt steers behavior measurably |
| Scored zero without prompt | 2 | Prompt is load-bearing |
| Failed below noise floor | 4 | Test harness variance too high or prompt is redundant |
The two zero-score configurations are the most interesting. These are agents where the system prompt is not just helpful, it is structural. Without it, the agent does not recognize its task boundaries or tool invocation patterns.
The thirteen survivors tell a different story. The prompt helps, but the model’s priors and few-shot examples carry most of the weight. These prompts are tuning knobs, not foundations.
The four noisy results expose a testing problem. If your baseline variance is high, you cannot measure small deltas. This happens when:
- The task has multiple valid solutions
- The model’s sampling temperature is too high
- The evaluation metric is too coarse
- The test set is too small
Why Anthropic Saw the Opposite
Cherny’s talk describes deleting the Claude Code system prompt and seeing intelligence improve. The explanation is scope and constraint.
Claude Code handles:
- Multi-file edits
- Dependency resolution
- Test generation
- Refactoring across modules
A system prompt that tries to cover all of those cases becomes a liability. It adds tokens without adding signal. It constrains the model in ways that hurt generalization.
Pinho’s agents are specialists:
- One reviews pull requests
- One extracts structured data from invoices
- One routes support tickets
A specialist prompt can be precise. It can say “extract these six fields” or “flag these three vulnerability patterns.” The narrower the task, the more a prompt can steer without constraining.
The trade-off is maintenance. Pinho maintains 19 prompts. Anthropic maintains one. When the model improves, Anthropic deletes instructions. Pinho has to test 19 configurations to find out which instructions are now redundant.
Implementation: Running Your Own Ablation
Here is a minimal harness for null-prompt ablation testing:
import asyncio
from dataclasses import dataclass
from typing import Optional
@dataclass
class AblationResult:
config_id: str
with_prompt_score: float
without_prompt_score: float
baseline_variance: float
@property
def delta(self) -> float:
return self.with_prompt_score - self.without_prompt_score
@property
def is_significant(self) -> bool:
# Delta must exceed 2x baseline variance
return abs(self.delta) > (2 * self.baseline_variance)
async def run_agent(config_id: str, system_prompt: Optional[str], test_case: dict) -> float:
# Your agent invocation here
# Return a numeric score (accuracy, coverage, etc.)
pass
async def establish_baseline_variance(config_id: str, system_prompt: str, test_cases: list, runs: int = 5) -> float:
scores = []
for _ in range(runs):
run_scores = [await run_agent(config_id, system_prompt, tc) for tc in test_cases]
scores.append(sum(run_scores) / len(run_scores))
mean = sum(scores) / len(scores)
variance = sum((s - mean) ** 2 for s in scores) / len(scores)
return variance ** 0.5 # Standard deviation
async def ablation_test(config_id: str, system_prompt: str, test_cases: list) -> AblationResult:
# Establish baseline with prompt
baseline_variance = await establish_baseline_variance(config_id, system_prompt, test_cases)
# Score with prompt
with_prompt_scores = [await run_agent(config_id, system_prompt, tc) for tc in test_cases]
with_prompt_score = sum(with_prompt_scores) / len(with_prompt_scores)
# Score without prompt
without_prompt_scores = [await run_agent(config_id, None, tc) for tc in test_cases]
without_prompt_score = sum(without_prompt_scores) / len(without_prompt_scores)
return AblationResult(
config_id=config_id,
with_prompt_score=with_prompt_score,
without_prompt_score=without_prompt_score,
baseline_variance=baseline_variance
)The key is the is_significant property. If your delta does not exceed twice the baseline variance, you cannot claim the prompt matters. Adjust the multiplier based on your risk tolerance.
Failure Modes and Observability
Ablation testing fails when:
Test set is not representative. If your test cases do not cover the edge cases where the prompt matters, you will underestimate its value. This is especially true for safety and constraint prompts.
Evaluation metric is misaligned. A prompt might reduce hallucination rate without changing accuracy. If you only measure accuracy, you miss the benefit.
Model version drift. The prompt that mattered for GPT-4 might be redundant for GPT-4.5. Run ablation tests after every model upgrade.
Sampling temperature hides signal. High temperature increases variance. Lower it during ablation testing to isolate prompt impact from stochastic noise.
For observability, log:
- Prompt version hash
- Model version
- Test case identifiers
- Raw scores (not just aggregates)
- Timestamp and run ID
Store these in a time-series database so you can track prompt efficacy over time. When you upgrade the model, you can replay the ablation test and see which prompts became redundant.
When Prompts Matter
System prompts have measurable impact when:
- The task is narrow and the prompt can be precise
- The model’s priors do not align with your use case
- You need to enforce constraints (output format, safety boundaries)
- The agent has access to tools and the prompt defines invocation patterns
Prompts become redundant when:
- The model’s priors already cover the task
- The prompt restates general instructions (“be helpful”)
- The task is so broad that the prompt cannot be precise
- Newer models have internalized the behavior you were prompting for
Technical Verdict
Run null-prompt ablation when you maintain more than five agent configurations or when you upgrade the underlying model. Establish a noise floor by running the same configuration multiple times with the prompt intact. Only claim the prompt matters if the delta exceeds twice the baseline variance.
Avoid this methodology if your evaluation metrics are unstable or your test set is too small. You will get false negatives (prompts that matter but do not show up in the delta) and false positives (noise that looks like signal).
Use this when you need to justify the maintenance cost of specialist prompts or when you are deciding whether to consolidate multiple agents into one general-purpose configuration. The data will tell you whether your prompts are steering behavior or restating the model’s defaults.