MADCAP is a CLI tool that implements multi-agent debate through a critic-and-profile loop. Three LLM agents (Answerer, Critic, Judge) argue until they converge on a response. The architecture is interesting because it avoids a central orchestrator. Instead, the Judge agent enforces termination conditions and the Critic agent maintains cross-round state to prevent circular arguments.
The project is a toy, but the plumbing is visible. Architectural decisions are not abstracted away by production tooling or enterprise frameworks. That makes it useful for understanding how debate systems handle convergence, state persistence, and the failure modes that occur when agents agree too quickly or argue forever.
Architecture: Three Agents, One Loop
MADCAP runs a structured debate in rounds. Each round follows the same sequence:
- Answerer proposes a response
- Critic challenges the response with specific objections
- Judge evaluates both sides and decides whether to continue or terminate
The Judge never sends the user’s original question to the debate. It first rewrites the question into a neutral, precise form. This rewrite step acts as an information bottleneck. The agents debate the rewritten question, not the user’s phrasing. This prevents the Answerer from anchoring on ambiguous language.
The source describes the Critic as maintaining a profile across rounds. The profile is a running list of objections the Critic has already raised. Before generating a new critique, the Critic checks this profile to avoid repeating itself. This cross-round profiling loop is the mechanism that forces convergence.
State Management: What Persists Between Rounds
Each agent maintains different state:
| Agent | Within-Round State | Cross-Round State |
|---|---|---|
| Answerer | Current draft response | None (starts fresh each round) |
| Critic | Current critique | Profile of past objections |
| Judge | Debate transcript | Confidence score, termination signal |
The Critic’s profile is the key convergence mechanism. After each round, the Judge appends the Critic’s objections to the profile. In the next round, the Critic receives this profile as context. If the Critic cannot generate a new objection that differs from the profile, the debate terminates.
The Judge maintains a confidence score that increases when the Answerer addresses objections and decreases when the Critic surfaces new problems. When confidence crosses a threshold or the Critic runs out of novel objections, the Judge writes a verdict and exits.
Termination Conditions: How Debate Ends
MADCAP uses three termination signals:
- Confidence threshold: Judge’s confidence score exceeds a preset value (specific threshold not disclosed in the source)
- Objection exhaustion: Critic cannot generate a new objection distinct from the profile
- Round limit: Hard cap on debate rounds (prevents infinite loops)
The round limit is a safety valve. In practice, objection exhaustion is the most common termination path. The Critic’s profile grows with each round, and eventually the Critic’s LLM call returns a critique that duplicates a past objection. The Judge detects this duplication and terminates.
The confidence threshold is less reliable. LLMs are poor at self-calibration, so the Judge’s confidence score is noisy. The project acknowledges this: the confidence label in the final output is informational, not a hard guarantee.
Encoding Debate Rules: Prompts and Configs
Agent personas and debate rules are encoded as prompt templates. Each agent has a system prompt that defines its role and constraints. The Answerer is instructed to be confident, the Critic is instructed to be skeptical, and the Judge is instructed to be impartial.
The prompts are not runtime-modifiable through the CLI. They are hardcoded in the source. This is a deliberate simplification. The project treats the debate structure as fixed and focuses on the convergence loop, not on tuning agent behavior.
The rewritten question, debate transcript, and Critic profile are passed as structured context in each LLM call. The CLI serializes this state as JSON between rounds. There is no database or external state store. Everything lives in memory during a single CLI invocation.
Conceptual Flow: Critic Profile Check
Here is a conceptual representation of how the Critic checks its profile before generating a new objection:
function generate_critique(answerer_response, critic_profile):
# Critic receives its past objections as context
prompt = """
You are a skeptical critic. The answerer proposed:
{answerer_response}
You have already raised these objections:
{critic_profile}
Generate a NEW objection that is distinct from your past objections.
If you cannot, respond with "NO_NEW_OBJECTION".
"""
critique = llm_call(prompt)
# Judge detects this signal to terminate the debate
if "NO_NEW_OBJECTION" in critique:
return None
return critiqueThe Judge appends the critique to the profile after each round. The next round’s Critic call includes the updated profile. This is the profiling loop. The source does not specify whether the Judge uses string matching or semantic similarity to detect duplicate objections.
Failure Modes: When Debate Breaks Down
MADCAP’s author flags several failure modes in the project documentation:
- Premature agreement: Agents converge too quickly because the Critic is not adversarial enough
- Circular arguments: Critic repeats objections in different phrasing, bypassing the profile check
- Judge bias: Judge favors the Answerer or Critic based on prompt phrasing
- Confidence drift: Judge’s confidence score does not correlate with actual answer quality
The profiling loop mitigates circular arguments but does not eliminate them. LLMs are good at paraphrasing, so the Critic can restate an objection in a way that looks new to the Judge. The project does not implement semantic similarity checks on the profile. That would require embeddings and a distance threshold, which adds complexity.
Premature agreement is harder to fix. If the Critic is not adversarial enough, the debate ends after one or two rounds. The project suggests tuning the Critic’s system prompt to be more aggressive, but this is a manual process.
Observability: What You See in the CLI
MADCAP outputs the full debate transcript to the terminal. You see:
- The rewritten question
- Each round’s Answerer response
- Each round’s Critic objections
- The Judge’s verdict and confidence label
This is useful for debugging. You can watch the debate diverge or converge. You can see when the Critic runs out of objections. You can see when the Judge’s confidence score jumps or drops.
The source does not mention structured logging or telemetry beyond the terminal output. The CLI is the observability layer. If you want metrics, you parse the terminal output.
Considerations: Debate vs. Other Agent Patterns
The following table compares multi-agent debate to other common agent coordination patterns. These are editorial observations based on the MADCAP architecture, not claims from the source:
| Approach | Convergence Mechanism | State Management | Common Failure Mode |
|---|---|---|---|
| Multi-Agent Debate | Critic profile, confidence | Cross-round profile | Circular arguments, premature agreement |
| Orchestrator-Driven | Central planner decides | Orchestrator holds state | Orchestrator becomes coordination bottleneck |
| Consensus Voting | Majority or weighted vote | Each agent votes | Groupthink without dissent mechanism |
Debate systems like MADCAP avoid a central orchestrator. The Judge is not a planner. It does not decide what the Answerer should say next. It only decides when to stop. This is a weaker form of control, but it allows the Answerer and Critic to explore the solution space more freely.
The trade-off is that debate systems are harder to steer. If the Critic goes off-topic, the Judge may not catch it until several rounds have passed. Orchestrator-driven systems can intervene earlier because the orchestrator sees the full plan.
Technical Verdict
Use MADCAP’s debate pattern when:
- You need to challenge a single LLM’s output with structured critique
- You want to avoid a central orchestrator that becomes a bottleneck
- You can tolerate multiple LLM calls per user query (higher latency, higher cost)
- You need an audit trail of how the answer was reached
Avoid this pattern when:
- You need low-latency responses (debate adds 3x to 5x LLM calls)
- You cannot afford the token cost of multi-round debate
- Your problem does not benefit from adversarial critique (e.g., simple lookups)
- You need deterministic convergence guarantees (debate can loop or stall)
The critic-and-profile loop is the interesting piece. It is a lightweight convergence mechanism that does not require a central planner. The trade-off is that it relies on the Critic’s ability to generate novel objections, which is not guaranteed. If you implement this pattern, budget for tuning the Critic’s prompt and for handling the failure modes the author flags.
Source Links
References
The source article references the following papers on multi-agent debate systems:
- Du et al., “Improving Factuality and Reasoning in Language Models through Multiagent Debate” (cited as [1] in source)
- Liang et al., multi-agent debate research (cited as [11] in source)
- “AI safety via debate” concept (cited as [2] in source)
Full citations and DOIs are available in the original source article.