Most multi-agent systems assume you can check the output. Unit tests, linters, compilers, and API responses give you fast feedback. Economic theory development has none of that. When an LLM agent proposes a welfare theorem or market equilibrium model, no machine can tell you if it is correct. You need human judgment, but you cannot afford to review every intermediate step.
pAI-Econ-claude addresses this problem with a gated human-in-the-loop architecture. Agents generate, critique, and coordinate through a shared workspace. Specialized gates diagnose failure modes and trigger loopbacks. Human checkpoints retain authority over decisions that are expensive to reverse. The system does not try to certify correctness. It tries to make mistakes auditable and recoverable.
The architecture was evaluated on five economic theory tasks. Two blinded evaluators preferred the gated version in four out of five cases. Mean failure severity dropped from 1.58 to 1.16, and overall usefulness rose from 2.60 to 3.10. The largest gain came when a reality check rejected a false market-structure premise and a proof review caught a false welfare claim.
The No-Ground-Truth Problem
In software engineering, you can run tests. In data science, you can validate against holdout sets. In economic research, you have neither. A proposed model might be internally consistent but economically meaningless. A welfare claim might be mathematically sound but rest on false assumptions about market structure.
This creates a coordination problem for multi-agent systems:
- Generation agents produce models, proofs, and claims.
- Critique agents identify logical gaps and economic implausibility.
- Coordination agents route work between generation and critique.
- Human reviewers make final calls on costly decisions.
Without ground truth, no agent can certify that the pipeline succeeded. The best you can do is flag likely failures and route them to the right reviewer.
Architecture Components
pAI-Econ-claude organizes agents around a shared workspace of inspectable intermediate records. Each record captures one step: a model draft, a critique, a revision, or a human decision. Agents read from and write to this workspace. Gates sit between stages and decide whether to proceed, loop back, or escalate.
Shared Workspace
The workspace is a structured log of all intermediate artifacts. Each entry includes:
- Artifact type: model draft, critique, revision, human checkpoint.
- Timestamp: when the artifact was created.
- Agent ID: which agent produced it.
- Parent ID: which artifact this one responds to.
- Content: the actual text or structured data.
- Status: pending, approved, rejected, escalated.
Agents query the workspace to find the most recent draft, the most recent critique, or the most recent human decision. The workspace does not enforce a strict DAG. Agents can loop back to earlier stages if a gate recommends it.
Specialized Gates
Gates are lightweight diagnostic agents. They do not generate new content. They inspect existing artifacts and decide whether to proceed, loop back, or escalate. Each gate targets a specific failure mode:
| Gate Type | Failure Mode | Action |
|---|---|---|
| Reality Check | False market-structure premise | Loop back to generation with critique |
| Proof Review | Logical gap in welfare claim | Loop back to revision with specific gap |
| Consistency Check | Contradictory assumptions across sections | Escalate to human for reconciliation |
| Completeness Check | Missing required components | Loop back to generation with checklist |
| Human Checkpoint | High-cost decision (e.g., publication) | Escalate to human for approval |
Gates do not certify correctness. They flag likely problems. A reality check might reject a model that assumes perfect competition in a market with network effects. A proof review might catch a missing step in a welfare derivation. A consistency check might notice that two sections assume different utility functions.
Critique Routing Protocol
When a generation agent produces a draft, the system routes it through a sequence of gates. Each gate can:
- Approve: pass the artifact to the next stage.
- Loop back: send the artifact back to generation or revision with a critique.
- Escalate: send the artifact to a human checkpoint.
The routing protocol is deterministic but configurable. You can add gates, remove gates, or change the order. The key constraint is that gates do not block indefinitely. If a gate loops back more than N times (default N=3), it escalates to human review.
Here is a simplified routing flow:
def route_artifact(artifact, workspace, config):
gates = config["gates"]
max_loops = config["max_loops"]
for gate in gates:
result = gate.inspect(artifact, workspace)
if result.action == "approve":
continue
if result.action == "loop_back":
loop_count = workspace.count_loops(artifact.id, gate.name)
if loop_count >= max_loops:
return escalate_to_human(artifact, gate, result.reason)
else:
return send_to_revision(artifact, result.critique)
if result.action == "escalate":
return escalate_to_human(artifact, gate, result.reason)
return mark_approved(artifact)The loop counter prevents infinite critique cycles. If a gate keeps rejecting the same artifact, the system assumes the problem requires human judgment.
Human Checkpoints
Human checkpoints are not optional. They are part of the architecture. The system escalates to humans when:
- A gate loops back more than N times.
- A consistency check finds contradictory assumptions.
- A completeness check finds missing required components.
- The artifact reaches a high-cost decision point (e.g., publication).
Humans do not review every artifact. They review the ones that gates cannot resolve. The workspace shows them the full history: the original draft, the critiques, the revisions, and the gate decisions. They can approve, reject, or send back with additional guidance.
State Management Across Cycles
State management is the hard part. Agents generate drafts, gates critique them, agents revise them, and humans intervene at arbitrary points. You need to track:
- Which version of the draft is current.
- Which critiques have been addressed.
- Which gates have approved the current version.
- Which human decisions are still pending.
The workspace handles this with a versioned artifact model. Each artifact has a version number. When an agent revises a draft, it creates a new artifact with an incremented version number and a parent pointer to the previous version. Gates inspect the current version. Humans see the full version history.
Here is the state schema:
class Artifact:
id: str
version: int
parent_id: Optional[str]
artifact_type: str # "draft", "critique", "revision", "decision"
agent_id: str
timestamp: datetime
content: dict
status: str # "pending", "approved", "rejected", "escalated"
gate_approvals: List[str] # which gates have approved this version
loop_count: dict # {gate_name: count}When a gate approves an artifact, it appends its name to gate_approvals. When a gate loops back, it increments loop_count[gate.name]. When a human approves, it sets status = "approved". When a human rejects, it sets status = "rejected" and optionally creates a new critique artifact.
Observability Primitives
You need to know which decisions were human-approved versus agent-approved. The workspace logs every state transition:
- Agent actions: generation, critique, revision.
- Gate decisions: approve, loop back, escalate.
- Human decisions: approve, reject, send back.
Each log entry includes:
- Timestamp: when the action occurred.
- Actor: which agent, gate, or human made the decision.
- Artifact ID: which artifact was affected.
- Action: what happened.
- Reason: why it happened (for gates and humans).
You can query the log to answer questions like:
- How many artifacts reached human review?
- Which gates triggered the most loopbacks?
- How many loops occurred before escalation?
- Which artifacts were approved without human review?
This is critical for auditing. If a published model turns out to be wrong, you can trace back through the log to see which gates approved it and which human checkpoints it passed.
Evaluation Results
The paper evaluated the architecture on five economic theory tasks. Two evaluators blinded to configuration ranked the gated version against an ungated baseline. They agreed on all five rankings:
- Task 1 (market equilibrium): Gated version preferred. Reality check rejected a false perfect-competition assumption.
- Task 2 (welfare theorem): Gated version preferred. Proof review caught a missing step in the welfare derivation.
- Task 3 (mechanism design): Baseline preferred. Gated version compressed an economically important mechanism too aggressively.
- Task 4 (game theory): Gated version preferred. Consistency check flagged contradictory utility functions.
- Task 5 (policy analysis): Gated version preferred. Completeness check caught a missing distributional impact section.
Mean failure severity fell from 1.58 to 1.16. Overall usefulness rose from 2.60 to 3.10. The largest gain came from catching false premises and logical gaps. The largest loss came from over-aggressive compression.
Failure Modes
The architecture does not eliminate failures. It makes them auditable. Here are the observed failure modes:
Over-Aggressive Compression
In Task 3, the gated version compressed a mechanism description to save tokens. The compression removed an economically important detail about timing. The baseline version kept the detail. Evaluators preferred the baseline.
This is a gate calibration problem. The completeness check should have flagged the missing detail, but it did not. The fix is to add a domain-specific check for mechanism timing.
False Negatives
Gates can miss problems. A reality check might approve a model with a subtle false premise. A proof review might miss a logical gap that requires domain expertise. The architecture does not solve this. It just routes the artifact to the next stage.
The mitigation is to tune gate sensitivity. You can make gates more conservative (flag more potential problems) at the cost of more human review. Or you can make them more permissive (flag fewer problems) at the cost of more false negatives.
Infinite Loops
Even with loop counters, you can get stuck. If a gate keeps rejecting an artifact and the revision agent keeps making the same mistake, you hit the loop limit and escalate. But the human might send it back with guidance that the revision agent still cannot follow.
The mitigation is to track human guidance separately. If a human sends an artifact back more than once, escalate to a different human or a synchronous review session.
When to Use This Architecture
Use gated human-in-the-loop when:
- No ground truth exists: You cannot write a test or validator for correctness.
- Mistakes are costly: A bad output has real-world consequences (financial loss, policy error, reputational damage).
- Human review is expensive: You cannot afford to review every intermediate step.
- Auditability matters: You need to trace decisions back to specific agents, gates, or humans.
Do not use it when:
- Ground truth is cheap: You can write tests, run validators, or check against known-good outputs.
- Mistakes are cheap: A bad output is easy to catch and fix downstream.
- Human review is cheap: You can afford to review everything manually.
- Latency is critical: Gates and human checkpoints add round trips.
Technical Verdict
pAI-Econ-claude shows that you can build reliable multi-agent systems without ground truth. The key is to separate generation, critique, and human judgment into distinct stages with explicit routing rules. Gates do not certify correctness. They flag likely failures and route them to the right reviewer.
The architecture works best when failure modes are predictable and gate logic is tunable. It works worst when failures are subtle and require deep domain expertise. The evaluation shows a clear improvement in auditability and a modest improvement in output quality.
If you are building agents for economic modeling, policy analysis, or other domains without machine-readable correctness signals, this is a practical starting point. The shared workspace and gate abstraction are reusable. The loop counters and escalation rules prevent runaway critique cycles. The observability primitives make mistakes traceable.
The code is public. The evaluation is reproducible. The failure modes are documented. This is infrastructure you can build on.