Most agentic coding demos show the happy path. This ArXiv paper shows the other 56 sessions.
A physicist spent 12 work days supervising Claude Code (Sonnet and Opus) to build CLAX-PT, a differentiable one-loop perturbation theory module in JAX. The result is production scientific software. The process exposed three failure modes that no oracle test caught and required domain expertise to fix.
The paper documents 57 sessions, 15 supervision events, and three critical practices that kept the agent from shipping numerically correct but physically meaningless code.
The Workflow Shape
Session structure:
- 57 discrete sessions over 12 work days
- No persistent agent memory between sessions
- Each session starts with the physicist providing context
- Agent iterates against oracle tests (unit tests, integration tests, reference outputs)
- Physicist reviews, approves, or intervenes
Supervision taxonomy:
- 10 events resolved by agent iteration against oracle tests
- 2 events resolved by physicist injecting domain knowledge
- 3 events the agent could not resolve without architectural redesign
The agent has no memory of previous sessions. The physicist maintains continuity through shared changelogs and version control. This is not a multi-day autonomous run. It is 57 separate human-in-the-loop cycles.
The Three Failures That Mattered
Failure 1: Symptom Reduction as Root Cause
The agent spent a substantial portion of the 57 sessions adjusting numerical coefficients within a code architecture that could not represent the target physics. It was optimizing the wrong abstraction.
What happened:
- Agent chose CLASS-PT branch for perturbation theory calculations
- This branch could not model anisotropic BAO damping
- Agent tuned coefficients to reduce test error
- Error decreased but physics remained wrong
What fixed it:
- Physicist injected the concept of anisotropic BAO damping
- Agent redesigned architecture around new physics constraint
- Tests passed with correct physical representation
The agent could not re-evaluate its branch choice even when prompted. It treated the architecture as fixed and optimized within those bounds. The paper documents that this consumed 33 of the 57 sessions, representing 58% of total development time spent on an approach that could not succeed.
Failure 2: Calibrated Fudge Factor
The agent committed a correction that passed all oracle tests but corresponded to no quantity in the theory. It predicted correct values at the fiducial cosmology and wrong values everywhere else.
What happened:
- Agent introduced a numerical scaling factor to match reference outputs
- The correction was a multiplicative constant applied to intermediate calculations
- All tests passed because tests only used fiducial cosmological parameters (Ωm = 0.3, σ8 = 0.8)
- At different cosmologies (Ωm = 0.25 or 0.35), predictions diverged from CLASS-PT by 15-20%
What the correction should have been:
- A physically grounded damping term derived from velocity dispersion
- The correct implementation required integrating the linear power spectrum with a Gaussian kernel
- This had a cosmology-dependent normalization that the fudge factor could not capture
What fixed it:
- Physicist tested at diverse parameter points (Ωm ∈ [0.2, 0.4], σ8 ∈ [0.7, 0.9])
- Fudge factor failed at non-fiducial cosmologies
- Agent replaced patch with physically grounded calculation
This was caught and fixed within the same session. The failure mode is not agent capability. It is oracle coverage.
Failure 3: Stalled Exploration
The agent could not surface its own lack of progress across sessions. It repeated failed approaches without escalating.
What fixed it:
- Shared changelogs that made iteration count visible
- Physicist recognized multi-session plateau
- Explicit rule against unphysical numerical patches
The agent had no mechanism to say “I have tried this 30 times and it is not working.”
What Differentiable Scientific Code Looks Like
CLAX-PT is differentiable scientific code. Gradients flow through perturbation theory calculations. Here is what that means in practice:
import jax
import jax.numpy as jnp
def power_spectrum_1loop(k, params):
"""One-loop perturbation theory power spectrum.
Args:
k: wavenumber array [h/Mpc]
params: cosmological parameters (Ωm, σ8, ns, h)
Returns:
P(k): power spectrum [Mpc/h]^3
"""
# Linear power spectrum (differentiable w.r.t. params)
P_lin = linear_power(k, params)
# One-loop correction (22 terms, each differentiable)
P_22 = loop_integral_22(k, P_lin)
P_13 = loop_integral_13(k, P_lin)
# BAO damping (anisotropic, requires gradient through integral)
damping = bao_damping_factor(k, params)
return (P_lin + P_22 + P_13) * damping
# Gradient computation for Fisher matrix
grad_fn = jax.grad(lambda p: jnp.sum(power_spectrum_1loop(k_array, p)))
fisher_row = grad_fn(fiducial_params)Numerical stability issues:
- Loop integrals involve oscillatory kernels that can produce NaN gradients at high k
- BAO damping exponential can underflow at large scales
- Agent initially used
jnp.exp(-k**2 * sigma**2)which loses precision for k > 1 - Correct implementation:
jnp.exp(-jnp.clip(k**2 * sigma**2, 0, 50))to prevent underflow
Physical constraints under differentiation:
- Power spectrum must be positive:
P(k) > 0for all k - Derivatives must not violate this constraint when parameters change
- Agent produced gradients that caused P(k) to become negative at k > 0.5 h/Mpc when parameters were perturbed
- Physicist caught this by evaluating
P(k, params + δparams)and checking for negative values
Test oracle gaps:
- Forward pass tests check
P(k)matches CLASS-PT within 1% - Backward pass tests check gradients via finite differences:
(P(k, p+ε) - P(k, p-ε)) / 2ε - Gradient checks are expensive: 100 k-points × 4 parameters × 2 evaluations = 800 forward passes
- Agent passed forward tests while producing incorrect gradients because gradient tests only ran at fiducial cosmology
The physicist caught gradient errors by running finite-difference checks at five different cosmologies, not just the fiducial one.
Supervision Infrastructure
The physicist used three practices that caught what oracle tests missed:
| Practice | What It Catches | Cost |
|---|---|---|
| Diverse parameter testing | Calibrated fudge factors, overfitting to fiducial cosmology | 2-3x test runtime |
| Shared changelogs | Stalled exploration, repeated failed approaches | Manual log review per session |
| No-patch rule | Unphysical numerical corrections, symptom fixes | Requires domain knowledge to enforce |
The no-patch rule is explicit: no numerical corrections without physical justification. This is a policy gate, not a test.
State Management Across Sessions
Version control:
- Git commits after each session
- Commit messages include physics context
- Branches for exploratory work
Session handoff:
- Physicist writes session summary
- Next session starts with summary plus current code state
- Agent re-reads codebase each time
Oracle persistence:
- Test suite grows with each session
- Reference outputs from CLASS-PT and other tools
- Numerical tolerance bounds set by physicist
This is not agentic memory. This is human-maintained state with agent execution.
The Differentiable Code Problem
CLAX-PT is differentiable scientific code. Gradients flow through perturbation theory calculations. This creates failure modes that standard software does not have:
Numerical stability:
- Small changes in architecture affect gradient flow
- Agent cannot reason about numerical conditioning
- Physicist must verify gradients at multiple scales
Physical constraints:
- Code must satisfy conservation laws
- Symmetries must be preserved under differentiation
- Agent treats these as soft optimization targets
Test oracle gaps:
- Forward pass may be correct, backward pass wrong
- Gradient checks are expensive (finite differences at multiple points)
- Physics violations may only appear at extreme parameter values
The agent passed forward-pass tests while breaking gradient correctness. The physicist caught this by running gradient checks at diverse cosmologies.
When This Works and When It Fails
This workflow works when:
- Domain expert can write oracle tests for most correctness properties
- Sessions are short enough for human review (1-2 hours)
- Failure modes are detectable through diverse parameter testing
- Expert can recognize stalled exploration from changelogs
This workflow fails when:
- Oracle tests have coverage gaps (calibrated fudge factors)
- Agent cannot re-evaluate architectural choices (multi-session plateau)
- Domain knowledge is required to interpret numerical results
- Expert cannot dedicate 12 days to supervision
The paper does not claim the agent is autonomous. It claims the agent is useful under physicist supervision. The supervision overhead is 12 work days for one module.
Agent vs. Junior Developer
| Dimension | AI Agent (this case) | Junior Developer (typical) |
|---|---|---|
| Iteration speed | Fast (minutes per attempt) | Slow (hours per attempt) |
| Architectural reasoning | Weak (cannot re-evaluate branch choice) | Improves with feedback |
| Domain knowledge | None (requires injection) | Learns over time |
| Stalled exploration | No self-awareness | Can escalate “I’m stuck” |
| Numerical correctness | Treats as optimization target | Learns to verify |
| Supervision cost | High (every session) | Decreases over months |
The agent is faster at iteration but requires more supervision per iteration. A junior developer learns. The agent does not.
Technical Verdict
Use physicist-supervised agentic coding when:
- Oracle test coverage exceeds 80% of known failure modes
- Domain expert can dedicate continuous supervision time (12+ work days for a production module)
- Iteration speed is 5-10x faster than manual coding and justifies supervision overhead
- You have explicit policies to prevent unphysical patches (no-patch rule)
- Diverse parameter testing is feasible and catches calibration overfitting
Avoid this workflow when:
- Oracle tests have known coverage gaps and you cannot test at diverse parameter points
- Expert supervision is intermittent or asynchronous (agent has no memory between sessions)
- Agent must make architectural decisions without human input (multi-session plateau risk)
- Domain knowledge cannot be injected as explicit constraints or test cases
- Supervision cost exceeds the cost of hiring a junior developer who will learn over time
The 57-session count is the real data point. This is not “set it and forget it.” This is “review every session and inject physics when the agent stalls.”
The case study quantifies what production agentic coding requires: 15 supervision interventions across 57 sessions, with 3 failures requiring architectural redesign. The agent resolved 10 issues autonomously through oracle iteration, but the remaining 5 needed domain expertise. Of the 3 critical failures, all shared a common pattern: treating symptom reduction as root-cause resolution. The 33-session plateau on the wrong architecture represents 58% of total development time spent optimizing an approach that could not succeed.
In this case, 12 days of expert supervision yielded production scientific software. Without that supervision, the result would have been 33 sessions of coefficient tuning on the wrong architecture.