Computer-using agents need verification infrastructure. When an agent claims it booked a flight, updated a spreadsheet, or configured a mobile app, something must confirm the work happened. Static VLM judgment of trajectory screenshots breaks down without ground truth. Generated inspection programs fail when application state is inaccessible or incomplete. VeriCUA proposes verification as agentic exploration: a second agent that actively probes the environment to gather evidence and assess task completion.
This matters for three production use cases: data curation (filtering bad agent trajectories), evaluation (measuring agent performance), and reinforcement learning (generating reward signals). All three require automated verification at scale across desktop, mobile, and web platforms.
The Verification Gap
Typical verification approaches rely on partial views:
VLM judges receive a sequence of screenshots and a task description. They predict success or failure based on visual evidence in the supplied observations. If the decisive state change happened off-screen or between frames, the judge cannot see it.
Generated inspection programs query application state through APIs or accessibility trees. They work when the target state is exposed and queryable. They fail when applications hide internal state, when APIs lack coverage, or when the verification logic requires visual confirmation.
Both approaches are platform-specific. A verifier tuned for desktop GUI inspection does not transfer to mobile apps or web environments without re-engineering.
Verification as Exploration
VeriCUA treats verification as a multi-step agent task. The verifier receives the original task description and available context (screenshots, accessibility trees, or both). It then:
- Constructs a task rubric by decomposing the task into verifiable requirements
- Selects checks (GUI exploration or code-based state inspection) for unresolved requirements
- Executes checks by taking actions in the environment or running inspection code
- Combines findings under explicit evidence rules to produce a verification judgment
The verifier maintains state across multiple turns. It can click through UI elements, navigate to different screens, or query APIs to gather evidence. It stops when all requirements are resolved or a budget is exhausted.
Offline vs Online Verification
Offline verification runs after task execution completes. The verifier inspects a static environment snapshot (final screenshots, final accessibility tree). It cannot take actions that modify state. This mode suits data curation and batch evaluation.
Online verification runs during or immediately after task execution in a live environment. The verifier can interact with the application to confirm state changes. This mode suits RL training loops where verification must happen before the environment resets.
Architecture Components
| Component | Responsibility | Failure Mode |
|---|---|---|
| Rubric constructor | Parse task into verifiable requirements | Misses implicit requirements, over-specifies trivial details |
| Check selector | Choose GUI exploration vs code inspection | Selects inaccessible checks, wastes budget on redundant checks |
| GUI explorer | Navigate UI to gather visual evidence | Gets stuck in navigation loops, misinterprets UI state |
| Code inspector | Generate and execute state queries | Generates invalid code, queries unavailable APIs |
| Evidence combiner | Aggregate findings under explicit rules | Conflicting evidence from GUI and code, partial success ambiguity |
The rubric constructor runs once at the start. The check selector, explorer, and inspector run in a loop until requirements are satisfied or the budget expires. The evidence combiner produces the final judgment.
State Management
The verifier tracks:
- Resolved requirements: which rubric items have supporting evidence
- Pending checks: which GUI or code checks are queued
- Exploration history: which UI states have been visited to avoid loops
- Evidence log: findings from each check with confidence scores
This state persists across verification turns. The verifier can revisit earlier findings if new evidence contradicts them.
Isolation Boundaries
Verification agents must not interfere with the task environment. In offline mode, this is straightforward: the environment is read-only. In online mode, isolation requires:
Separate sessions: The verifier runs in a cloned environment or a separate user session. Changes made during verification do not affect the original task state.
Rollback capability: If the verifier must interact with the live environment, it logs all actions and rolls them back after verification completes.
Observation-only fallback: When isolation is impossible, the verifier restricts itself to read-only operations (screenshot analysis, accessibility tree inspection without interaction).
The paper does not specify isolation mechanisms in detail. Production deployments will need explicit session management, environment cloning, or sandboxing depending on the platform.
Integration with RL Training
Verification agents fit into RL pipelines as reward signal generators. Two integration patterns:
Synchronous verification: The verifier runs immediately after each episode. The RL agent waits for verification before starting the next episode. This adds latency but ensures reward accuracy.
Asynchronous verification: The verifier runs in parallel with ongoing episodes. Rewards arrive delayed and are matched to episodes by ID. This maintains throughput but complicates credit assignment if verification takes multiple episodes to complete.
Separate eval stage: Verification runs offline on saved trajectories. The RL agent trains on a proxy reward (heuristic or model-based), and verification filters the dataset periodically. This decouples training speed from verification cost but introduces staleness.
The paper focuses on offline and online verification as distinct modes, not RL integration specifics. Practitioners will need to choose a pattern based on verification latency and training throughput requirements.
Failure Modes
Ambiguous UI states: The verifier sees a loading spinner or intermediate state and cannot determine if the task succeeded. It must decide whether to wait, explore further, or mark the task as indeterminate.
Non-deterministic outcomes: The task produces different valid results on each run (e.g., “find a cheap flight” with multiple acceptable options). The verifier must recognize equivalence classes rather than exact matches.
Partial success: The agent completed 80% of a multi-step task. The verifier must decide whether to reward partial progress or treat it as failure. This depends on the RL training objective.
Verification budget exhaustion: The verifier runs out of action budget before resolving all requirements. It must produce a judgment based on incomplete evidence, which introduces false negatives.
Conflicting evidence: GUI exploration suggests success, but code inspection reveals missing state. The evidence combiner must have tie-breaking rules or confidence thresholds.
Example: Desktop Spreadsheet Task
Task: “Add a new row to the sales spreadsheet with today’s revenue.”
Rubric:
- Spreadsheet application is open
- A new row exists at the bottom
- The new row contains today’s date
- The new row contains a revenue value
Verification flow:
- Check accessibility tree for spreadsheet application (requirement 1: resolved)
- Generate code to count rows before and after (requirement 2: resolved if count increased)
- Take screenshot of the last row (requirements 3 and 4: visual inspection)
- If date or revenue is unclear, click into the cell and inspect the formula bar
The verifier combines findings: if all four requirements are resolved with high confidence, the task succeeds. If any requirement is unresolved or contradicted, the task fails.
Code Snippet: Evidence Combination
class EvidenceCombiner:
def __init__(self, requirements):
self.requirements = requirements
self.evidence = {req: [] for req in requirements}
def add_finding(self, requirement, source, confidence, result):
"""Record evidence from GUI or code check."""
self.evidence[requirement].append({
'source': source, # 'gui' or 'code'
'confidence': confidence,
'result': result # True, False, or 'indeterminate'
})
def judge(self):
"""Produce final verification judgment."""
for req, findings in self.evidence.items():
if not findings:
return 'indeterminate' # Missing evidence
# Aggregate by source type
gui_votes = [f for f in findings if f['source'] == 'gui']
code_votes = [f for f in findings if f['source'] == 'code']
# Code evidence overrides GUI if confidence is high
if code_votes and max(f['confidence'] for f in code_votes) > 0.9:
if any(f['result'] == False for f in code_votes):
return 'failure'
# GUI evidence used when code is unavailable
if gui_votes and not code_votes:
if any(f['result'] == False for f in gui_votes):
return 'failure'
return 'success'This is a simplified combiner. Production versions need weighted voting, confidence thresholds per requirement type, and handling for conflicting evidence.
Trade-offs
| Approach | Strength | Weakness |
|---|---|---|
| Static VLM judgment | Fast, no environment interaction | Misses off-screen state, no exploration |
| Generated inspection code | Precise state queries | Requires API access, platform-specific |
| Agentic exploration | Adapts to available evidence | Higher latency, budget management complexity |
Agentic exploration adds verification latency. A static VLM judge runs in one forward pass. A verification agent may take 5-10 turns to resolve all requirements. This matters for RL training throughput.
Agentic exploration also adds failure surface. The verifier can get stuck in navigation loops, generate invalid inspection code, or misinterpret ambiguous UI states. Static judges fail predictably; agentic verifiers fail in more varied ways.
The benefit is coverage. Agentic exploration can verify tasks where static judges and generated programs both fail because it combines multiple evidence sources and adapts its strategy based on what it finds.
Technical Verdict
Use VeriCUA-style verification when:
- You need cross-platform verification (desktop, mobile, web) without per-platform engineering
- Task completion evidence is scattered (some in UI, some in application state)
- You are curating agent trajectories at scale and can tolerate verification latency
- False positives (claiming success when the task failed) are more costly than false negatives
Avoid it when:
- Verification must run in real-time (sub-second latency requirements)
- The task has a single, easily queryable success condition (use generated inspection code)
- You lack environment isolation and cannot risk verification actions interfering with task state
- Your RL training loop is bottlenecked on episode throughput and cannot absorb verification latency
For production RL pipelines, consider hybrid approaches: use fast heuristic rewards during training and run agentic verification offline to filter the dataset periodically. This balances training speed with data quality.