AJX: Agent Journey Experience Testing for Product Usability

Your next user is an agent. AJX (Agent Journey Experience) is a CLI and skill bundle that tests what your product feels like from an agent’s point of view. It runs an agent against your CLI, SDK, docs, or API, then returns a structured list of product improvements: missing JSON flags, conflicting documentation, unhelpful error messages, and token-wasting detours.

This is not an eval framework. It does not measure model capability. It measures product-agent fit by instrumenting the agent’s journey and extracting behavioral failures into a backlog.

Architecture: Outer Loop Controls Inner Loop

AJX uses a two-loop design:

• Outer loop: The AJX harness skill and CLI. It orchestrates the test, configures the environment, and analyzes the run after completion.
• Inner loop: The agent under test (Claude, Codex, kiro-cli, or any harness you configure). It attempts the task using your product.

The outer loop does not pollute the inner loop’s execution context. It starts the inner agent in a clean environment (Docker, VM, or local shell), provides the prompt and definition of done, then waits. Once the inner loop finishes, the outer loop examines logs, conversation traces, and outputs to extract metrics and asks.

Configuration Shape

You configure AJX with plain text or a config file:

• Harness: Which agent framework to test (claude, codex, kiro-cli).
• Environment: Docker, VM, or local. Optionally specify skills, model, and credentials.
• Prompt: The task the agent should complete.
• Definition of done: Machine-readable success criteria (e.g., “CSV file exists with 100 rows” or “deployment returns 200 status”).
• Runs: Number of trials to average behavior.

The outer loop validates the definition of done by checking outputs, exit codes, or file artifacts. Partial completion counts as failure.

What AJX Measures

AJX tracks four dimensions of agent friction:

1. Retries: How many attempts before success.
2. Tokens: Total token spend, including wasted context from rephrasing or reparsing.
3. Time: Wall-clock duration.
4. Behavioral failures: Rageprompting (repeated similar prompts), wrong turns, corrections, and assumption errors.

It outputs a detailed report and a list of asks: specific product changes that would reduce friction. Examples from real runs:

• CLI missing --json flag, forcing the agent to parse human-readable output.
• Docs disagree with actual CLI behavior.
• Skills pull the agent toward deprecated patterns.
• Error messages lack actionable next steps.
• Long-running commands provide no progress feedback, causing the agent to retry prematurely.

Extracting Asks Without Manual Tagging

AJX does not require you to annotate logs. It uses the outer loop’s analysis phase to:

1. Parse conversation traces for repeated phrases, corrections, and retries.
2. Diff expected vs. actual outputs to find mismatches.
3. Correlate errors with documentation links the agent followed.
4. Identify tool calls that failed or returned unparseable data.

The outer loop agent (not the inner loop) generates the asks by reasoning over the trace. It looks for patterns like:

• Agent tries a command, gets an error, rephrases the same command three times.
• Agent reads docs that say feature X exists, tries it, gets “unknown flag.”
• Agent spends 50% of tokens parsing unstructured output.

This produces a structured list of asks with severity, category (docs, CLI, error messages), and suggested fix.

CI/CD Integration

AJX envisions a loop-until-green workflow:

1. Run AJX against your product with a set of canonical tasks.
2. If AJX returns asks, prioritize and fix them.
3. Re-run AJX until it returns “all green” with no asks.

You can track a usability score over time (tokens per task, retry rate) and set thresholds in CI. For example, block merges if a new feature increases retry rate by more than 20% or if token spend exceeds a budget.

The CLI supports ajx status to check the last run and ajx init --task to configure a new test. The author tested AJX with AJX and found bugs in both commands (status returned wrong codes, init behaved differently than expected).

Trade-offs and Failure Modes

Dimension	Trade-off	Risk
Outer loop complexity	Adds orchestration overhead and requires agent-to-agent instrumentation.	Outer loop bugs can mask inner loop behavior.
Definition of done	Requires explicit success criteria. Vague criteria produce false positives.	Teams may write overly permissive checks to avoid flakiness.
Harness diversity	Supports multiple agent frameworks, but each needs integration code.	New harnesses require custom adapters.
Ask extraction	Relies on outer loop reasoning. Quality depends on outer loop model capability.	Weak outer loop may miss nuanced friction or hallucinate asks.
Cost	Running agents repeatedly is expensive. Each test consumes tokens in both loops.	Budget constraints limit test coverage.

Failure modes:

• Flaky tests: If the inner loop agent is non-deterministic, retries may vary wildly. Run multiple trials and average.
• Outer loop hallucination: The outer loop may invent asks that do not reflect real friction. Validate asks by re-running the test after fixes.
• Environment drift: If the inner loop environment does not match production, asks may not transfer. Use Docker or VM snapshots for consistency.

Example: Testing a CLI

You want to test the Vercel CLI. You configure AJX with:

• Prompt: “Build and deploy a simple hello world with Vercel CLI.”
• Definition of done: “Deployment URL returns 200 status.”
• Harness: Claude with default skills.
• Environment: Docker with Node.js and Vercel CLI pre-installed.

AJX starts the inner loop. The agent reads the Vercel docs, runs vercel deploy, hits an authentication error, retries with vercel login, then succeeds. The outer loop examines the trace and finds:

• Agent spent 30% of tokens parsing human-readable output from vercel deploy.
• Agent retried vercel login twice because error message did not explain how to pass a token.
• Docs mentioned --json flag but agent did not use it initially.

AJX outputs asks:

1. Add --json flag to all commands for machine-readable output.
2. Update error message for auth failure to include --token example.
3. Add vercel doctor command to check auth status.

You fix these, re-run AJX, and see retry rate drop from 3 to 1 and token spend drop by 25%.

Code Snippet: Outer Loop Analysis

This is conceptual pseudocode showing how the outer loop extracts asks:

def analyze_run(trace, definition_of_done):
    asks = []
    
    # Check for repeated tool calls
    tool_calls = extract_tool_calls(trace)
    for tool, calls in group_by_tool(tool_calls):
        if len(calls) > 2 and all_similar(calls):
            asks.append({
                "severity": "high",
                "category": "cli",
                "issue": f"{tool} called {len(calls)} times with similar args",
                "suggestion": "Add --json flag or improve error message"
            })
    
    # Check for doc/behavior mismatch
    docs_read = extract_docs_urls(trace)
    errors = extract_errors(trace)
    for doc_url in docs_read:
        for error in errors:
            if mentions_feature_from_doc(error, doc_url):
                asks.append({
                    "severity": "critical",
                    "category": "docs",
                    "issue": f"Docs at {doc_url} disagree with behavior",
                    "suggestion": "Update docs or fix feature"
                })
    
    # Check for token waste
    token_spend = calculate_tokens(trace)
    if token_spend > expected_baseline * 1.5:
        asks.append({
            "severity": "medium",
            "category": "ux",
            "issue": f"Token spend {token_spend} exceeds baseline by 50%",
            "suggestion": "Reduce output verbosity or add structured output"
        })
    
    # Validate definition of done
    if not check_success(definition_of_done):
        asks.append({
            "severity": "critical",
            "category": "task",
            "issue": "Task did not meet definition of done",
            "suggestion": "Review task requirements"
        })
    
    return asks

The outer loop runs this analysis after the inner loop completes. It does not instrument the inner loop in real time.

Security Boundaries

AJX runs agents in isolated environments. The inner loop should not have access to the outer loop’s credentials or state. Use Docker or VMs with network policies to prevent the inner loop from exfiltrating data or calling unintended APIs.

The outer loop must authenticate to the inner loop’s environment (to start it and collect logs), but the inner loop should not authenticate to the outer loop. This prevents a compromised inner loop from poisoning the analysis.

Observability

AJX produces:

• Trace logs: Full conversation history from the inner loop.
• Metrics: Retries, tokens, time, error count.
• Asks: Structured list of product improvements.
• Visual journey: Timeline of tool calls, errors, and retries (rendered in the local web console).

You can export these to CSV or JSON for dashboarding. The CLI supports ajx report --format json to integrate with observability tools.

Technical Verdict

Use AJX when:

• You are building a CLI, SDK, or API that agents will use.
• You want concrete metrics on agent usability, not just “it works.”
• You need a backlog of product improvements prioritized by agent friction.
• You can afford the token cost of running agents repeatedly.

Avoid AJX when:

• You are testing model capability (use evals instead).
• You need real-time production observability (use tracing tools instead).
• Your product is not agent-facing (traditional UX testing is more appropriate).
• You cannot define clear success criteria (AJX requires explicit definitions of done).

AJX is early. Expect rough edges in harness integrations and ask extraction. But if you are racing to make your product agent-friendly, AJX operationalizes the question “what does my product feel like to an agent?” and turns the answer into a backlog.

Source Links

• Primary: AJX Website
• GitHub: AJX CLI and Skill
• Discussion: Show HN