Your Agent Says Done. The Cron Tab Disagrees: Why Silent Failures Are the Default in Unattended Agent Runs

You wake up to a wall of “completed” notifications from the jobs that ran overnight. The output file is empty. Nobody notices until someone opens the log.

A 1,000-run empirical study on cron-scheduled agents with injected failures shows a 100% silent-failure rate when agents receive empty input. The agent reports success. The cron tab sees exit code 0. The artifact never materializes.

This is not a bug in one framework. It is the default behavior when you remove the human from the loop and ask an LLM to self-report task completion.

The Completion Signal vs. Actual Success Gap

When an agent says “done,” what does that mean?

LLM completion signal: The model finished generating tokens and returned a structured response.
Tool execution acknowledgment: The function call returned without throwing an exception.
Hallucinated status: The model inferred success from context, not from checking state.

In attended workflows (Slack bots, chat interfaces, human-in-the-loop pipelines), a human spots the empty response and retries. In unattended workflows (cron jobs, scheduled ETL, overnight data pulls), the agent runs, reports success, and exits. The orchestrator sees exit code 0 and moves on.

The gap appears when the agent’s internal state machine conflates “I finished my loop” with “I produced the artifact you asked for.”

The 1,000-Run Fleet Study

The study tracked 1,000 cron executions on a single WSL2 laptop running scheduled agent tasks. Each run logged:

Status (success, failure, partial)
Error class (network timeout, empty input, schema mismatch, tool crash)
Duration
Schedule lateness

Key finding: Agents were honest about loud failures (network timeouts, tool crashes, schema validation errors). On silent failures (empty input, missing data source, no-op tool calls), they reported success 8 out of 8 times.

Failure Taxonomy

Failure Class	Agent Self-Report	Actual Outcome	Detection Method
Network timeout	Failure	Failure	Exception propagated
Tool crash	Failure	Failure	Stack trace logged
Schema mismatch	Failure	Failure	Validation error raised
Empty input	Success	Failure	Output file size = 0
Missing data source	Success	Failure	No rows written
No-op tool call	Success	Failure	State unchanged

The pattern: if the agent loop completes without throwing an exception, the orchestrator sees success. If the artifact never materializes, nobody notices until a downstream consumer fails.

The Five-Line Validation Check

The study introduced a post-run validation step that caught all silent failures:

def validate_agent_output(task_id, expected_artifact_path):
    if not os.path.exists(expected_artifact_path):
        raise ArtifactMissingError(f"Task {task_id}: no output file")
    
    if os.path.getsize(expected_artifact_path) == 0:
        raise EmptyArtifactError(f"Task {task_id}: zero-byte output")
    
    # Optional: schema or row-count check
    with open(expected_artifact_path) as f:
        if len(f.readlines()) < 1:
            raise NoDataError(f"Task {task_id}: no rows written")

This check runs after the agent reports success but before the orchestrator marks the job complete. It converts silent failures into loud failures that trigger retries or alerts.

Observability Primitives for Scheduled Agents

When you remove the human from the loop, you need instrumentation that answers three questions:

Did the agent finish its loop? (Exit code, completion signal)
Did the agent produce the expected artifact? (File existence, size, schema)
Did the agent change the world in the expected way? (State snapshot, diff, side-effect log)

Exit Codes Are Not Enough

Cron and systemd timers rely on exit codes. An agent that completes its loop without throwing an exception returns 0. The scheduler sees success. The artifact never appears.

Exit codes tell you the process terminated cleanly. They do not tell you the task succeeded.

Structured Logs Are Not Enough

Agents log tool calls, LLM responses, and intermediate state. But logs are append-only. If the agent hallucinates success and writes “Task completed successfully” to the log, the orchestrator has no way to distinguish that from actual success without parsing the log and checking external state.

Logs are useful for debugging after the fact. They are not a substitute for state validation.

State Snapshots Are the Missing Primitive

The most reliable signal is a state snapshot taken after the agent reports success:

File-based tasks: Check file existence, size, and schema.
Database tasks: Count rows written, check for expected columns.
API tasks: Query the remote system to confirm the change landed.

This is not a health check. It is a task-specific assertion that the world changed in the expected way.

Architecture: Validation Layer in the Orchestrator

Here is how the validation layer fits into a cron-scheduled agent pipeline:

┌─────────────────────────────────────────────────┐
│  Cron Trigger (every 6 hours)                   │
└────────────────┬────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────┐
│  Agent Executor                                 │
│  - Load task config                             │
│  - Call LLM with tools                          │
│  - Execute tool calls                           │
│  - Return structured status                     │
└────────────────┬────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────┐
│  Validation Layer (runs before exit)           │
│  - Check artifact exists                        │
│  - Validate schema / size / row count           │
│  - Raise exception if assertion fails           │
└────────────────┬────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────┐
│  Orchestrator (marks job complete or retries)   │
└─────────────────────────────────────────────────┘

The validation layer runs in the same process as the agent. If validation fails, the process exits with a non-zero code. The orchestrator sees the failure and triggers a retry or alert.

Injected Failure Grid Methodology

The study used a controlled injection grid to measure false-success rates:

Baseline run: Agent extracts data from a known-good source.
Empty input: Source file exists but contains zero rows.
Missing source: Source file does not exist.
Network partition: Source is unreachable (simulated with firewall rule).
Tool no-op: Tool returns success but writes nothing.

Each failure class ran 8 times. The agent’s self-reported status was compared to a rule-based verifier that checked file size, row count, and state diff.

Result: Loud failures (network partition, missing source with exception) were reported correctly. Silent failures (empty input, tool no-op) were reported as success 100% of the time.

Likely Failure Modes in Production

When you deploy unattended agents, expect these failure modes:

Empty input goes unnoticed: The agent processes zero rows and reports success.
Partial writes look like success: The agent writes 10 rows instead of 1,000 and exits cleanly.
Retry loops exhaust without alerting: The orchestrator retries 3 times, sees exit code 0 each time, and gives up.
Downstream consumers fail silently: The next job in the pipeline expects 1,000 rows, gets 0, and crashes without logging why.

All of these are silent until a human checks the output or a downstream system breaks.

Security Boundaries: When Validation Becomes a Backdoor

Adding a validation layer introduces a new attack surface. If the validator runs arbitrary code based on task config, an attacker who controls the config can inject commands.

Mitigation:

Run the validator in a restricted subprocess with no network access.
Use a schema-based validator (JSON Schema, Pydantic) instead of arbitrary Python.
Log all validation failures to a separate audit log that the agent cannot write to.

The validator should be simpler than the agent. If it is complex enough to need its own validation, you have a recursion problem.

Technical Verdict

Silent failures are the default in unattended agent runs because LLM completion signals do not map to task success. A five-line validation check that asserts on artifact existence and schema catches 100% of silent failures in empirical testing.

The right observability primitive is not logs or exit codes. It is a state snapshot taken after the agent reports success and compared to task-specific assertions. This validation layer runs in the orchestrator, not in the agent, and converts silent failures into loud failures that trigger retries or alerts.

Use This Pattern If

Your agents run unattended on a schedule (cron, systemd timers, cloud schedulers like AWS EventBridge or GCP Cloud Scheduler). No human reviews output before downstream systems consume it.
Task success depends on external state changes (files written to disk, rows inserted into a database, API calls that mutate remote resources). The agent’s self-report is not sufficient proof of success.
Downstream systems depend on the artifact (ETL pipelines, data syncs, report generation, model training jobs). A missing or malformed artifact causes cascading failures.
You need deterministic retry logic. Silent failures that report success exhaust retry budgets without triggering alerts. Validation converts them into retriable errors.

Skip This Pattern If

A human reviews every output before it matters. Chat-based agents, Slack bots, or interactive workflows where a user can spot empty responses and retry manually.
The task is idempotent and safe to retry blindly. If running the agent twice produces the same result and causes no harm, you can rely on exit codes and retry on timeout.
You are still prototyping and figuring out what success means. Adding validation before you have stable task definitions creates churn. Start with logs and manual inspection, then add validation when the task shape stabilizes.
The agent writes to append-only logs or metrics systems. If the worst-case failure is a missing log entry (not a missing artifact that breaks a pipeline), the cost of validation may exceed the risk.

Cost-Benefit Trade-Off

Validation overhead is negligible (file existence and size checks take microseconds, row-count queries take milliseconds). The cost of silent failures in production is high: debugging time, downstream system breakage, data quality incidents, and lost trust in automation.

If you are moving agents from prototype to production and scheduling them to run unattended, add the validation layer before you add the cron job. The alternative is waking up to a wall of “completed” notifications and an empty output file.

Source Links

Primary article on dev.to
silent-failure-watch repository (1,000-run dataset and injection grid)

Agentic AI, minus the noise.