Dogfooding Agent Security: Four Bugs in the Security Tool Itself

Agentmetry builds a local-first flight recorder for AI coding agents. It hooks into Cursor and Claude Code at the tool boundary, captures every file write and shell command, then runs sequence detection to flag suspicious patterns: credential access followed by network egress, denied approvals that ran anyway, download cradles piped into shells.

The team finally pointed the tool at their own codebase. It found five bugs. Four were in Agentmetry itself.

This is not a customer success story. This is a post-mortem of what breaks when you build agent security infrastructure and then actually use it on the system that produces it.

The Recorder Was Not Recording

The architecture splits into two pieces:

Hooks fire inside the IDE and POST each tool call to a local orchestrator
Orchestrator writes events to a persistent trail and runs correlation

If the orchestrator is unreachable, the hook writes to a spool file instead of dropping the event. The spool replays on the next orchestrator start. That design is correct. An earlier version silently lost events on every restart.

What was missing: anything to keep the orchestrator running.

The orchestrator went down. The hooks kept working. For five days they wrote 1,880 events (4.6 MB) into a spool that never drained. The dashboard showed a healthy feed. The trail verified. The doctor command reported no problems.

Nothing anywhere said that capture had not reached the trail since Monday.

There was a deadline, too. Spooled events older than seven days are not replayed, because injecting a week-old tool call into today’s correlation window produces false sequences. The oldest entries were five days old at discovery. The team had roughly 45 hours before the flight recorder started deleting its own black box.

Two fixes:

The recorder now installs as a supervised background service that restarts within a minute of dying (which itself had three separate failures)
doctor now fails loudly when the spool is backed up

An empty feed and a stopped recorder used to look identical. That ambiguity is the whole bug.

Draining the Spool Destroyed Events

The drain logic read the spool file, replayed each event to the orchestrator, then truncated the file. The truncate happened in a separate step after the replay loop.

If the orchestrator rejected an event (schema mismatch, duplicate ID, clock skew), the drain loop continued. The truncate still ran. The rejected events were gone.

The spool is append-only. The drain should have been atomic: replay all or replay none. Instead it was best-effort with silent data loss.

The fix: drain now writes a .processed marker only after the orchestrator confirms receipt of every event. If any event fails, the entire spool is retried on the next cycle. The spool file is only truncated after the marker is written and verified.

This is a classic two-phase commit problem dressed up in agent tooling clothes.

Detection Logic Fired on Its Own Build Process

The sequence detector flagged this pattern:

File read: src/detector/rules.ts
Network egress: POST to api.agentmetry.ai
File write: dist/detector/rules.js

The rule was “read source code, exfiltrate it, then cover tracks by writing a compiled artifact.”

What actually happened: the build process read TypeScript, uploaded telemetry (which includes a hash of the source tree), then wrote JavaScript.

The detection logic had no concept of a build step. It saw three tool calls in sequence and fired. The rule was correct. The context was missing.

The fix: detections now include a build-step exemption that checks for:

Parent process is npm, yarn, pnpm, tsc, or esbuild
File write path is inside dist/, build/, or .next/
Network destination is an allowlisted telemetry endpoint

This is a narrow fix. The broader problem is that sequence detection without process tree context will always have a high false-positive rate in development environments.

The Trail Verified, But the Events Were Corrupt

The trail is a cryptographically signed append-only log. Each event includes a hash of the previous event. The verify command walks the chain and confirms integrity.

Verification passed. But when the team replayed the trail to regenerate detections, half the events failed to parse.

The bug: the hook serialized events to JSON before signing them. The orchestrator deserialized, validated the schema, then re-serialized before writing to the trail. The two serialization steps used different JSON encoders with different whitespace and key ordering.

The signature was valid. The schema was valid. The content was subtly different.

The fix: the hook now signs the canonical wire format (protobuf), not the JSON representation. The orchestrator writes exactly what it received. Verification now checks both the signature and the schema.

This is a reminder that cryptographic integrity and semantic integrity are not the same thing.

Architecture: What a Self-Auditing Security Tool Looks Like

Component	Responsibility	Failure Mode
IDE Hook	Capture tool calls at boundary	Silent failure if orchestrator down
Spool File	Buffer events when orchestrator unreachable	Unbounded growth, data loss on drain
Orchestrator	Write to trail, run detections	Process death, no supervision
Trail	Cryptographically signed log	Signature valid but content corrupt
Detector	Sequence analysis over tool calls	False positives without process context
Dashboard	Display feed and detections	Shows stale data when spool is backed up

The core tension: a security tool that runs locally must be resilient to its own failures. If the orchestrator dies, the hooks must keep working. If the spool fills up, the system must alert. If detections fire on the build process, the rules must adapt.

Every layer has a failure mode that looks like success until you audit the actual data flow.

Code: Atomic Spool Drain

async function drainSpool(spoolPath: string, orchestratorUrl: string) {
  const events = await readSpoolFile(spoolPath);
  const marker = `${spoolPath}.processed`;

  for (const event of events) {
    const response = await fetch(`${orchestratorUrl}/ingest`, {
      method: 'POST',
      body: event.wireFormat, // protobuf, not JSON
      headers: { 'Content-Type': 'application/protobuf' }
    });

    if (!response.ok) {
      // Do not truncate. Retry entire spool next cycle.
      await fs.unlink(marker).catch(() => {});
      throw new Error(`Orchestrator rejected event ${event.id}`);
    }
  }

  // All events confirmed. Mark spool as processed.
  await fs.writeFile(marker, new Date().toISOString());
  await fs.truncate(spoolPath, 0);
}

The key change: truncate only after the orchestrator confirms every event. If any event fails, the marker is deleted and the entire spool is retried.

What This Exposes About Agent Security Tooling

The meta-problem: a security tool that monitors agents is itself an agent. It makes tool calls (file I/O, network requests, process inspection). It has state (the trail, the spool, the detection rules). It has failure modes (orchestrator down, spool full, schema drift).

If you build agent security infrastructure, you inherit all the problems you are trying to solve for your customers. The difference is that your customers will not see your bugs until you publish a blog post about them.

The dogfooding gap: most security vendors showcase customer wins. Few publish their own vulnerabilities. Agentmetry did both. The bugs they found are not exotic. They are the kind of bugs that happen when you ship fast and audit later.

The lesson is not “security tools are broken.” The lesson is “security tools are software, and software has bugs, and the only way to find them is to use the tool on itself.”

Technical Verdict

Use this approach when:

You build agent security tooling and need to validate detection logic on real workloads
You want to expose the gap between designed behavior and actual behavior in production
You can afford the operational cost of running your security tool on the same system that builds it

Avoid this approach when:

You cannot tolerate the risk of your security tool failing silently (use external monitoring)
Your detection rules are not mature enough to handle build-time tool calls (you will drown in false positives)
You do not have a way to supervise the orchestrator process (the recorder will stop recording and you will not notice)

The hardest part is not building the tool. The hardest part is building the observability that tells you when the tool is lying to you.

Agentic AI, minus the noise.