mech.app
AI Agents

Pokayoke: How Deterministic Guardrails Enforce Repository Conventions When Agents Forget

LLM-generated AST checks and custom linters as executable guardrails for coding agents that drift on multi-constraint codebases.

Source: pokayoke.codes
Pokayoke: How Deterministic Guardrails Enforce Repository Conventions When Agents Forget

Coding agents excel at one-off tasks but drift when they encounter multi-constraint codebases. You declare conventions in AGENTS.md, the agent acknowledges them, then three commits later: it forgets to reference the dependency catalog or regresses on a metric you were tracking. Prompts alone don’t scale when the ruleset is complex or the agent context window rotates.

Pokayoke treats this as a deterministic enforcement problem. Instead of hoping the agent remembers, you instantiate repository conventions as executable checks: AST traversals, large-scale string matching, or custom linters that existing tools like ESLint don’t support. The agent writes the rule once, and the guardrail runs every time.

The Problem Space

Coding agents work well when the task is isolated. They struggle when:

  • The repository has idiosyncratic conventions (workspace-only catalog references, custom module boundaries, metric thresholds).
  • The agent must respect multiple constraints simultaneously (formatting, naming, dependency sourcing, test coverage).
  • The context window rotates and the agent loses track of earlier decisions.

Traditional linters handle syntax and style. They don’t handle arbitrary repository-specific invariants like “all package.json dependencies in this workspace must reference the catalog” or “no direct imports from src/internal outside the module boundary.”

You can write custom scripts, but maintaining a “script cupboard” of ad-hoc checks is cumbersome. Each new convention requires boilerplate for AST parsing, file traversal, and error reporting.

How Pokayoke Works

Pokayoke is a TypeScript/Bun library that turns repository conventions into runnable checks. The workflow:

  1. Define a rule: You (or the agent) write a TypeScript function that encodes the convention.
  2. Instantiate the check: Pokayoke wraps it in a standard interface with file traversal, error formatting, and repair hooks.
  3. Run the guardrail: Integrate it into your CI pipeline, pre-commit hook, or agent feedback loop.

The library provides:

  • AST utilities for parsing TypeScript, JavaScript, and JSON files.
  • File system traversal with glob patterns and workspace awareness.
  • Error reporting that agents can parse and act on.
  • Repair hooks so the agent can fix violations automatically.

Example: Catalog-Only Dependencies

You want every package.json in a workspace to reference dependencies from a shared catalog, not hardcoded versions. ESLint can’t enforce this. Pokayoke can:

// Pseudocode for illustration
import { defineRule, parseJson, traverseWorkspace, writeJson } from 'pokayoke';

export const catalogOnlyDeps = defineRule({
  name: 'catalog-only-dependencies',
  description: 'All package.json dependencies must reference the catalog',
  
  async check(context) {
    const violations = [];
    
    await traverseWorkspace(context.root, 'package.json', async (filePath) => {
      const pkg = await parseJson(filePath);
      
      for (const [name, version] of Object.entries(pkg.dependencies || {})) {
        if (typeof version === 'string' && !version.startsWith('catalog:')) {
          violations.push({
            file: filePath,
            line: null,
            message: `Dependency ${name} must use catalog reference`,
            fix: `Change ${name} from ${version} to catalog:${name}`
          });
        }
      }
    });
    
    return violations;
  },
  
  async repair(context, violation) {
    const pkg = await parseJson(violation.file);
    const depName = violation.message.split(' ')[1];
    pkg.dependencies[depName] = `catalog:${depName}`;
    await writeJson(violation.file, pkg);
  }
});

The agent writes this once. The check runs deterministically. If it fails, the agent sees the violation and can retry with the repair hook.

Architecture

Pokayoke sits between the agent and the repository. It does not replace ESLint, Prettier, or TypeScript’s compiler. It complements them by handling rules they can’t express.

LayerResponsibilityTool
Syntax & styleFormatting, semicolons, indentationPrettier, ESLint
Type safetyType errors, unused variablesTypeScript compiler
Repository conventionsCatalog references, module boundaries, custom metricsPokayoke
Agent orchestrationTask planning, code generation, retry logicYour agent framework

Integration Points

Pre-commit hook: Run Pokayoke checks before the agent commits. If a check fails, block the commit and surface the error to the agent.

CI pipeline: Add Pokayoke to your GitHub Actions or GitLab CI. Treat violations as build failures.

Agent feedback loop: When the agent generates code, run Pokayoke checks immediately. If a violation occurs, append the error to the agent’s context and let it retry.

# .github/workflows/pokayoke.yml
name: Repository Conventions
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: oven-sh/setup-bun@v1
      - run: bun install
      - run: bun run pokayoke check

State Management

Pokayoke is stateless by design. Each check runs independently and returns a list of violations. You own the baseline tracking logic in your orchestration layer.

If you want to track violations over time (e.g., “don’t introduce new violations, but allow existing ones”), store the baseline in a JSON file and diff against it:

const baseline = await readJson('.pokayoke-baseline.json');
const current = await runChecks();
const newViolations = current.filter(v => !baseline.includes(v));

if (newViolations.length > 0) {
  throw new Error('New violations introduced');
}

Security Boundaries

Pokayoke rules are TypeScript code. They run with the same permissions as your build process. If an agent writes a malicious rule, it can read files, make network requests, or execute arbitrary commands.

Potential mitigation strategies:

  • Code review: Treat agent-generated rules like any other code. Review them before merging.
  • Sandboxing: Run Pokayoke checks in a container or VM with limited permissions.
  • Static analysis: Scan rules for dangerous patterns (e.g., eval, child_process.exec) before execution.
  • Allowlisting: Only permit rules that import from a predefined set of safe utilities.

The library itself does not enforce sandboxing. You handle it at the orchestration level.

Observability

Pokayoke outputs structured JSON for each violation. Agents parse this output to identify violations and trigger repairs; human-readable logs are insufficient for agentic feedback loops.

{
  "rule": "catalog-only-dependencies",
  "file": "packages/api/package.json",
  "line": null,
  "message": "Dependency lodash must use catalog reference",
  "fix": "Change lodash from ^4.17.21 to catalog:lodash"
}

You can pipe this to your observability stack (Datadog, Honeycomb, Grafana) to track:

  • Violation frequency: Which rules fail most often?
  • Agent repair success rate: How often does the agent fix violations on the first retry?
  • Drift over time: Are violations increasing or decreasing?

Example: send violations to Honeycomb after each CI run:

const violations = await runChecks();
await fetch('https://api.honeycomb.io/1/events/pokayoke', {
  method: 'POST',
  headers: { 'X-Honeycomb-Team': process.env.HONEYCOMB_KEY },
  body: JSON.stringify({
    violations: violations.length,
    rules: violations.map(v => v.rule),
    timestamp: new Date().toISOString()
  })
});

Failure Modes

Agent ignores violations: The agent sees the error but doesn’t fix it, either because the fix is too complex or the agent’s context window is full.

Mitigation: Surface violations as blocking errors. If the agent can’t fix them, escalate to a human. In the feedback loop, append the violation JSON to the agent’s next prompt with explicit instructions: “Fix this before proceeding.”

Potential edge cases to test for: Rules may miss corner cases like scoped packages (@org/package) or workspace protocol references (workspace:*). Write tests for your rules. Pokayoke rules are just TypeScript functions. Test them like any other code.

Rule performance: AST parsing is expensive. A rule that traverses every file in a large monorepo can slow down CI.

Mitigation: Use glob patterns to limit scope. Cache parse results. Run checks in parallel.

Conflicting rules: Two rules enforce contradictory conventions. Example: one rule mandates catalog references, another mandates exact versions for security-critical packages.

Mitigation: Pokayoke does not resolve conflicts. You handle this in your orchestration layer by prioritizing rules or disabling conflicting ones.

Deployment Shape

Pokayoke is a library, not a service. You install it as a dev dependency in your repository:

bun add -d pokayoke

You define rules in a pokayoke/ directory at the repository root:

repo/
├── pokayoke/
│   ├── catalog-only-deps.ts
│   ├── module-boundaries.ts
│   └── index.ts
├── package.json
└── bun.lockb

You run checks via a script in package.json:

{
  "scripts": {
    "check:conventions": "pokayoke check"
  }
}

The agent can invoke this script directly or you can integrate it into your CI pipeline.

When Agents Write Rules

The library includes a SKILL.md file that agents can read to generate rules autonomously. The agent:

  1. Reads the repository conventions from AGENTS.md or a similar file.
  2. Identifies conventions that require deterministic enforcement.
  3. Writes a Pokayoke rule using the library’s API.
  4. Adds the rule to the pokayoke/ directory and commits it.

Example agent prompt:

“The repository requires all package.json dependencies to reference the catalog. Write a Pokayoke rule that enforces this convention.”

The agent generates the rule from the earlier example. You review it, merge it, and the guardrail is live.

Comparison with Existing Tools

ToolScopeExtensibilityAgent-Friendly
PrettierCode formattingLimited, opinionatedLow
TypeScript compilerType safetyNoneLow
PokayokeRepository-specific conventionsFull, write arbitrary TypeScriptHigh

Pokayoke lowers the barrier by providing a simpler API and letting agents write rules directly.

Technical Verdict

Use Pokayoke when your coding agents regress on metrics or forget constraints despite AGENTS.md declarations. Best for TypeScript/Bun monorepos where agents write rules once and guardrails run deterministically on every commit. The feedback loop is explicit: violations block commits or trigger agent retry with structured error context. If your conventions are complex (catalog references, module boundaries, custom invariants) and existing linters can’t express them, Pokayoke fills the gap.

Avoid Pokayoke when your conventions are already expressible in existing linters. If you don’t trust agents to write safe, correct rules, the security surface is too large. If you want a hosted service with a UI, this is a library, not a platform. The project is at v0.0.5 alpha, so expect API changes and incomplete documentation. If you need cross-language support beyond TypeScript/Bun, look elsewhere.

The core idea is sound: treat repository conventions as code, not prompts. If you’re running coding agents in production and they drift on multi-constraint codebases, you need deterministic guardrails. Pokayoke is one way to build them.