mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

AI Agents

AWS Emulator: How 29,684 Agent Commits Built a Local Testing Clone of Every AWS Service

Inside AWS's agent-generated Rust emulator: orchestration patterns, commit graph forensics, API surface mapping, and the Docker dependency that reveals...

Source: simonwillison.net
AWS Emulator: How 29,684 Agent Commits Built a Local Testing Clone of Every AWS Service

AWS just shipped their first official local emulator for testing against AWS services without provisioning real infrastructure. The project generated 29,684 commits in three weeks using an agent swarm. This is not a marketing story about AI productivity. This is a case study in orchestrating agents to replicate a massive API surface, the commit patterns that reveal task decomposition strategy, and the architectural decisions that expose agent assumptions.

The Orchestration Challenge

AWS operates over 200 services with thousands of API endpoints. Building a local emulator means replicating request/response behavior, error codes, IAM policy evaluation, and state transitions for S3, DynamoDB, Lambda, SQS, and everything else.

The commit graph shows 29,684 commits in the first three weeks. That pace suggests parallel agent execution with minimal human review cycles. The plumbing questions:

  • How do you partition work across agents without overlap?
  • How do you ensure consistent error handling across 200+ service implementations?
  • What prevents agents from generating conflicting state management patterns?

The likely orchestration pattern: agents were assigned by service boundary, not by endpoint. Each agent received an OpenAPI spec (AWS publishes these for most services), a set of test cases derived from AWS SDK documentation, and a Rust project template. The coordinator merged outputs and ran integration tests to catch conflicts.

Commit Graph Forensics

29,684 commits in three weeks is 1,413 commits per day, or 59 per hour assuming 24-hour operation. That volume indicates:

  1. Micro-commits per endpoint: Each API operation likely generated its own commit (handler function, test case, mock data structure).
  2. Automated merge strategy: No human could review at this pace. The system likely auto-merged commits that passed CI.
  3. Parallel service development: Multiple agents working on different services simultaneously, with a dependency graph to handle cross-service references (e.g., Lambda invoking SQS).

The failure mode: agents generating syntactically correct but semantically wrong behavior. An S3 GetObject mock that returns 200 for non-existent keys instead of 404. A DynamoDB PutItem that ignores conditional expressions.

API Surface Mapping Strategy

AWS publishes Smithy models (their IDL) and OpenAPI specs for most services. The agent workflow likely looked like:

// Agent receives Smithy operation definition
operation GetObject {
    input: GetObjectInput,
    output: GetObjectOutput,
    errors: [NoSuchKey, InvalidObjectState]
}

// Agent generates handler skeleton
pub async fn get_object(
    state: &EmulatorState,
    input: GetObjectInput,
) -> Result<GetObjectOutput, GetObjectError> {
    // Agent fills in mock logic based on spec
    let bucket = state.buckets.get(&input.bucket)?;
    let object = bucket.objects.get(&input.key)
        .ok_or(GetObjectError::NoSuchKey)?;
    
    Ok(GetObjectOutput {
        body: object.data.clone(),
        content_type: object.content_type.clone(),
        // ... other fields from spec
    })
}

The validation problem: how do you confirm this matches production AWS behavior? You cannot call real AWS APIs during generation (cost, credentials, rate limits). The solution: agents generate test cases from AWS SDK examples and documentation, then humans run a validation suite against real AWS to catch divergence.

The Docker Dependency Problem

The emulator is written in Rust, a language that compiles to native binaries. Yet it requires Docker on macOS and Windows. This reveals agent assumptions about cross-platform development.

Likely causes:

  1. Linux-specific syscalls: Agents generated code using epoll, io_uring, or other Linux-only APIs for async I/O.
  2. Filesystem assumptions: Hardcoded /tmp paths or POSIX file permissions that don’t map to Windows.
  3. Dependency on Linux containers: Some AWS services (Lambda, ECS) require container runtime emulation, so agents defaulted to Docker for consistency.

The fix requires either:

  • Rewriting I/O layer to use cross-platform abstractions (tokio’s platform-agnostic runtime)
  • Shipping platform-specific binaries with conditional compilation
  • Accepting Docker as the lowest common denominator

The agent-generated codebase likely lacks the conditional compilation directives (#[cfg(target_os = "linux")]) needed for true portability.

State Management Across Services

AWS services maintain complex state relationships. An S3 bucket has objects, versioning settings, lifecycle policies, and event notifications that trigger Lambda. Agents must coordinate state:

State ChallengeAgent SolutionFailure Mode
Cross-service referencesShared in-memory state store with service-keyed namespacesRace conditions if agents don’t use proper locking
Event propagationPub/sub bus for S3→Lambda, DynamoDB→StreamsMissing events if agents implement async incorrectly
IAM policy evaluationCentralized policy engine called by all servicesAgents bypass engine and hardcode allow/deny
Resource quotasGlobal quota tracker updated on create/deleteAgents forget to decrement on resource cleanup

The emulator likely uses a centralized EmulatorState struct with service-specific sub-states:

pub struct EmulatorState {
    s3: S3State,
    dynamodb: DynamoDBState,
    lambda: LambdaState,
    iam: IAMState,
    event_bus: EventBus,
}

Agents generate service-specific state but rely on coordinator-provided scaffolding for cross-service communication.

Test Coverage Strategy

With 200+ services and thousands of endpoints, exhaustive testing is impossible. The agent strategy likely prioritized:

  1. Happy path coverage: Every endpoint has at least one test that returns 200.
  2. Error condition sampling: Common errors (404, 403, 400) but not exhaustive error combinations.
  3. Integration smoke tests: S3→Lambda, DynamoDB→Streams, but not deep multi-service workflows.

The gap: edge cases that only appear in production. An S3 multipart upload that fails on part 47 of 50. A DynamoDB transaction that deadlocks with specific key patterns. Agents cannot discover these without real-world usage data.

Observability Gaps

Agent-generated code typically lacks instrumentation. The emulator probably has:

  • Basic request/response logging
  • Error counters per service
  • No distributed tracing across service boundaries
  • No performance profiling hooks

Adding observability requires human intervention to insert tracing spans, metrics collection, and structured logging. Agents can generate boilerplate but struggle with context-aware instrumentation decisions (which operations are slow enough to warrant detailed tracing?).

Security Boundary Assumptions

The emulator runs locally, so agents likely skipped production security concerns:

  • No TLS termination (assumes localhost HTTP)
  • No rate limiting (assumes trusted local use)
  • Simplified IAM (may not enforce all policy conditions)
  • No audit logging (no CloudTrail equivalent)

This is appropriate for a testing tool but limits use cases. You cannot use this emulator to test IAM policy edge cases or rate limit handling.

Technical Verdict

Use this when:

  • You need fast, offline AWS integration tests without provisioning real resources
  • You are testing happy-path service interactions (S3 upload, DynamoDB query, Lambda invoke)
  • You accept Docker as a runtime dependency
  • You can tolerate behavioral differences from production AWS

Avoid this when:

  • You need to test IAM policy edge cases or security boundaries
  • You require bit-for-bit compatibility with production AWS error responses
  • You need cross-platform native binaries without Docker
  • You are testing performance characteristics (latency, throughput, cold starts)

The agent swarm approach worked because AWS had well-defined API specs and a bounded problem space. The 29,684 commits represent successful task decomposition, but the Docker dependency and likely observability gaps show where agent assumptions diverge from production requirements. This is a useful testing tool, not a production AWS replacement.