Spread is a production API serving real-time grocery prices from three New Zealand supermarket chains: Pak’nSave, New World, and Woolworths. It exposes product search, current price observations, and competitor comparisons through both REST endpoints and a Model Context Protocol (MCP) server. The project started as the data layer for ButterUp.app, a price comparison tool, then split into a standalone API when the scraping and normalization infrastructure became useful on its own.
The interesting plumbing sits in three places: the scraping pipeline that handles schema drift across three different backends, the normalization layer that reconciles product identifiers and units for apples-to-apples comparisons, and the MCP server that lets agents query prices without building their own collection stack.
Scraping Pipeline Architecture
Spread scrapes three separate supermarket backends. Each chain runs different e-commerce platforms with distinct HTML structures, API endpoints, and rate limits. The pipeline needs to extract product names, SKUs, prices, unit sizes, and availability without breaking when a retailer pushes a site redesign.
Key scraping challenges:
- Schema detection: Each chain’s product page uses different CSS selectors and JSON-LD schemas. The scraper maintains a selector map per chain and runs a validation step on every fetch to detect structural changes before they propagate downstream.
- Rate limiting: Supermarket sites throttle aggressive crawlers. The pipeline spaces requests with jitter, rotates user agents, and respects
robots.txtcrawl delays. If a chain blocks an IP, the system flags the data source as stale rather than retrying aggressively. - Product matching: The same physical product appears under different SKUs across chains. Spread uses fuzzy matching on product names, brand strings, and unit sizes to link equivalent items. This matching layer runs after scraping but before normalization.
The scraper runs on a scheduled job (likely cron or a task queue like Celery). Each chain gets its own scraping task with independent retry logic. If one chain’s site goes down, the other two continue updating.
Normalization Layer
Raw scraped data arrives in inconsistent formats: prices in cents or dollars, weights in grams or kilograms, pack sizes as strings like “6 x 330ml” or “2L”. The normalization layer converts everything into a canonical schema so API consumers can compare prices across chains without parsing unit strings.
Normalization steps:
- Price conversion: All prices convert to cents (integer) to avoid floating-point errors in financial comparisons.
- Unit standardization: Weights normalize to grams, volumes to milliliters. The parser handles common patterns like “kg”, “g”, “L”, “ml”, “each”, “pack of X”.
- Unit price calculation: For multi-packs, the system calculates price per unit (e.g., per 100g or per item) so consumers can compare a 500g pack at one chain against a 750g pack at another.
- Product deduplication: Fuzzy matching groups equivalent products across chains. The system stores a canonical product ID and links each chain’s SKU to it.
Stale data gets flagged when a chain’s last successful scrape exceeds a threshold (probably 24 hours). The API response includes a last_updated timestamp and a stale boolean so consumers know when to trust the data.
MCP Server Implementation
The MCP server exposes Spread’s data to AI agents through the Model Context Protocol. Agents can call tools like search_products, get_price, and compare_prices without managing API keys or pagination themselves.
MCP tool design:
# Example MCP tool definition (conceptual)
{
"name": "compare_prices",
"description": "Compare prices for a product across NZ supermarkets",
"parameters": {
"product_query": {
"type": "string",
"description": "Product name or search term"
},
"chains": {
"type": "array",
"items": {"type": "string"},
"description": "List of chains to compare (paknsave, newworld, woolworths)"
}
},
"returns": {
"type": "array",
"items": {
"chain": "string",
"product_name": "string",
"price_cents": "integer",
"unit_price": "string",
"last_updated": "timestamp"
}
}
}The MCP server likely runs as a separate process (Node.js or Python) that wraps the REST API. It handles authentication (API keys), caching (to avoid redundant queries), and error translation (HTTP errors to MCP error codes).
Caching strategy:
- Price data changes slowly (hours, not seconds), so the MCP server can cache responses for 15-30 minutes.
- Cache keys include product query and chain list. Invalidation happens on a TTL basis, not on upstream updates.
- Stale data gets served with a warning flag rather than blocking the agent’s request.
API Key and Rate Limiting
Spread uses API keys for authentication and enforces usage limits to prevent abuse. The 7-day free trial suggests a tiered pricing model with different rate limits per plan.
Rate limiting architecture:
| Component | Strategy | Failure Mode |
|---|---|---|
| API Gateway | Token bucket per API key | 429 Too Many Requests with Retry-After header |
| MCP Server | Shared pool for all MCP clients | Degrades to cached data if quota exceeded |
| Scraper | Per-chain rate limit with backoff | Marks chain as stale, continues other chains |
| Database | Connection pooling with max connections | Queues requests, returns 503 if pool exhausted |
The API likely uses Redis or a similar in-memory store to track request counts per key. Each request increments a counter with a TTL matching the rate limit window (e.g., 1000 requests per hour). When the counter exceeds the limit, the gateway returns a 429 response.
Observability and Failure Detection
The pipeline needs to detect when a supermarket site changes structure, goes offline, or starts returning garbage data. Observability hooks include:
- Scraper health checks: Each scraping task logs success/failure and the number of products extracted. A sudden drop in product count triggers an alert.
- Schema validation: After parsing, the system checks that required fields (price, product name, SKU) are present and non-null. Missing fields indicate a schema change.
- Price anomaly detection: If a product’s price jumps by more than 50% in a single update, the system flags it for manual review before serving it through the API.
- Staleness tracking: The API response includes
last_updatedtimestamps. Consumers can decide whether to trust data that’s hours or days old.
When a chain’s scraper fails repeatedly, the system marks that chain’s data as stale and continues serving the last known good prices with a warning. This prevents a single chain’s downtime from breaking the entire API.
Deployment Shape
Spread likely runs on a cloud provider (AWS, GCP, or Fly.io) with the following components:
- Scraper workers: Background jobs (cron or task queue) that fetch and parse supermarket data.
- API server: REST endpoints for product search, price queries, and comparisons. Probably a Python (FastAPI/Flask) or Node.js (Express) app.
- MCP server: Separate process exposing MCP tools. Could be co-located with the API server or run as a standalone service.
- Database: PostgreSQL or similar relational DB for storing products, prices, and API keys. Includes indexes on product names and SKUs for fast search.
- Cache layer: Redis for rate limiting counters and API response caching.
The scraper and API server can scale independently. If scraping load increases (e.g., adding more chains), you add more worker processes without touching the API tier.
Technical Verdict
Use Spread when:
- You need New Zealand grocery data without building a scraping pipeline.
- You’re building a price comparison tool, shopping agent, or analytics dashboard.
- You want MCP integration for agent-driven workflows without managing API pagination and caching yourself.
Avoid Spread when:
- You need global grocery data (it only covers NZ).
- You require sub-hour price updates (scraping frequency is probably daily or every few hours).
- You need historical price trends (the API focuses on current prices, not time series).
The core value is the normalization layer. Scraping three supermarket sites is tedious but straightforward. Reconciling product identifiers, units, and pricing formats so competitor comparisons return meaningful data is the hard part. If you’re building a shopping agent or price tracker, Spread saves you from writing that reconciliation logic yourself.