Nao introduces context engineering as a distinct layer between data warehouses and agent execution. Instead of treating agent memory as a RAG pipeline problem, Nao uses a file-system-like context builder that separates data catalog construction (via CLI) from agent deployment (via chat UI). This split enforces boundaries between data engineering and runtime execution, making agent performance testable and versionable.
The project ranks 8th in GitHub’s TypeScript trending list with 1,904 stars. It supports BigQuery, Snowflake, Databricks, and PostgreSQL with a self-hosted deployment model that includes Slack bot integration.
Architecture: CLI for Context, UI for Execution
Nao splits the analytics agent lifecycle into two distinct phases:
Phase 1: Context Creation (nao-core CLI)
- Data team builds a file-system-like context structure
- Adds schemas, metadata, documentation, tools, and MCP servers
- Defines rules and modeling logic
- Tests agent performance against known queries
- Versions context changes like code
Phase 2: Agent Deployment (nao chat UI)
- Business users query the agent in natural language
- Agent reads from the pre-built context
- Generates SQL, executes queries, renders visualizations
- Collects feedback for context refinement
This separation means the agent never constructs its own memory at runtime. The context is an artifact, not a dynamic retrieval system.
Context Engineering vs. RAG
Traditional text-to-SQL tools use vector embeddings to retrieve relevant schema fragments. Nao treats context as a curated file system instead.
| Approach | Memory Construction | Update Mechanism | Testing Strategy |
|---|---|---|---|
| RAG-based | Runtime retrieval from vector store | Re-embed on schema change | Query accuracy metrics |
| Nao context | Pre-built file structure via CLI | Version-controlled context files | Unit tests per context version |
The file-system model means you can inspect, diff, and review context changes before deployment. You add a new table by editing a file, not by re-indexing embeddings.
Unit Testing Agent Performance
Nao includes a testing framework for agent reliability. You define test cases as query-answer pairs, then run them against a context version.
nao test --context-version v1.2.3A test case might look like:
# tests/revenue_queries.yaml
- query: "What was revenue last quarter?"
expected_sql_pattern: "SELECT SUM(revenue) FROM sales WHERE quarter = 'Q4'"
expected_result_shape: "single_value"
- query: "Show me top 10 customers by spend"
expected_sql_pattern: "ORDER BY total_spend DESC LIMIT 10"
expected_result_shape: "table"The test runner executes each query, compares generated SQL against patterns, and validates result shapes. Failures indicate context drift or ambiguous metadata.
This approach treats agent performance as a regression testing problem. You version the context, run tests, and track pass rates over time.
State Management and Observability
Nao’s state model is simple: the context is read-only at runtime, and all state lives in user sessions.
Context Layer
- Immutable during agent execution
- Loaded once at startup
- Updated only through CLI commands
Session Layer
- Conversation history
- Generated SQL queries
- User feedback (thumbs up/down)
- Execution traces
Observability comes from two sources:
- User feedback: Every answer can be marked correct or incorrect, feeding back into context refinement.
- Query logs: All generated SQL is logged with the original natural language query, making it easy to spot patterns in failures.
The system does not include built-in tracing for LLM calls or token usage. You bring your own observability stack if you need distributed tracing.
Deployment Shape
Nao is self-hosted. You run the chat UI as a web service and optionally deploy a Slack bot that shares the same context.
Deployment components:
- nao-core CLI (runs on data engineer’s machine)
- nao chat UI (Node.js service, connects to data warehouse)
- Optional Slack bot (shares context with chat UI)
- Data warehouse (BigQuery, Snowflake, Databricks, PostgreSQL)
- LLM API (OpenAI, Anthropic, or self-hosted)
The chat UI reads the context directory at startup. To update the agent, you rebuild the context with the CLI and restart the service. No hot-reloading, no dynamic context updates.
Security Boundaries
Nao enforces security at three layers:
- Data warehouse credentials: The chat UI connects to your warehouse with service account credentials. Users never see raw SQL or connection strings.
- LLM API keys: You provide your own keys. Nao does not proxy LLM calls through a third-party service.
- Context access control: The context is a file system. You control who can edit it through standard file permissions and version control.
The system does not include row-level security or user-based query filtering. If a user can access the agent, they can query any data in the context. You handle access control at the warehouse level or by deploying multiple agents with different contexts.
Likely Failure Modes
Context drift: If the warehouse schema changes but the context is not updated, the agent generates invalid SQL. The unit testing framework catches this, but only if you run tests regularly.
Ambiguous metadata: If two tables have similar names or overlapping columns, the agent may choose the wrong one. The file-system model makes this easier to debug (you can inspect the context files), but it does not prevent the problem.
LLM hallucination: The agent can still generate plausible but incorrect SQL if the context is incomplete or the query is ambiguous. User feedback helps identify these cases, but there is no automatic correction.
Deployment lag: Updating the context requires a service restart. If you need real-time schema updates, you will need to build a hot-reload mechanism or run the CLI in a continuous deployment pipeline.
Technical Verdict
Use Nao when:
- You need a production analytics agent with explicit context boundaries
- Your data team wants to version and test agent performance like code
- You prefer self-hosted deployment over managed services
- You have stable schemas and can tolerate restart-based updates
Avoid Nao when:
- You need real-time schema updates without service restarts
- You require row-level security or user-based query filtering
- You want a managed service with built-in observability and tracing
- Your schemas change frequently and unpredictably
Nao’s strength is the separation of context creation from agent execution. This makes agent performance testable and versionable, but it also means you need a deployment pipeline to keep the context in sync with your warehouse.