Natera built a production voice agent that lets patients book mobile phlebotomy appointments over the phone with natural conversation. The system achieves 100% tool-calling accuracy and sub-7-second end-to-end latency in production. The architecture is interesting because it solves three hard problems at once: bridging telephony and LLM streams without blocking, masking unavoidable latency through event-driven design, and verifying patient identity mid-conversation without breaking flow.
This is not a demo. Natera handles real patient calls for appointment scheduling, which means the agent must understand scheduling constraints, verify insurance, and authenticate callers while maintaining conversational continuity. The plumbing behind this involves Amazon Bedrock AgentCore, a dual-WebSocket bridge, and a progressive-trust authentication pattern that most voice agent implementations skip.
Architecture: Dual-WebSocket Bridge
The core challenge is coordinating two independent real-time streams: telephony audio coming from the patient and LLM responses from Bedrock AgentCore. Most voice agent architectures introduce blocking delays when they serialize these streams or wait for full turn completion before responding.
Natera’s solution uses two WebSocket connections:
- Telephony WebSocket: Streams audio from the phone system to the agent orchestrator
- AgentCore WebSocket: Streams LLM responses and tool invocations back to the orchestrator
The orchestrator sits between these two sockets and manages state without blocking either stream. When audio arrives from the patient, it gets transcribed and sent to AgentCore. When AgentCore emits a response or tool call, the orchestrator routes it to the appropriate backend system and synthesizes audio back to the patient.
The key insight is that the orchestrator does not wait for AgentCore to finish thinking before starting the next action. It processes events as they arrive and pipelines operations wherever possible.
Event-Driven Latency Masking
Sub-7-second latency is not about making the LLM faster. It is about hiding unavoidable delays through event-driven design. Natera calls this “latency masking.”
Here is how it works in practice:
- Streaming transcription: Audio chunks get transcribed incrementally, not after the patient finishes speaking
- Partial response synthesis: The agent starts generating audio as soon as AgentCore emits the first tokens, not after the full response completes
- Parallel tool execution: When AgentCore invokes a tool (like checking appointment availability), the orchestrator fires the API call immediately and streams results back without waiting for the entire tool response
- Conversational overlap: The agent can acknowledge the patient’s input with a short phrase while still processing the full request in the background
This is not speculative optimization. The case study reports sub-7-second end-to-end latency in production, which includes network round trips, LLM inference, tool execution, and audio synthesis.
| Latency Component | Traditional Approach | Event-Driven Masking |
|---|---|---|
| Transcription | Wait for silence detection | Stream audio chunks incrementally |
| LLM response | Wait for full completion | Synthesize audio from first tokens |
| Tool execution | Block until result returns | Fire API call, stream partial results |
| User feedback | Silent gap during processing | Acknowledge immediately, process in background |
Progressive-Trust Authentication
Healthcare voice agents face a hard constraint: they must verify patient identity before discussing protected health information or booking appointments. Traditional authentication (enter your member ID, date of birth, etc.) breaks conversational flow and adds 30-60 seconds to every call.
Natera uses progressive-trust authentication. The agent starts the conversation without full authentication and collects identifying information naturally during the scheduling discussion. As the patient provides details (name, reason for appointment, preferred location), the agent incrementally builds confidence in their identity.
The authentication flow looks like this:
- Initial greeting: Agent asks how it can help without requiring credentials
- Natural information gathering: Patient mentions they need a blood draw, provides their name
- Contextual verification: Agent asks for date of birth or member ID as part of confirming the appointment details
- Progressive escalation: If the agent cannot verify identity with soft signals, it escalates to explicit authentication before completing the booking
This pattern keeps the conversation natural while meeting compliance requirements. The agent does not ask for credentials upfront, but it also does not perform privileged actions until identity is confirmed.
Tool-Calling Accuracy
The case study reports 100% tool-calling accuracy in production. This is notable because most agentic systems struggle with tool selection and parameter extraction, especially in voice interactions where transcription errors compound LLM mistakes.
Natera achieves this through:
- Constrained tool definitions: Each tool has a narrow, well-defined purpose (check availability, book appointment, verify insurance)
- Explicit parameter validation: The orchestrator validates tool parameters before execution and asks clarifying questions if required fields are missing
- Feedback loops: When a tool call fails, the agent explains why and guides the patient toward a valid request
The architecture does not rely on the LLM to get tool calls perfect on the first try. It assumes mistakes will happen and builds recovery paths into the orchestration layer.
Deployment Shape
The system runs on AWS infrastructure with these components:
- Amazon Bedrock AgentCore: Handles LLM inference, tool orchestration, and conversational memory
- WebSocket orchestrator: Custom service that bridges telephony and AgentCore streams
- Telephony integration: Connects to Natera’s existing phone system via SIP or cloud telephony provider
- Backend APIs: Appointment scheduling, insurance verification, patient records
The orchestrator is stateless. It holds conversational context in memory during active calls but does not persist state between sessions. This simplifies scaling and failure recovery. If an orchestrator instance crashes, the telephony system reconnects to a new instance and AgentCore reconstructs context from the conversation history.
Failure Modes
Voice agents fail differently than text-based agents. Here are the likely failure modes and how Natera’s architecture handles them:
Transcription errors: The agent asks clarifying questions when it detects low confidence in the transcription. It does not guess or proceed with ambiguous input.
Tool execution failures: If an API call to the scheduling system fails, the agent explains the issue and offers alternatives (call back later, speak to a human agent).
Authentication failures: If the agent cannot verify identity after multiple attempts, it escalates to a human agent rather than blocking the patient.
Network interruptions: The WebSocket orchestrator detects dropped connections and attempts to reconnect. If reconnection fails, the telephony system routes the call to a fallback queue.
LLM hallucinations: The orchestrator validates all tool calls and responses against known constraints before executing them. If AgentCore suggests an invalid appointment time, the orchestrator rejects it and asks for clarification.
Code Sketch: WebSocket Orchestrator
Here is a simplified version of the orchestrator logic that bridges telephony and AgentCore streams:
import asyncio
import websockets
import json
class VoiceAgentOrchestrator:
def __init__(self, telephony_ws, agentcore_ws):
self.telephony_ws = telephony_ws
self.agentcore_ws = agentcore_ws
self.conversation_state = {}
async def handle_telephony_stream(self):
"""Process incoming audio from patient"""
async for message in self.telephony_ws:
audio_chunk = message['audio']
transcript = await self.transcribe(audio_chunk)
# Send to AgentCore without waiting for response
await self.agentcore_ws.send(json.dumps({
'type': 'user_input',
'text': transcript,
'session_id': self.conversation_state['session_id']
}))
async def handle_agentcore_stream(self):
"""Process responses and tool calls from AgentCore"""
async for message in self.agentcore_ws:
event = json.loads(message)
if event['type'] == 'response_chunk':
# Start synthesizing audio immediately
audio = await self.synthesize_partial(event['text'])
await self.telephony_ws.send({'audio': audio})
elif event['type'] == 'tool_invocation':
# Execute tool without blocking response stream
asyncio.create_task(self.execute_tool(event))
async def execute_tool(self, tool_event):
"""Execute tool call and stream results back"""
tool_name = tool_event['tool']
params = tool_event['parameters']
# Validate parameters before execution
if not self.validate_params(tool_name, params):
await self.agentcore_ws.send(json.dumps({
'type': 'tool_error',
'message': 'Missing required parameters'
}))
return
# Call backend API
result = await self.call_backend_api(tool_name, params)
# Stream result back to AgentCore
await self.agentcore_ws.send(json.dumps({
'type': 'tool_result',
'tool': tool_name,
'result': result
}))
async def run(self):
"""Run both streams concurrently"""
await asyncio.gather(
self.handle_telephony_stream(),
self.handle_agentcore_stream()
)The key pattern is that both streams run concurrently. The orchestrator does not block one stream waiting for the other. Tool execution happens in a separate task so the response stream can continue while the tool runs.
Observability Gaps
The case study does not mention observability, which is a red flag for production voice agents. Here are the metrics you would need to track:
- Latency breakdown: Time spent in transcription, LLM inference, tool execution, and audio synthesis
- Tool call success rate: How often tool calls succeed on the first try vs. requiring clarification
- Authentication success rate: How often the agent successfully verifies identity without escalation
- Conversation completion rate: How often patients complete their booking vs. abandoning the call
- Error recovery rate: How often the agent recovers from transcription errors or tool failures without human intervention
Without these metrics, you cannot diagnose latency spikes, authentication failures, or tool-calling regressions.
Technical Verdict
Use this architecture when:
- You need sub-10-second latency for voice interactions
- Your use case requires mid-conversation authentication without breaking flow
- You have well-defined tools with clear success and failure modes
- You can invest in a custom orchestrator to bridge telephony and LLM streams
Avoid this architecture when:
- Your use case tolerates higher latency (text-based agents are simpler)
- You need complex multi-turn reasoning that cannot be pipelined
- Your tools have unpredictable latency or failure modes
- You lack the infrastructure team to operate custom WebSocket orchestrators
The dual-WebSocket bridge and event-driven latency masking are not trivial to implement. You need engineers who understand async programming, WebSocket lifecycle management, and real-time audio processing. But if you need production-grade voice agents with low latency and high reliability, this architecture shows a clear path.