RookeryOS — Voice Streaming Engine (Sonar)

Sub-second voice for agents — WebRTC UDP replacing Gemini Live

Overview

Sonar is the voice streaming engine for RookeryOS. It replaces cloud-dependent voice services (Gemini Live) with a local, CPU-optimized pipeline that delivers sub-second audio latency at a fraction of the cost.

Key Metrics:

Latency: <300ms end-to-end (vs 1-3s with cloud APIs)
Cost: ~$33/mo new infrastructure (vs $200+/mo cloud voice)
Processing: Local STT (Whisper) + Local TTS, no cloud API calls
Transport: WebRTC UDP for real-time audio

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Sonar Voice Pipeline                       │
│                                                               │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐               │
│  │  Client   │    │  Signaling│    │  STT/TTS │               │
│  │  Browser  │◄──►│  Server  │◄──►│  Service │               │
│  │           │    │          │    │          │               │
│  │  Microphone│   │  WebRTC  │    │  Faster  │               │
│  │  Speaker  │    │  Session │    │  Whisper │               │
│  │           │    │  Manager │    │  (STT)   │               │
│  └──────────┘    └──────────┘    │          │               │
│                                   │  Local   │               │
│                                   │  TTS     │               │
│                                   └────┬─────┘               │
│                                        │                     │
│                                   ┌────┴─────┐              │
│                                   │  Agent   │              │
│                                   │  Pod     │              │
│                                   │  (Hudson)│              │
│                                   └──────────┘              │
└──────────────────────────────────────────────────────────────┘

Audio Flow

User speaks
  │
  ├── 1. Browser captures audio (WebRTC)
  │     MediaRecorder → Opus chunks
  │
  ├── 2. WebRTC UDP → Signaling Server
  │     <100ms transport latency
  │
  ├── 3. STT Service (Faster Whisper)
  │     Opus → WAV → Whisper → Text
  │     ~150ms processing
  │
  ├── 4. Agent LLM (Hudson Gateway)
  │     Text → Z.ai GLM-4.5 → Response text
  │     ~500ms inference
  │
  ├── 5. TTS Service (Local)
  │     Response text → Audio
  │     ~100ms processing
  │
  └── 6. WebRTC UDP → Browser → Speaker
        <100ms transport

Components

STT/TTS Service (`apps/stt-tts-service`)

A Python FastAPI service that handles speech-to-text and text-to-speech.

Component	Technology	Purpose
STT	Faster Whisper	Speech-to-text (CPU/GPU modes)
VAD	Silero VAD	Voice activity detection
TTS	Local TTS	Text-to-speech (CPU/GPU modes)
Audio Processing	librosa, soundfile	Audio format conversion
Codec	Opus	Audio compression

API Endpoints

Method	Path	Purpose
`POST`	`/stt`	Speech-to-text transcription
`POST`	`/tts`	Text-to-speech synthesis
`GET`	`/health`	Service health check
`WebSocket`	`/voice-stream`	Real-time audio streaming

Audio Processing

# Input: Opus audio from WebRTC
# Processing:
1. Opus → WAV conversion (soundfile)
2. VAD: Detect speech segments (Silero)
3. STT: Transcribe segments (Faster Whisper)
4. Output: Text with timestamps

# Output: WAV audio for TTS
# Processing:
1. Text → Audio synthesis (Local TTS)
2. WAV → Opus conversion
3. Stream via WebRTC

WebRTC Signaling

WebRTC connection management for real-time audio:

Client (Browser)
  │
  ├── SDP Offer (via signaling server)
  │     Contains: codecs, ICE candidates, media constraints
  │
  ├── Signaling Server (API)
  │     Relays SDP between client and voice service
  │
  ├── ICE Candidates (STUN/TURN)
  │     NAT traversal for direct UDP connection
  │
  └── Direct UDP Audio Stream
        Opus codec, 48kHz sample rate

Voice Multiplexer (`apps/api/src/services/voice-multiplexer.ts`)

Manages multiple concurrent voice sessions:

Session creation and teardown
Audio stream routing between client and agent
Voice activity detection for turn-taking
Interruption handling (barge-in)

Integration Points

Console Integration

The Platform Admin console connects to Sonar via:

Component	File	Purpose
VoicePipeline	`apps/console/src/components/VoicePipeline.tsx`	Voice call interface
VoicePipelineSettings	`apps/console/src/components/VoicePipelineSettings.tsx`	Configuration
useVoicePipeline	`apps/console/src/hooks/useVoicePipeline.ts`	Voice call management
useWebRTCVoice	`apps/console/src/hooks/useWebRTCVoice.ts`	WebRTC connection
useAssessmentVoice	`apps/console/src/hooks/useAssessmentVoice.ts`	Voice assessments

API Routes

Route	File	Purpose
`/voice-stream`	`apps/api/src/routes/voice-stream.ts`	WebSocket voice streaming
`/api/v1/voice/*`	Various	Voice API endpoints

Lighthouse Integration

Voice recordings are processed into knowledge:

Recording → Sonar Pipeline → Transcription → Lighthouse ingest
                                         ↓
                              Available to agents as knowledge

Deployment

Kubernetes

Namespace: voice
Replicas: 1
Image: aw2-stt-tts
Service: ClusterIP :8000
Resources:
  requests: { cpu: 500m, memory: 1Gi }
  limits: { cpu: 2000m, memory: 4Gi }
Route: api.agentworksstudio.com/voice/*

Cost Model

Component	Monthly Cost
Voice Service (e2-standard-2 spot)	~$25
Storage (recordings)	~$5
Network (WebRTC)	~$3
Total	~$33/mo

File Locations

apps/stt-tts-service/
├── audio_utils.py               # Audio processing utilities
├── requirements.txt             # Python dependencies
├── Dockerfile                   # Container build
└── main.py                      # FastAPI service

apps/api/src/
├── routes/voice-stream.ts       # WebSocket voice endpoint
└── services/voice-multiplexer.ts # Session management

apps/console/src/
├── components/VoicePipeline.tsx
├── hooks/useVoicePipeline.ts
├── hooks/useWebRTCVoice.ts
└── hooks/useAssessmentVoice.ts

PreviousLighthouse Knowledge Layer Next Security Model