Speed = Natural Conversation
Latența peste 300ms face conversația să se simtă nenaturală. Ținta: sub 200ms end-to-end pentru experiență seamless.
Latency Breakdown
Network (caller)
~30ms
Audio capture
~20ms
STT processing
~50ms
LLM response
~60ms
TTS synthesis
~30ms
Network (return)
~30ms
Total End-to-End Latency~220ms
Optimization Techniques
🌐 Network Layer
- ✓ Edge deployment (PoP near users)
- ✓ Direct peering with carriers
- ✓ UDP for media (not TCP)
- ✓ Minimize hops
- ✓ Geographic load balancing
🎤 Audio Layer
- ✓ Small audio frames (20ms)
- ✓ Adaptive jitter buffer
- ✓ Early packet processing
- ✓ Streaming STT (not batch)
- ✓ Streaming TTS output
🧠 AI Layer
- ✓ Streaming LLM responses
- ✓ Speculative execution
- ✓ Model quantization
- ✓ GPU acceleration
- ✓ Parallel processing
⚙️ Infrastructure
- ✓ Connection pooling
- ✓ Warm containers
- ✓ In-memory caching
- ✓ Async operations
- ✓ Zero-copy audio
Streaming Architecture
// Streaming pipeline - process audio as it arrives
async function processAudioStream(audioStream) {
// Stream 1: Audio → STT (streaming)
const transcriptStream = stt.streamingRecognize(audioStream);
// Stream 2: Transcript → LLM (streaming)
const responseStream = llm.streamCompletion(transcriptStream);
// Stream 3: Response → TTS (streaming)
const audioResponse = tts.streamSynthesize(responseStream);
// Start playing audio as soon as first chunk ready
// Don't wait for full response!
return audioResponse;
}
// Result: User hears response starting ~150ms after speaking
// vs ~800ms+ with batch processing❌ Batch Processing
Wait for user to finish → Process all → Return all
~800ms+ latency
✓ Stream Processing
Process chunks as they arrive → Return immediately
~150ms latency
Edge Deployment
Global PoP Locations
București
12ms
Frankfurt
25ms
Amsterdam
28ms
London
32ms
Paris
30ms
Warsaw
22ms
Voice AI processing runs at nearest PoP to minimize round-trip time.
Jitter Buffer Tuning
Static Buffer
Fixed delay, predictable but may be too much or too little.
Buffer size:60ms (fixed)
Added latency:60ms (constant)
Adaptive Buffer (Recommended)
Adjusts based on network conditions. Lower latency on good networks.
Min buffer:20ms
Max buffer:200ms
Current (avg):35ms
Latency Monitoring
142ms
P50 Latency
198ms
P95 Latency
287ms
P99 Latency
94%
<200ms Target
Latency Impact on Experience
<150ms
Excellent
Feels like real-time, natural conversation
150-200ms
Good
Slight delay, still comfortable
200-300ms
Acceptable
Noticeable delay, may cause interruptions
>300ms
Poor
Frustrating, users talk over each other