Voice — talking to Sapphire and hearing her answer

What it is

Sapphire has a full voice stack: speech-to-text (STT) for your voice in, text-to-speech (TTS) for her voice out, a wake word so she answers to her name, and a hands-free Conversation mode with real interruptions. Everything runs locally by default; cloud providers are opt-in. This guide covers all of it — from holding the mic button to tuning echo cancellation.

Using it

The four ways to voice

All four share the same STT and TTS providers, the same transcription cleanup, and the same privacy gates.

Voice input — STT

Pick a provider in Settings → STT:

Two support layers clean up what the mic hears:

• Voice activity detection (VAD) — Silero VAD (a tiny ML model, robust to background noise) decides speech vs. silence. If Silero is off or unavailable, a Classic VAD fallback uses amplitude percentile thresholding. The STT tab shows a live Silero status badge and a Test my voice (5s) button that scores your mic and suggests a threshold.
• Hallucination filter — Whisper invents canned phrases on silence ("Thanks for watching", "[music]"...). A shared filter at the provider boundary drops them for every voice lane, so a 3am noise doesn't become a conversation.

Hold-to-record specifics: the browser records while you hold 🎤, releasing sends the audio to the server for transcription, and the transcribed text is sent as your message. Leaving the button or hiding the tab stops the recording safely.

Voice output — TTS

Pick a provider in Settings → TTS:

Plugin providers appear in the dropdown automatically when their plugin is enabled. The Test TTS button at the bottom of the tab speaks a test line and reports latency.

Voice, pitch, and speed are per-chat, not global. They live in the chat sidebar under the TTS (Voice) accordion — a voice dropdown plus Pitch and Speed sliders. Each chat (and each persona) remembers its own; switching chats switches her voice. Defaults: voice Heart (af_heart), pitch 0.98, speed 1.3.

• Speed is applied at synthesis time (providers clamp to their supported range).
• Pitch is native for Kokoro and Piper. For providers without native pitch it's applied by resampling the audio afterward — which also changes duration, so keep it near 1.0 there.

Where her voice comes out:

• Web chats — audio plays in your browser. Every assistant message also has a replay action to speak it again; while TTS is playing, the mic button becomes a ⏹ stop button.
• Wake word turns — audio plays on the server's speakers.
• Conversation mode — server speakers (Local mic) or your browser (Browser mic).

Streaming TTS

On by default. With Streaming on (Settings → TTS → Streaming section), Sapphire synthesizes and starts speaking each chunk as the LLM finishes it — speech starts a second or two after generation begins instead of after the whole reply. Only streaming-capable providers do this (Kokoro, Piper); others fall back to whole-reply playback automatically, no harm done.

Tuning lives in the same section: split mode (paragraph keeps prosody natural, sentence gets audio out faster), min/max chunk sizes, pauses after paragraph/sentence boundaries, and how *stage directions* and (parentheticals) are rendered as pauses.

Streaming TTS is also what gives Conversation mode its voice — see below.

The mic button and flyout

The composer's mic is a split button: [🎤 | ▾].

• 🎤 — hold to record (release to send). If TTS is playing it shows ⏹ and a tap stops playback instead. While Conversation mode is on it shows 🎙 and a tap ends the session.
• ▾ — opens the device flyout: a volume slider + mute for this browser, and the Conversation segment with three positions: Off | Local mic | Browser mic.

The flyout is device settings (this machine's audio); the sidebar is chat settings (her voice in this chat). A phone call never lights the mic button — it isn't this device's session.

Wake word

Turn it on in Settings → Wakeword. Sapphire then listens continuously on the server's mic (openWakeWord, ONNX) and answers when she hears the model's phrase.

Models:

• Bundled with Sapphire: hey_sapphire (the default).
• Built into openWakeWord: alexa, hey_mycroft, hey_jarvis, hey_rhasspy, timer, weather.
• Custom: drop .onnx or .tflite files into user/wakeword/models/ — they appear in the dropdown automatically. Community collections exist on GitHub.

Changing the model in Settings hot-swaps the detector — no restart. The threshold (default 0.6) sets sensitivity: lower catches more but false-triggers more.

A wake turn, start to finish: wake word detected → short acknowledgment tone → she records until you stop talking (or aborts after a few seconds of no speech) → transcribe → voice-command check → the reply streams live into any open web page and speaks on the server's speakers. The Stop button on the page works on wake turns too.

Wake word needs a working STT provider — with STT disabled she'll hear her name but can't understand what follows. Detection is suppressed while you're actively using the web UI (recording or mid-chat) and while Conversation mode is running.

Voice commands

The voice-commands plugin (enabled by default) handles instant actions before the LLM ever sees them — the utterance must match exactly:

These are ordinary plugin hooks — other plugins can register their own triggers.

Conversation mode

The headline voice feature: continuous listening with real turns and real interruptions. No wake word per utterance, no button. You talk, she answers, you can talk over her and she stops.

Prerequisites:

Starting and stopping: mic flyout → Conversation → Local mic (the server's microphone and speakers) or Browser mic (this browser over a WebSocket — works from your phone on the couch). Off ends it, as does tapping the green 🎙 mic button. The two sources are mutually exclusive; picking one hands off from the other. The mode is ephemeral — it doesn't survive a restart, and the wake word is restored the moment it ends. If the mic can't be acquired, the mode simply doesn't engage and wakeword listening stays intact — she is never left deaf.

How a turn works. Every audio frame is scored by Silero VAD against CONVERSATION_VAD_THRESHOLD. When speech starts, she accumulates your utterance; when you've been silent for CONVERSATION_ENDPOINT_SILENCE_MS (default 700ms), the utterance ends and goes to STT. Utterances shorter than CONVERSATION_MIN_SPEECH_MS (default 200ms) are discarded as blips — a cough doesn't start a turn. Her reply streams to the page and to the speakers as it generates.

Barge-in. While she's speaking, sustained speech from you for CONVERSATION_BARGE_HOLD_MS (default 90ms) cuts her off — the LLM stream is cancelled, the audio stops, and your new utterance becomes the next turn. Barge-in only arms once her audio is actually flowing: talking during her silent thinking phase (reasoning models, tool calls) won't kill the turn, because there's nothing to interrupt yet.

Start word (optional). Set CONVERSATION_START_WORD (e.g. hey sapphire, sapphire) and she only acts on utterances that begin with one of those phrases — the phrase is stripped and the rest is the message. Unlike the wake word this is matched on the transcription (fuzzy, so "hey staff fire" still works — tune CONVERSATION_START_WORD_FUZZY), and there's no pause: it's a filter, not a trigger. Great for rejecting a TV in the room. Blank = respond to everything.

Echo — the three audio tiers. The classic failure is her hearing herself through your speakers and barging in on her own voice:

Duplex-tier extras (config-only, not in the Settings UI): CONVERSATION_AEC_DELAY_MS aligns the echo reference to your speaker round-trip, CONVERSATION_BARGE_GUARD_MS ignores barge-ins briefly while DTLN locks onto her voice onset, and CONVERSATION_BARGE_RMS_FLOOR rejects quiet residual echo as a false barge.

Tuning lives in Settings → Conversation. All knobs take effect on the next activation (flip Off/On) — no restart:

• Too many false turns from background noise → raise CONVERSATION_VAD_THRESHOLD (0.6–0.7) and/or CONVERSATION_MIN_SPEECH_MS (250–400), or set a start word.
• She cuts you off mid-sentence → raise CONVERSATION_ENDPOINT_SILENCE_MS (900–1200).
• Keyboard clatter interrupts her → raise CONVERSATION_BARGE_HOLD_MS (150–250).
• You can't interrupt her → lower CONVERSATION_BARGE_HOLD_MS, and on open speakers check the DTLN tier is actually loaded (log says using duplex/DTLN-… audio tier).

Phone calls run on the same conversation engine as independent external sessions (up to CONVERSATION_EXTERNAL_SLOTS at once) with their own per-call tuning — they never touch your local mic or wake word. See PHONE-CALLS.md.

Voice privacy

In a private chat (vault open — see PRIVACY.md), cloud voice is refused in both directions: cloud STT would ship your voice audio off-machine, cloud TTS would ship her reply text. The gate reads each provider's is_local flag and fails closed — a provider that can't prove it's local is treated as cloud. Hold-to-record surfaces the refusal as a toast; wake word and conversation turns log it and stay silent. There is deliberately no override toggle: switch to a local provider or type.

Hosted / managed installs

Managed installs have no server-side audio: the Audio and Wakeword tabs are hidden, local-mic recorder knobs disappear, and STT/TTS run through the Sapphire Router providers. Hold-to-record and browser-mic conversation still work — the browser is the microphone.

Settings

Per-chat voice, pitch, and speed are in the chat sidebar (TTS (Voice) accordion), not below — there are no global keys for them.

STT (Settings → STT)

TTS (Settings → TTS)

Wakeword (Settings → Wakeword)

Conversation (Settings → Conversation)

Audio devices (Settings → Audio)

Quick Troubleshooting

• She hears herself and self-interrupts (open speakers) → conversation echo tier → set CONVERSATION_DTLN to 256 or 512 (models must exist in user/models/dtln/), or wear headphones; check the log for "using duplex/DTLN … tier" vs. "falling back to headphone tier".
• Conversation mode replies show on screen but she's silent → Streaming TTS is off or the provider can't stream → enable Settings → TTS → Streaming and use Kokoro or Piper.
• "Local mic" won't turn on → wake word off or mic unavailable → enable Wakeword first (local conversation borrows its mic); check Settings → Audio input device and the log's [CONV] lines. On failure wakeword is restored automatically.
• Can't barge in / interrupts don't register → she may still be "thinking" (barge-in arms only once audio flows) → wait for her voice, then talk over it; if it still won't cut, lower CONVERSATION_BARGE_HOLD_MS.
• Keyboard/cough interrupts her → raise CONVERSATION_BARGE_HOLD_MS (150–250) and CONVERSATION_MIN_SPEECH_MS.
• She cuts you off mid-sentence → raise CONVERSATION_ENDPOINT_SILENCE_MS (900–1200).
• Wake word misses you → lower WAKEWORD_THRESHOLD (0.4–0.5), check the Audio tab input device with 🎙 Test, and confirm the right model is selected. False triggers → raise the threshold.
• Wake tone plays but nothing happens → STT is disabled or still loading → pick a provider in Settings → STT; the mic button tooltip shows "STT loading — downloading speech model" while a model downloads.
• Mic button stuck "loading" / provider switch seemed to work but voice is dead → the provider failed to construct (bad key, missing CUDA, model download failure) and fell back to disabled → check the app log for [stt] Failed to create / [tts] Failed to create, fix the cause, re-select the provider.
• TTS test says "Reachable, but synthesis failed" → the server answers but can't make audio; the reason is shown and is in user/logs/kokoro.log. A voice never used before downloads from Hugging Face on first use (one reach, then cached for good); a box that can't reach it says so in that log at boot.
• Pitch sounds wrong / audio speeds up or slows down with pitch → the provider has no native pitch, so pitch is a resample (changes duration too) → keep pitch near 1.0 on ElevenLabs/gTTS; Kokoro and Piper shift pitch natively.
• No audio in the browser at all, toast about autoplay → the browser blocked audio before your first interaction → click anywhere on the page and retry.
• Voice refused with a "Private chat" message → the vault gate: the active STT or TTS provider isn't local → switch to a local provider (Faster Whisper / Kokoro / Piper) or type. Note plugin providers must declare themselves local in their manifest, or they're treated as cloud.
• Silero badge says unavailable → the VAD model isn't loaded (first-run download or offline) → Classic VAD carries voice detection in the meantime; check the badge again after connectivity.

See also

• PHONE-CALLS.md — telephony: the same conversation engine over a real phone line
• PRIVACY.md — the vault, private chats, and why cloud voice gets refused
• CHATS.md — per-chat settings, including the sidebar voice/pitch/speed
• PERSONAS.md — bundling a voice with a prompt/toolset as a persona
• PLUGINS.md — enabling the Piper / ElevenLabs / gTTS provider plugins
• TROUBLESHOOTING.md — general diagnosis beyond voice

Reference for AI

STT: registry core/stt/providers (faster_whisper local, fireworks_whisper cloud, sapphire_router managed, none). STT_ENABLED/TTS_ENABLED are DERIVED from provider != 'none', not user keys. Silero VAD (core/stt/silerovad.py) with Classic percentile fallback (RECORDER* keys, only active when Silero off/unavailable). Hallucination filter at provider boundary (core/stt/hallucination.py) — all lanes inherit. Routes: POST /api/transcribe (browser hold-to-record; 403 on private-chat cloud-STT gate), GET /api/stt/vad-status, POST /api/stt/vad-test.

TTS: registry core/tts/providers (kokoro local subprocess :5012, sapphire_router managed, none) + plugin providers (piper local, elevenlabs, gtts). 28 Kokoro voices (11 AF, 9 AM, 4 BF, 4 BM). supports_streaming: kokoro + piper only. supports_pitch: kokoro + piper (native); others = client-side resample (duration changes). Voice/pitch/speed are PER-CHAT settings (voice, pitch, speed in chat settings; sidebar "TTS (Voice)" accordion; applied on chat switch via tts.set_voice/set_pitch/set_speed). Defaults af_heart / 0.98 / 1.3; pitch clamped 0.5–2.0. Routes: POST /api/tts (play=server speakers, file=browser blob), /api/tts/stream (SSE chunks; 503 if streaming off → client falls back), /api/tts/test, /api/tts/stop, GET /api/tts/status, /api/tts/voices.

STREAMING TTS: core/tts/streaming.py (chunker) + stream_pump.py. Pump enabled iff TTS_ENABLED && TTS_STREAMING_ENABLED && provider.supports_streaming. Web replies auto-speak in browser (SSE tts_chunk); streaming off = whole-blob after generation. Strips think/reasoning/tools blocks + code fences before synth.

PLAY SPEECH HERE (2026-10-17): TTS_LOCAL_PLAYBACK, Settings > Voice, default on. "Make speech" and "play it on this computer" are two switches. Off = synthesis and every TTS hook unchanged, nothing plays on this machine or in the browser: speak()/speak_sync() return after pre_tts; the pump fires the chunk hooks and emits nothing (counted as voiced, no drop notice); /api/tts file mode answers 204. Lane ownership: the stream owner stamps stream.tts_plays_here (default True = the browser; cadence's device lane and the conversation driver's phone/Discord turns stamp False) and the pump takes plays_here; tts.render(plays_here=True) for the browser route and the MCP persona, default False for devices/Telegram. Not governed: satellites, phone/Discord, Telegram voice notes, voice preview, ding. Also: a plugin that forwarded a chunk sets skip_tts in tts_chunk_audio = consumed (not lost); the browser's whole-blob route now goes through render (gate + cleanup + pre_tts), no longer the raw primitive, and render reasons in QUIET_REASONS are silence on purpose (204), the rest 503. Plugin side: docs/plugin-author/voice-routing.md.

WAKEWORD: core/wakeword/ (openWakeWord). Builtin: alexa, hey_mycroft, hey_jarvis, hey_rhasspy, timer, weather; bundled: hey_sapphire (core/wakeword/models); custom: user/wakeword/models/*.onnx|.tflite auto-discovered (GET /api/settings/wakeword-models). WAKEWORD_MODEL change = hot-swap (reload_wakeword_model), no restart. Wake turn: tone → record (RECORDER_NO_SPEECH_TIMEOUT abort) → transcribe → post_stt hook → process_llm_query(voice_turn=True) → streams to page (VOICE_TURN events) + server speakers. Suppressed during web activity (_web_active) and conversation mode. Requires STT.

VOICE COMMANDS: core/plugins/voice-commands — pre_chat hooks with voice_match, exact match, bypass LLM. stop|halt|be quiet|shut up → stop TTS + cancel; reset|reset chat|clear chat|clear history → clear history.

CONVERSATION MODE ("true speech"): core/conversation/ — engine.py (IDLE/USER_SPEAKING/RESPONDING state machine), driver.py (STT→chat_stream→sink; barge = scoped cancel_generation + sink.stop), manager.py, vad.py (SpeechGate w/ hysteresis: exit threshold = threshold−0.15). Sources: local_source (headphone tier), duplex_source (DTLN-AEC 256/512, one duplex stream = source AND sink, models user/models/dtln/, soft-fallback to headphone), browser_source (WS /ws/conversation). UI: mic flyout conv-seg Off|Local mic|Browser mic (features/convo.js owns transitions; truth via conversation_mode_changed SSE). Routes: PUT/GET /api/runtime/true-speech (local), WS /ws/conversation (browser; refused 4409 if local active). Local requires active wakeword (fail-safe handoff, restored on any failure/exit); browser doesn't. AUDIO REQUIRES streaming TTS + capable provider (driver feeds sink only tts_chunk events). Barge-in arms on first tts_chunk (not content — thinking is uninterruptible by design). Start word = STT-side fuzzy prefix gate (match_start_word), no pause, strips prefix. Tuning read fresh per activation. Settings tab "Conversation" (💬). External sessions (phone) = manager.start_external, CONVERSATION_EXTERNAL_SLOTS cap (default 2), own driver/gate per call, never touches wakeword; one live session per chat. Mode is ephemeral (not persisted).

PRIVACY: core/voice_privacy.py — private chat + non-local provider = refuse (stt_gate_reason/tts_gate_reason). Fail-closed on missing is_local metadata; fail-open only on systemic settings errors. No override toggle by design. Gates: /api/transcribe 403, wake path pre-record, conversation driver per-session, pump first-push, /api/tts file-mode 403. NOTE: piper/gtts/elevenlabs manifests don't declare is_local → all treated as cloud by the gate (piper is factually local — known gap).

MANAGED MODE: Audio + Wakeword settings tabs hidden; recorder keys hidden; sapphire_router STT/TTS providers.

DEVICES: unified device manager; AUDIO_INPUT_DEVICE/AUDIO_OUTPUT_DEVICE null = auto (Linux prefers pipewire/pulse/default). Test buttons on Audio tab.