Building a Private Room-Scale Voice Assistant from Local Parts

Overview

I built Ramone Voice Assistant to turn SPECULAR-CORE into a room-scale spoken system rather than another browser tab waiting for input. The goal was simple on paper: say Ramone, have the machine hear me through the Focusrite microphone, route speech through Home Assistant, answer in the bm_daniel Kokoro voice, and control the room without sending the core voice loop to a cloud assistant.

The actual work was a chain of boundary problems. Docker lived in WSL2. The microphone and speaker lived on Windows. Home Assistant lived in a container. Ollama lived on the host. Wake detection needed .tflite models, not spelling changes. The satellite needed to speak Wyoming cleanly enough for Home Assistant to trust it. Every piece worked alone before the whole system worked as a voice assistant.

The recorded June 2026 stack ran from L:\ramone-voice. That was deliberate. I wanted the project state off the C drive so Windows updates, reinstalls, or OS experiments would not scatter the assistant across user-profile folders. The remaining machine-local dependencies are explicit: Python, pyaudio, wyoming-satellite, Docker, WSL2, Focusrite device ordering, and the startup script that launches the Windows satellite.

The working path captured at the end of the June build was:

You say "Ramone"
-> Focusrite microphone captured by Windows Python helper
-> OpenWakeWord detects the custom ramone model
-> Wyoming satellite streams speech to Home Assistant
-> Faster Whisper transcribes the command
-> Local intents, scripts, or Ollama handle the request
-> Kokoro generates speech as bm_daniel
-> Windows speaker helper plays the response

This case study is the build record for that path: what ran, what failed, what fixed it, and why the captured shape was stable enough for repeated local use at the end of the build.

System Architecture

In the June 2026 snapshot, Ramone was split across Docker, Windows, WSL2, Home Assistant, and local model serving. The split followed the hardware boundary: Docker hosted Home Assistant, Whisper, OpenWakeWord, MQTT, and Kokoro; Windows owned the Focusrite interface and speaker routing; and Home Assistant used a direct host-reachable Ollama endpoint for conversational fallback.

The Docker services sit on the ramone-net bridge network. Home Assistant calls whisper:10300, openwakeword:10400, and wyoming-kokoro:10200 by service name. The Windows satellite advertises itself back to Home Assistant over the WSL gateway, the Windows-side address detected by wsl ip -4 route show.

That split across two operating systems is the recurring theme of this build. No single hostname reaches everything, so every boundary has to be crossed deliberately.

Container boundary

The Docker stack is defined in L:\ramone-voice\docker-compose.yml. It exposes the operational ports directly because I wanted every service to be testable from PowerShell while the pipeline was still unstable. That mattered during debugging; I could query Docker logs, probe individual ports, and restart one service without dismantling the entire voice path.

The OpenWakeWord command loads custom models from L:\ramone-voice\custom-wake-words:

--custom-model-dir /custom
--preload-model okay_nabu
--threshold 0.07
--trigger-level 1
--refractory-seconds 4.0

The low threshold is intentional but not final dogma. It made the custom Ramone model usable from more than one position in the room. The mic gain later moved back to 1.4x because reach and false wake risk are the same control surface wearing different hats.

Windows boundary

The satellite starts from L:\ramone-voice\windows\start-ramone-satellite.ps1. It launches a hidden PowerShell process, redirects logs into L:\ramone-voice\logs, and runs run-ramone.ps1, which assembles the actual Python command.

The microphone helper, ramone-mic.py, captures:

16000 Hz
16-bit PCM
mono
1024-frame chunks
input_device_index = 1

The speaker helper, ramone-snd.py, plays:

22050 Hz
16-bit PCM
mono
device-index -1

-1 matters. It routes to the Windows default output device, which is the only consistently audible target across speakers and headphones. A named device can open cleanly and still be silent if Windows routes the audio elsewhere.

Assistant boundary

The Home Assistant Assist pipeline captured for this build was named Ramone. It used stt.faster_whisper, tts.openai backed by Kokoro, tts_voice = bm_daniel, wake_word_id = ramone, and conversation.ramone for fallback reasoning.

The conversation layer in this snapshot was intentionally smaller than the wider SPECULAR-CORE model fleet. The recorded conversation model was llama3.1:8b with:

keep_alive = 3600
max_history = 8
num_ctx = 4096
think = false

That is enough context to talk without turning a light command into a lecture. It also keeps the model warm after startup, which avoids the first-turn cold load that made larger models feel broken in a voice interface. It was also the ceiling that eventually forced the rebuild in Part 2.

Phase I — Service Boundaries And Tunnels

The first external symptom was not an audio problem. It was Cloudflare Error 1033 on ha.atlas-systems.uk, while a separate tunnel named ramone was already running for the website. The names were close enough to invite the wrong fix: repointing or merging tunnels.

The fix was architectural rather than clever. The homeassistant tunnel stayed responsible for ha.atlas-systems.uk. The ramone tunnel stayed responsible for the website. Tunnel health was checked by name and UUID, not by guesswork.

The same boundary rule carried into the Windows service work. A helper script attempted to install a dedicated CloudflaredHomeAssistant service, but Windows sc syntax failed and left only the generic Cloudflared service running. I did not route the website tunnel through the Home Assistant tunnel as a shortcut. I kept the deployment boundary intact and debugged each service on its own terms.

Windows Satellite

The satellite is the most important custom piece in the build because it owns the real audio hardware. Home Assistant can discover Wyoming services and Docker can host speech engines, but the room hears and speaks through Windows. The bridge had to be stable enough to run continuously, survive HA reconnects, and recover after failed speech recognition.

The launcher stack is split into small files:

Dynamic WSL networking

WSL2 can change its internal address after restart. Hard-coding the gateway would make the assistant brittle after a reboot, so run-ramone.ps1 reads the active WSL route table and advertises the current Windows gateway to Home Assistant through zeroconf.

That detail is easy to miss because the assistant feels like one device. It is not. It is a Windows process pretending to be a network satellite for a Linux container. If the advertised host is wrong, the satellite looks healthy while Home Assistant talks to nothing.

Log-first operation

The satellite writes an output and an error log into L:\ramone-voice\logs, and the error log was the primary truth source for the whole build. Its most valuable property was not visibility but sequencing: a healthy standby shows Waiting for wake word, and a good interaction shows Streaming audio, a transcript event, a synthesize event, and the output device opening. When one of those is missing, the failing stage names itself. That stopped a lot of guesswork.

Phase II — Wyoming Disconnect Loop

The first version of the satellite reached Home Assistant, then disconnected repeatedly. The log loop was clean enough to be deceptive:

INFO:root:Waiting for wake word
DEBUG:root:Server disconnected
DEBUG:root:Server set: ...
INFO:root:Waiting for wake word

The assistant looked alive, but HA did not keep a stable session with it. Wake detection could work in isolated tests while the full voice loop still failed.

I fixed it by wrapping the installed package instead of modifying Python site-packages. wyoming_satellite_compat.py imports wyoming_satellite.__main__, patches selected handlers, and runs the original main() entrypoint. The wrapper adds ping to pong support, which stopped the repeated disconnect loop.

The same file also fixed a second state problem. If STT returned stt-no-text-recognized, the satellite could remain in a streaming state and fail to rearm wake detection cleanly. The wrapper now catches that error, stops streaming, sends wake detection again, and logs Waiting for wake word.

This became the pattern for the rest of the build: do not patch dependency internals when a controlled wrapper can isolate the local behavior. The assistant is easier to rebuild because the custom behavior lives in L:\ramone-voice\windows, not inside a mutable Python install.

Speech Pipeline

The speech pipeline has three independent jobs: detect wake, transcribe speech, and synthesize the answer. Keeping them independent made the system debuggable. A failure in one service did not require rebuilding the others.

Faster Whisper STT

The STT service uses rhasspy/wyoming-whisper with base-int8 and English forced:

--model base-int8
--language en
--initial-prompt "Ramone is spelled R-A-M-O-N-E and pronounced ruh-MOHN."

The initial prompt exists because the wake word and the assistant name are phonetically awkward. Whisper heard Ramon, Rah-mone, and other variants during testing. The prompt does not improve wake detection, but it improves the text after wake once audio is already flowing into STT.

Kokoro TTS

The TTS path uses Kokoro FastAPI behind a Wyoming bridge that exposes an OpenAI-style TTS interface to Home Assistant. That is why the HA integration reports tts.openai even though the speech engine is local Kokoro.

The voice used in this snapshot was bm_daniel. Earlier testing used OpenAI-style names such as onyx because the TTS route was still being proven. Once the Kokoro bridge advertised bm_daniel correctly, the Assist pipeline moved back to the intended voice.

Conversation fallback

The first goal was not to make Ramone chatty. It was to stop Ramone from being absurd. When the fallback model saw too much generic Home Assistant context, it explained logs, APIs, Alexa-like devices, and fictional associations instead of acting like a voice system.

In June the answer was a small model and a narrow prompt. llama3.2:3b moved to llama3.1:8b once timing tests showed the difference was acceptable warm: about 32s cold, which is unusable in a room, against about 0.79s warm, which is fine. The rest of that story — why an 8B model in a 4096-token window stopped being enough, and what replaced it — is Part 2.

Phase III — Wake Word Training

The wake-word problem looked like a spelling problem until it was tested properly. okay_nabu detected reliably. ramone did not, even when STT and the prompt understood Rah-mone, ruh moan, and ruh-MOHN.

The first training config used broad target phrases:

Ramone
ruh moan
rah moan
ruh mone
rah mone
ray moan

The local pipeline generated synthetic positives, adversarial negatives, augmented data, trained a classifier, and exported ONNX. The RTX 5070 accelerated the positive clip generation well, but the adversarial negative generation hit CUDA error: device not ready. Once PyTorch poisoned the CUDA context, the pipeline slowed dramatically and could not be trusted.

The CPU fallback was reliable but slow:

CUDA_VISIBLE_DEVICES=

The GPU retry used:

CUDA_VISIBLE_DEVICES=0
CUDA_LAUNCH_BLOCKING=1

That made the failure surface synchronously, but the practical fix came from a cleaner model source. A downloaded ruh_mown .tflite and .onnx pair was installed into L:\ramone-voice\custom-wake-words and renamed into the shape Home Assistant expected:

ramone.tflite
ramone.onnx

Direct testing then proved the model:

WAKE DETECTED: ramone
Detected wake words: ramone

The wake stack still keeps okay_nabu in the Docker preload list for service readiness, but the active satellite listens for ramone.

Phase IV — Audio Routing And Self Triggering

Once wake detection worked, the assistant still failed in a more irritating way: it heard, transcribed, and synthesized, but the room stayed silent. The logs showed audio opening and closing, so the TTS service was not the cause.

The fix was to standardize on -SoundDeviceIndex -1. That routes output through the active Windows default device, which made the wake tone and spoken response audible through speakers and headphones.

Then the assistant started hearing itself. Kokoro speech through the room speakers could retrigger OpenWakeWord or leak into the next STT capture. The logs made this visible through unexpected transcripts such as wake phrase fragments and responses being interpreted as new user input.

The compatibility wrapper now guards three periods:

POST_TTS_WAKE_SUPPRESS_SECONDS = 3.0
POST_TRANSCRIPT_WAKE_SUPPRESS_SECONDS = 3.0
TTS_MAX_WAKE_SUPPRESS_SECONDS = 20.0
POST_WAKE_AUDIO_DROP_SECONDS = 0.75

POST_WAKE_AUDIO_DROP_SECONDS drops the first 0.75s of microphone audio after wake detection. That stops the wake phrase itself from becoming the user request. The TTS suppression windows ignore wake detections while the assistant is speaking or while audio is still settling in the room.

Home Control And Routine Layer

The assistant became useful when common actions moved out of the LLM and into Home Assistant intents. Local intents are faster, more reliable, and easier to test than asking a model to infer every device action from natural language.

The phrase grammar lives in:

L:\ramone-voice\config\custom_sentences\en\ramone.yaml

The intent responses live in:

L:\ramone-voice\config\intents.yaml

The actual actions live in:

L:\ramone-voice\config\scripts.yaml

That gives the system a clean path:

spoken phrase -> custom sentence -> intent_script -> script -> spoken response

Lighting routines

The core routines control three main light entities:

light.floor_lamp
light.smart_led_bulb
light.tv_backlight_3_lite

The named scenes are intentionally human:

The intent responses carry Ramone's voice without asking the LLM to improvise:

Work mode set. The room is pretending discipline exists.
Bright mode set. Subtle as a dental lamp.
Reading mode set. Very scholarly. Suspiciously so.

Those lines matter more than they look. They make the system feel like the same character whether the response came from a deterministic Home Assistant intent or from the Ollama fallback.

Spotify guards

Spotify became a separate integration problem. OAuth, account restrictions, developer app settings, and device availability all created failure modes that were outside the core voice loop. The scripts captured in this build did not assume Spotify could always play. They inspect media_player.spotify_atlas supported features before calling play, pause, skip, previous track, or volume changes.

That means the spoken response can be honest:

Spotify is connected, but Home Assistant cannot start playback right now.
Open Spotify on your PC or phone and start one song first.

The important decision was not to block the whole assistant on Spotify. Lights, weather, system status, TTS, STT, and chat all work without it. Spotify is a controlled edge, not a dependency for the voice core.

System and weather

System status reads HASS.Agent sensors for SPECULAR-CORE CPU and RAM. Weather reads weather.forecast_home. Both were moved into local intents because they are simple factual queries that should not wake a model unless the user asks for a broader conversation.

The system status response is deliberately short:

CPU 42%, RAM 61% - busy, but not on fire.

The result is a voice interface that can answer fast for operational queries and still talk when the request is actually conversational.

Phase V — Personality And Model Behavior

The first Ramone prompt solved latency by making the assistant terse. It also made him dull. The next attempt made him more open, but that exposed a different problem: models like to explain the wrong thing when they are given ambiguous transcript debris.

The failure mode was obvious in spoken output. Ramone explained Alexa-like devices, fictional associations, logs, APIs, and even scolded rough language as if the room were a corporate demo.

The canonical prompt moved into L:\ramone-voice\RAMONE_VOICE_PROMPT.md. The Home Assistant Ollama entry used for this snapshot copied that text into core.config_entries, with HA stopped before storage edits so it did not overwrite the change.

The recorded prompt kept several constraints:

• write the spoken name as Rah-mone for TTS pronunciation
• answer smart-home commands in one or two short sentences
• answer real conversation in two to five short sentences
• ask one follow-up when it helps
• never scold profanity in a private environment
• never explain logs, API tools, Alexa, Echo, Disney, Cars, musicians, or hidden functions
• treat freaky time, relaxing mode, gaming mode, and reading mode as scene names

The model then moved to llama3.1:8b. A warm local test returned in about 0.79s, which was acceptable for a voice fallback. The cold load was much slower, so the Home Assistant config keeps the model alive for 3600 seconds.

A single prompt file was the right size for this build and the wrong size for the next one. It can hold a character for one surface; it cannot hold one across several, and it cannot be argued with when the character drifts, because there is nothing to compare the drift against. Part 2 covers where the character went.

Calibration And Portability

Wake Range And Room Calibration

Wake range had to be measured in the room, not imagined from the desk. The assistant detected well at close range, but couch and bed pickup changed the audio level enough to require a test harness.

I added test-ramone-room-position.ps1. It stops the satellite, asks for a physical position, tests multiple mic multipliers, saves one WAV per attempt, and restarts the satellite afterwards. The couch test produced:

The strange result is the useful one. More gain did not always mean better detection. At 2.2x, the model became less predictable from the couch. The live default moved to 1.4x because it detected from the couch without pushing the wake pipeline into the odd behavior seen at higher amplification.

The satellite state recorded during the room tests was:

mic_volume_multiplier = 1.4
wake_word_name = ['ramone']
sound device = default

If false wakes continue, the next clean adjustment is raising the OpenWakeWord threshold from 0.07 toward 0.10. I did not make that move first because lowering mic gain solved the immediate sensitivity problem without reducing legitimate couch pickup.

Startup And Portability

The project now lives under L:\ramone-voice, which makes the build more portable than a typical Windows voice experiment. The Docker services, custom wake models, prompt, Home Assistant config, logs, scripts, and training notes live together. That matters because the assistant crosses OS boundaries. If the files are scattered, rebuild time becomes archaeology.

Boot startup came through the existing SPECULAR_BOOT.bat, which gained a block for the Docker stack and the Windows satellite. Adding it exposed an old Windows batch problem worth one sentence: unescaped >> inside display text is a redirection, not a decoration, and it had been quietly creating a junk file named ] on every boot.

The assistant still depends on machine-local pieces the repo cannot carry — Python and pyaudio, the Wyoming packages, the Focusrite driver and its USB device ordering, Docker inside WSL2, the Cloudflared credentials, and Spotify's OAuth state. The difference is that the boundaries are documented. A reinstall does not require rediscovering why ramone-snd.py uses -1, why Home Assistant sees a satellite at a WSL gateway address, or why .storage backups must not be published.

Outcomes

At the end of the June 2026 build, Ramone worked as a private voice surface for SPECULAR-CORE. The system wakes by name, hears speech through the Focusrite input, routes requests through Home Assistant, controls room lighting and media scripts, answers through a local Kokoro voice, and falls back to a bounded Ollama personality when the request is conversational rather than operational.

The lesson from this half is that useful voice assistants are not one model. They are boundaries. Wake detection, STT, intent routing, TTS, playback, personality, and device control each need their own failure model; once those boundaries are explicit, the assistant stops being a novelty and becomes another local system service. Part 2 is about what that conclusion missed.