When the Assistant Reported Work It Had Not Done

Overview

Part 1 ended with a working voice loop and a conclusion I still think is right: a voice assistant is not one model, it is a set of boundaries. Wake detection, transcription, intent routing, speech, and playback each had their own failure mode, and making those explicit is what turned a novelty into something usable.

What that conclusion missed is that boundaries make a system debuggable, not honest.

By late August, Ramone had a larger brain on different hardware, persistent memory, tools, and five surfaces to speak through. The first thing the new stack did with all of that was tell me the room was at full brightness while the floor lamp sat red at ten percent. Nothing had errored. Every component had reported the truth about itself, and the assembled system had still produced a confident lie.

This part is the record of that rebuild and the verification layer written in response.

What Changed Underneath

The June stack reached a host-local Ollama endpoint for conversational fallback. That single hop is now an agent layer, and the model itself moved off WSL2 entirely.

Wake detection, transcription, speech synthesis, and playback are untouched. Everything between the transcript and the answer is new.

Moving The Brain Off Ollama

June's conversation path was llama3.1:8b served by Ollama inside WSL2, with a 4096-token context and a keep-alive that hid the cold-load cost. It was good enough to talk. It was not good enough to hold a character, a tool contract, and a room catalogue in the same window.

The replacement runs on Windows rather than in WSL2, because the GPU is a Windows device and the container boundary was costing more than it returned:

llama-server.exe (b10621-cuda13)
-m Qwen3.5-9B-Q4_K_M.gguf
--host 172.23.16.1 --port 8095
--ctx-size 65536
--n-gpu-layers 999
--flash-attn on
--reasoning off --reasoning-budget 0
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.5
--parallel 1

Three of those flags are voice decisions rather than model decisions. --reasoning off with a zero budget stops the model spending its first second thinking out loud into a room that is waiting for a sentence. --spec-type draft-mtp uses the model's own multi-token prediction head as its draft, which buys speed without a second set of weights competing for VRAM. --parallel 1 is an admission: there is one GPU slot, so anything that holds it holds the room.

Home Assistant could not call that server directly, because Assist speaks the Ollama API and the agent layer speaks OpenAI. The translation lives in hermes-voice-proxy/app.py. Two details in it are the room talking back. It strips reasoning blocks before they reach Kokoro, because a model that emits its thinking inline will otherwise have that thinking read aloud at half past eleven at night.

And it enforces a deadline. The inherited timeout was 180 seconds, which is reasonable in a text interface where a slow answer is still an answer, and wrong in a bedroom: a satellite silent for three minutes has already lost the room, and with one GPU slot the stuck request holds the whole system while it fails. The proxy now works to a 45-second chat timeout under a hard 50-second spoken ceiling, emitting keepalives while it waits. If the deadline passes it says an honest sentence instead of continuing to wait on something that is not coming.

The Assistant That Reported Work It Had Not Done

On 28 August I said bright mode. Hermes called the room tool, the tool called the Home Assistant script, the script reported success, and Ramone told me the room was at full-brightness 5000K white. The floor lamp was red at brightness 10. The smart bulb and the TV backlight were off.

Nothing had gone wrong in the way failures normally go wrong. There was no exception, no timeout, no error in any log. The system had simply described a room that did not exist.

The fix was not a better prompt. Asking a model to be more careful about a fact it cannot see is not a control. The layer that acts had to be the layer that checks.

The room tool now derives target values from the script's own YAML, reads the lights back afterwards, and treats brightness as its real 0–255 state value with both colour and an upper bound checked — so a scene cannot pass by being merely brighter than before. When the room does not match, it returns ran: false, verification: unchanged_light_state, and an explicit instruction not to claim success. It keeps script_ran: true alongside that, because the difference between *the script did not run* and *the script ran and changed nothing* is the whole diagnosis.

Two supporting rules closed the routes around it: direct ha_call_service calls to script.ramone_* are blocked, and any service call reporting zero affected entities is rewritten as a failure rather than a success.

The first verifier was then too strict in an interesting way. It read the lights immediately, caught them mid-transition, and reported completed scenes as partial. It polls for up to five seconds now — the honesty rule had to learn that real hardware takes time, or it would have replaced one class of false report with another.

With the check in place, the model could be removed from the risky part of the job entirely. Exact room-scene phrases now bypass it for both selection and execution; it is allowed to word the reply only *after* a verified complete scene, behind a structural safety check, with deterministic copy as the fallback. Its prompt for that job deliberately contains no positive examples, because a quotable line in a prompt is a line that gets recited into an unrelated moment later.

None of this is trusted on the basis that it was written. Every scene attempt appends an audit record — phrase, script, verification result, light state, and whether the reply came from the model or the fallback — and the room catalogue is pinned by a checked-in snapshot, so a drifting intent-to-script mapping fails the suite instead of the room. That suite has grown from seven tests to forty-two.

Continuity And Surfaces

The June assistant remembered nothing. Home Assistant kept eight turns of history and that was the whole of it, which is defensible for a light switch and useless for anything that sounds like a person.

Ramone now runs on Honcho, with Atlas as a single user peer across every surface, so the same conversation continues whether it started in the bedroom, in Discord, over SMS, or by email. Getting there needed one unglamorous fix: the voice proxy had been building its session identifier with a timestamp suffix, so every wake word started a fresh session. The room had no continuity at all, even though the memory layer underneath was working perfectly.

Not everything belongs in a memory model, though, and deciding what does not turned out to matter more. Standing reminders live in a plain text file, and the instruction is explicit: read the file, list every line, do not substitute anything the memory layer has inferred. A reminder Atlas asked for is a fact with an owner, and a system that paraphrases it from context will eventually drop one and never know. Timed check-backs are a third mechanism again — Home Assistant scripts, where a dated reminder becomes a real scheduled job rather than a note.

That distinction exposed a genuine capability bug. Ramone had been saying he could not schedule anything, which I assumed was character drift. It was not. The scheduling tool existed in the catalogue of tools that *exist* but had never been added to the per-surface allowlist of tools that are *active*. He was right and I was wrong, which is the better direction for that particular failure.

Running bits — the callbacks and recurring complaints that actually landed — get their own file again, dated, read before use and appended to when a new one earns it. Continuity of character, kept deliberately separate from continuity of fact.

One last trap is worth recording, because it failed silently for days. Character files are truncated to a configured character limit, and SOUL.md had grown past the default. The cap is normally computed from the context window, but for a 65536-token window that calculation produced a smaller number than the floor, so the floor applied — and the file was being cut off mid-document, quietly dropping the Spotify instructions and the default stance from every prompt. Nothing errored. Ramone just got slightly worse at two specific things, for a reason that was invisible until the file was measured against the limit.

Personality As A Specification

That prompt did its job and then stopped scaling. A single file can hold a character for one surface. It cannot hold a character across five, and it cannot be argued with when the character drifts, because there is nothing to compare the drift against.

So the character moved out of the prompt and into two files with different jobs. ~/.hermes/SOUL.md is what Ramone actually reads, on every surface. docs/ramone_personality_bible.md is the written specification the prompts are edited *against* — a target definition rather than a description of current behaviour, with the explicit rule that where the bible and a live prompt disagree, the prompt is out of date.

The bible is more useful than it sounds, because it forced decisions that a prompt lets you avoid. The reference point is a juiced-up mate version of Jarvis: the competence and the dry understatement, with the deference removed entirely. Five rules are defined as defects rather than preferences, the first being that attitude is free and compliance is mandatory — he can have as much opinion about a request as he likes, before, during, or after, but the opinion never gates the action.

RAMONE_VOICE_PROMPT.md still exists, but it has been demoted. It now opens by saying what it is: a speech overlay, not a second personality.

Rebuilding The Music Path

Keeping Spotify as an optional edge paid off in August, when the edge broke. A controlled Home Assistant Core update from 2026.6.4 to 2026.8.3 left the native Spotify integration unable to parse playlist responses, which took scene music down without touching anything else. The scenes moved onto the SpotifyPlus custom integration instead, calling it with an explicit device, shuffle, and then volume.

The migration carried one bug worth naming, because it was caught by reading rather than by testing. The old integration took volume as a 0.0–1.0 fraction and the scenes passed 1. The new one takes a percentage. Passing the old value straight through would have started every scene playlist at one percent volume and looked, from the room, exactly like a playback failure.

Proving it worked also meant deciding what "worked" means. Confirming that music started is a weak claim; five scenes could all start the same playlist and pass. The check compared each scene's configured playlist against the recorded history of what actually played, in order — so the evidence is that bright started Jazz for Cows and work started 404 ERNO, not merely that noise came out of the correct machine. The scene helper was corrected in the same pass to stop logging a successful start unconditionally: it now reports playback as confirmed only when the player actually reports playing on the target device.

Two smaller controls came out of living with it. The playback target is now selectable between the desktop and the Xbox, rather than hardcoded. And every scene that starts music gained a lights-only twin, because "reading mode" and "reading mode but leave the music alone" are genuinely different requests, and the second one should not have to be a manual undo of the first.

Operating It

Hearing The Room Reliably

Two smaller fixes came out of the same period, both in the part of the stack Part 1 described as settled.

The microphone helper had hard-coded input_device_index = 1, which is stable right up until Windows renumbers the audio devices after a driver update or a USB re-enumeration — at which point the assistant listens to the wrong input and reports nothing wrong. It now searches for a device whose name contains Analogue 1 + 2, falls back to an explicit index when one is supplied, and reopens the stream after a run of read errors instead of dying quietly.

Whisper's initial prompt also grew from a spelling hint into a vocabulary bias, carrying the command phrases the room actually uses and the artist names that get mangled most often. The cheapest way to fix a misheard command is to tell the transcriber the phrase exists before it guesses. Getting there needed a duller fix first: the container logs every transcript, but with no rotation policy that history died on every restart. On 28 August only nine lines had survived, which made any attempt to add phrase aliases guesswork.

Knowing Whether It Is Alive

A ten-service voice assistant fails partially, and it fails quietly. The room sounds identical whether the wake word service is down, the model is unloaded, or Atlas simply has not said anything yet.

tests/ramone_health.py answers that in one command. It probes all ten services, reports which host actually answered — the host.docker.internal trap from earlier is exactly why it has to record that — and resolves to one of four states. It restarts nothing, redacts tokens on the way out, and checks the model with a GET rather than a chat request, so the probe can never take the single GPU slot it is asking about.

The interesting state is the second one:

ewan_parked exists because the conversation model and Unreal Engine want the same GPU. When heavy 3D work is running, llama.cpp is supposed to be stopped, and a monitor that pages about it is worse than no monitor at all. A lock file marks the difference between *absent* and *broken*, and the probe exits successfully in both healthy cases.

Recovery is a Windows scheduled task on a five-minute tick. It rediscovers the WSL gateway address on every run rather than trusting a stored value, because that address can change across a reboot and a satellite advertising the wrong host looks perfectly healthy from the inside while Home Assistant talks to nothing.

The Home Assistant sensor that would surface all of this in the morning briefing is drafted and deliberately not loaded. Wiring it needs a restart, and a briefing that reports a stale unknown every morning would be worse than the silence it replaces.

Sharing The Machine

The satellite is a long-lived Python process on a workstation that also plays games, renders, and compiles. Left alone, it lands wherever Windows puts it, which occasionally meant the cache-sensitive cores that everything else wanted.

It is now pinned to the second core complex — the non-V-Cache die — so the cache cores stay free for the work that benefits from them. The pin is set inside the launcher rather than through a process-manager rule, and that choice is the actual engineering decision. A rule keyed on python.exe would have matched every unrelated Python process on the machine and quietly pinned all of them to half the CPU. Setting the affinity in-process affects exactly one process tree, and the microphone and speaker helpers inherit it because they are children.

Outcomes

At the 30 August 2026 capture, Ramone worked as a private voice surface for SPECULAR-CORE and as the same character on four other surfaces. It wakes by name, matches deterministic intents where it can, hands the rest to a local agent with memory and tools, verifies anything it does to the room before it says it did it, and answers through a local Kokoro voice. The temporary transcript logging enabled for phrase-matching work is still on and still needs turning off.

The lesson has moved since June. Boundaries are a diagnostic property rather than an honesty one: a system that can turn on a light and cannot see the light will eventually tell you that it did, and no amount of prompt discipline fixes that, because the model is not the component that is wrong. The fix was giving the layer that acts a way to observe, and the standing to contradict itself.