# Myna Project Plan

**Created:** 2026-06-12 · **Last updated:** 2026-08-10

**Feature 006 (coverage & quality gates, 2026-08-10):** implemented on
integration-220627 — one-command Rust/Python coverage (`workshop run myna
cov` / `py-cov`), instrumented use-case exercise + merged populations and
dead-code report (`exercise`, `deadcode`), self-hosted 80% patch-coverage
gate per PR (`patch-cov`), static battery (fmt / machete / cargo-deny
offline-invariant / ruff / mypy-strict-core / shellcheck / actionlint), weekly
advisory audits, and the spread evaluation (verdict: **adopt**; decision
record + confined-e2e suite + fake-backend test snap in `fake-snap/`,
`spread.yaml`, `tests/spread/`, nightly `.github/workflows/spread.yml` —
first VM/CI validation pending). Spec/tasks: `specs/006-coverage-quality-gates/`.
**Status:** Living document — update task status in place as work lands.

This plan turns IE114 (UbuSTT API), UD129 (Desktop STT Integration), and the
testbed phasing from CLAUDE.md into assignable tasks. Task IDs are stable;
reference them in branches and PRs (e.g. `t02-fake-adapter`).

**IE115 (2026-06-17):** a competing braindump proposing a WebSocket API shaped
on OpenAI's Realtime API has appeared (`IE115-spec.txt`). It does *not* yet
supersede IE114 — it is still a braindump. It happens to ratify the two pivots
we already made against IE114 (audio-push + WebSocket), but imports a lot of
OpenAI speech-to-speech baggage that a local transcriber doesn't want.
Reconciliation is **Workstream F**; our push-backs live in
`docs/IE115-deviations.md`.

**Update (2026-07-01):** Workstream F is resolved. The team decided IE115 will be
a *suitable subset* of OpenAI's Realtime API (compatibility + remote-backend +
industry-contribution reasons), extended with additive events. Our liveness event
and capabilities discovery were adopted; the flat-profile and drop-conversation-
item push-backs were overruled for compatibility; translation is out of scope.
Full mapping in `docs/IE115-resolution.md`; async lifecycle diagrams in
`docs/architecture/ie115-lifecycle.md`. Remaining open: error taxonomy (T31 —
with the recorded requirement that the disposition ride the wire, see the T31
row), backend discovery / model selection (T48), overload-lag signal, GPU
memory pressure. Versioning (T35) is now settled *including* the IE115 dialect
(2026-07-06): the vocabulary is additive on both dialects (unknown events are
ignored, version bumps reserved for semantic changes), IE115 is declared
unversioned-additive, and the server announces its `protocol_version` on the
`session.created` greeting it sends on every connect (`ie115-wire.md` §1).
Next focus shifts to the **orchestrator subsystem** (Charles) and the
inference snap server (Ivano).

**Update (2026-07-26):** the UD136 UX spec (*Ubuntu Desktop Speech-to-text UX*,
Juan Ruitiña, drafting / under review) lands the product direction for the
desktop experience: an accessibility-first keyboard trigger (KEY_DICTATE/
XF86Dictate hardware key preferred, a configurable default combo — Super+J
suggested), OSK / context-menu / app-triggered entry points (the latter two
post-MVP), a state-cue model with visual + **sound** feedback, an explicit
error-state list, and — the big engineering consequence — **transcription
visible as the user speaks**. That reverses the earlier "unstable partials
dropped as confusing" stance: a **streaming mode ships alongside batch mode**,
user-configurable and **gated on hardware tier**. The review thread steers the
display semantics toward **chunks of committed text** shown progressively (not
last-wins rewritten hypotheses), which ratifies the append-only revision
contract T08's design note recommended; the spec body still shows an
uncommitted in-field "hypothesis" with differentiating formatting — contested
in review, to close with design. Also reviewed 2026-07-26: the colleagues'
**canonical/whisper-snap** (Go IE115 adapter + Collabora WhisperLive, snap
packaging in PR #2) — a live IE115 **streaming** peer we can experiment
against before our server grows streaming; findings folded into T08/T63.
Folded in: T08 (streaming, now a requirement), new tasks T58–T62 (D, from
UD136), T63 (F, interop fixture).

Open spec questions (transport, event vocabulary, capabilities discovery,
error model, performance contract) are tracked as tasks in workstream E so
they get owners instead of lingering.

## Workstreams

- **A — Testbed** (phases 0–4): produce the evaluation matrix and
  reference-hardware tiers.
- **B — Inference snap**: an IE114-conformant ASR inference snap, modelled on
  the qwen3/gemma4 snaps in `reference/`.
- **C — Transport**: the real IE114 wire protocol, behind the existing
  abstraction.
- **D — Desktop client**: the UD129 dictation experience, UX specified by
  **UD136** (2026-07-22, under review — implications folded into T58–T62).
  Taken over by Charles (2026-07-07), starting with the audio adapter as a
  Rust crate (T49–T52).
- **E — Spec work**: feed findings back into IE114/UD129.
- **F — IE115 reconciliation**: integrate the IE115 WebSocket proposal where it
  improves on IE114, and document where we deviate from its OpenAI-Realtime
  lineage. Feeds E (it *is* spec work, but kept separate so the IE115 decisions
  get their own owners and don't get lost inside the IE114-update task).
- **G — Orchestrator subsystem (Rust)**: the client-side dictation brain — the
  session + model-residency FSM from `docs/architecture/ie115-lifecycle.md`,
  mediating the audio adapter (D/Charles), the inference snap (B/Ivano), the
  hotkey (T21) and the injector (T22). Every external boundary is a trait with a
  mock, so it is buildable now: the existing Python `myna-server` stands in for
  the inference snap, a WAV source for the audio adapter, stdin for the hotkey,
  stdout for the injector.

Dependencies at a glance: A is unblocked now and feeds E; C unblocks B and D's
integration work; B and D converge in the end-to-end milestone (T22); G consumes
B and D behind interfaces (stubbed until they land) and builds directly against
the lifecycle diagram.

## Status legend

`todo` · `in progress` · `done` · `blocked(<task>)`

---

## Workstream A — Candidate-adapter testbed

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T01 | Scaffolding: package layout, core types, transport abstraction | done | Claude/Charles | — | `myna.core` events/audio/session/transport; ADRs written |
| T02 | Fake adapter + harness skeleton (Phase 0) | done | Claude/Charles | T01 | `uv run python -m myna.testbed` produces a ResultRecord; contract tests in `tests/test_contract.py` pass |
| T03 | Test-fixture audio corpus: short utterances with reference transcripts, per UD129 accuracy matrix (quiet/noise, accents, commands, long-form, technical vocab) | done (synthetic tier) | Claude/Charles | — | `dev/generate_fixtures.py` synthesizes all matrix categories via espeak-ng (CC0, offline, deterministic noise variants); manifest schema + loader in `myna.testbed.corpus`. Real recorded speech is T25 |
| T04 | WAV/file audio source for the harness (batch + real-time pacing) | done | Claude/Charles | T02 | `WavFileSource` plays T03 fixtures; chunk size + realtime pacing configurable; covered in `tests/test_sources_and_corpus.py` |
| T05 | Virtual PipeWire source node feeding fixture audio at real-time rate (Phase 1) | todo | | T04 | Works headless in the Taipei lab; node teardown clean across runs |
| T06 | Accuracy metrics: WER/CER of `done` transcript vs reference, normalization rules documented | done | Claude/Charles | T03, T04 | `myna.testbed.metrics` (`word_error_rate`/`character_error_rate` with S/D/I breakdown + documented NFKC/casefold/punctuation normalization); pure, scoreable from stored records; `tests/test_metrics.py`. Used by `dev/bench.py` |
| T07 | First real adapter, commit-on-finalize (Phase 2): faster-whisper | done | Claude/Charles | T04 | `myna.testbed.whisper.FasterWhisperAdapter`; `whisper` uv extra; integration test skips cleanly without extra/model; `HF_HUB_OFFLINE=1` run verified; CPU default (GPU is explicit, mirroring engine selection) |
| T08 | Streaming adapter (Phase 3): progressive whisper emission plus small transducer snaps | done (whisper/parakeet; sherpa packaged) | Claude/Charles | T07 | Feature 007 shipped the wire contract; feature 008 shipped whisper local-agreement streaming and the Parakeet snap. Parakeet uses configurable SilenceCut committed chunks (`stream-arm-seconds`, `stream-silence-cut-seconds`, `stream-force-cut-seconds`); sherpa packages native partials plus endpoint commits. Remaining: nemotron native loop (008 US2) and the concluding report. Watermarks: `results/streaming-watermarks.json` |
| T09 | Nemotron adapter via NeMo (native streaming transducer), `att_context_size` sweepable | done (commit-on-finalize) | Claude/Charles | T07 | `myna.testbed.nemotron.NemotronAdapter`; `nemotron` extra (`nemo_toolkit[asr]`); `att_context_size` latency dial in the candidate label; `myna-server --adapter nemotron` so `bench.py` measures it over a socket (no snap). **Verified 2026-06-14:** decode works first try (no API iteration needed); real-speech dictation via `dev/dictate.py` is perfect and near-instant. See *Measured findings* — finalize ~0.027s (native transducer wins latency decisively), but synthetic WER is unreliable (espeak is OOD for it). English-only (`non-english-german` 100%). Native frame-at-a-time streaming is the follow-up (converges with T08) |
| T10a | Qwen3-ASR adapter via the **pure-C runtime** (antirez/qwen-asr) through ctypes FFI | done (commit-on-finalize) | Claude/Charles | T07 | `myna.testbed.qwen.QwenCAdapter`; **zero pip deps** (ctypes stdlib + `libqwen_asr.so`, int16→f32 via stdlib `array`) — the parsimonious, multilingual (30 langs), no-GPU runtime. `myna-server --adapter qwen-c --model <dir>`; lib via `QWEN_ASR_LIB`. **Verified 2026-06-15** end-to-end through `run_session` (FFI load+decode correct on real clips, 0% WER) + 16 offline unit tests (`tests/test_qwen_unit.py`). Snapped (qwen-snap `cpu` engine, builds the `.so` from source). Probe (Ryzen 9 7950X3D, optimistic ceiling): offline ~1.4–1.6s warm for 5–6s clips (edge of UD129 target; over budget on a laptop), streaming 1.86x realtime (sub-realtime on weaker CPUs → live streaming is a follow-up, not MVP). Native streaming has a monotonic commit frontier matching our `final` contract when revisited. **GPU "best quality" runtime (vLLM, T10b) is on the `qwen3-vllm-gpu` branch** (verified out-of-confinement, snap parked — see that branch's `docs/qwen-vllm-confinement.md`); proves runtimes are switchable per family via the engine mechanism |
| T11 | Matrix runner + result aggregation (Phase 4): sweep candidates × fixtures × configs, aggregate JSONL into comparison tables | done (single-strategy) | Claude/Charles | T06, T08 | `dev/bench.py` sweeps fixtures against a live socket → `results/bench.jsonl`; `dev/aggregate.py` micro-averages WER/CER + finalize percentiles into a cross-`--label` table (overall + by-category), deduped by (label, clip); `dev/run-matrix.sh` drives engine/model switches → bench → aggregate in one command. **Config-driven multi-backend runner added 2026-07-02:** `dev/matrix.py` + `dev/matrix.yaml` sweep *targets* (whisper/nemotron/qwen) each provisioned on a socket — provisioner `server` (spawns `myna-server` itself: local-first, no snap/sudo) or `snap` (drives an installed snap: engine/model switch + restart). Per target it takes a **cold** sample (model-load-from-cold, `--cold`) then a **warm** sweep, stamps `hardware` provenance onto every record (machine/cpu/gpu/tier, T12), and calls aggregate. New metrics wired through: `time_to_ready` (cold load), `rtf`, p50/p95 finalize; aggregate shows a machine column + distinct cold-load column. Per-`server`-target **peak RSS/VRAM** sampled (psutil + nvidia-smi) into a `matrix-resources.jsonl` sidecar and shown as columns; server logs redirected to `results/matrix-logs/`. Verified locally (no snaps): whisper-base/cpu 9.14% WER / 640 MB, qwen-c/cpu 1.52% / 2.8 GB, nemotron/cuda 0.51% / 1.15 GB VRAM (RTF 0.01, 25 ms finalize, 5.8 s cold load). Covers the model×hardware axis; streaming emission watermarks are recorded in `results/streaming-watermarks.json`. Surfaced + fixed the `en-GB` adapter bug |
| T12 | Hardware tier report from Taipei lab runs | todo | | T11 | Written tiers proposal delivered to IE114/UD129 owners (feeds T19). **Unblocked 2026-07-02:** `dev/matrix.py` now stamps machine/cpu/gpu/tier provenance and captures cold-load + RTF per target, so a lab node just runs `matrix.py` with its `hardware:` filled in and ships the JSONL |
| T25 | Real recorded-speech corpus tier: redistributable human recordings (e.g. LibriSpeech/Common Voice subsets) added to the manifest, covering accents and noise authentically | done (clean+noise tier; accents follow-up) | Claude/Charles | T03 | `dev/fetch_real_corpus.py` downloads LibriSpeech `dev-clean` (CC-BY-4.0), decodes FLAC→16k mono WAV (ffmpeg), and writes `corpus/real/` (schema-v1 manifest, per-clip `source`/`license`, NOTICE for attribution). Selection is deliberately trivial (first N utterances in archive order) — **kept simple at Charles's request**; 12 `quiet` clips + 2 `noise` variants (real speech + seeded SNR-10, reusing the synthetic mixer). **Regenerated, not committed** — gitignored like `fixtures/`; dev takes the ~337 MB download hit (cached under gitignored `.cache/`). **Thesis verified 2026-06-14:** same Nemotron model scores **0.0% WER on the real clips** vs **44.6% on synthetic espeak** (100% on the pangram — empty output) — synthetic WER was indeed misleading; real WER is now trustworthy. Unblocks T12 (tiers) and T19 (perf contract). **Honest scope:** trivial first-N selection means low speaker variety (one speaker at the default N), and dev-clean is clean read speech — so this covers real-voice + noise authentically but **not accents/speaker diversity**; an accent-labelled corpus (Common Voice/VCTK; credentialed/large download) is the follow-up |

## Workstream B — ASR inference snap

Reference material: `reference/qwen3-snap` and `reference/gemma4-snap`
(snapcraft layout, `engines/*/engine.yaml` + `server` scripts, components for
runtimes/models, install hooks for hardware selection), and
`reference/inference-snaps-cli` (`modelctl`, IE108 CLI compliance).
Design note: `docs/asr-inference-snap-design.md` (T13 output).

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T13 | Study + design note: how qwen3/gemma4 snaps structure engines/components/hooks, and what an ASR snap does differently (audio-push server instead of llama.cpp server) | done | Claude/Charles | — | `docs/asr-inference-snap-design.md`. Decisions: one snap per family, Whisper first; copy gemma4's v2 engine/runtime/model schema; testbed adapter doubles as the snap's server; socket-activated UDS daemon; no audio interfaces needed |
| T14a | `myna.server`: standalone entry point wrapping an `SttService` adapter behind the real transport on a UDS path | done | Claude/Charles | T07, T16 | `myna-server --socket … --model …` (`server/src/myna/server/`); subprocess integration test in `server/tests/test_server.py`; verification client `dev/transcribe.py` |
| T14b | whisper-snap skeleton: snapcraft.yaml (core24), `cpu` engine, tiny/base/small models, `modelctl` v2 wiring, install hooks | done — installed & verified | Claude/Charles | T14a | `whisper-snap/`; installed, fixture clips transcribed through the snap socket; model selection (tiny→base) verified live. Warm finalize 0.5–0.8s on base CPU. Skeleton deviations: weights downloaded at runtime (components → T15), socket world-connectable (access control → T14c). Snapd findings recorded in design note (`network-bind` required for Unix `listen()`; `ws+unix` protocol unknown to `modelctl status`) |
| T14c | Socket exposure + access control for confined clients (content interface vs file perms + polkit) | done (confined half) | Claude | T14b, T17 | **Resolved for confined clients 2026-07-23 (T57/feature 005):** each inference snap slots `$SNAP_COMMON/run` as a writable content share (`ubustt-socket`); the `myna` client snap plugs it (socket at `$SNAP_DATA/backend/run/ubustt.sock`), verified live end-to-end. snapd's writable content shares exist precisely for named sockets (content.go). Remaining with T17: identity-based access control (polkit) beyond "an admin connected the plug". |
| T15 | Engine variants + hardware selection (nvidia-gpu engine, remaining model components, VRAM-aware model gating follow-up with inference-snaps-cli team) | done | Claude/Charles | T14b, T12 | **Model components landed (CPU+GPU):** weights ship as `model-{tiny,base,small}` components via `dev/download-models.sh`; server loads `MODEL_DIR=$SNAP_COMPONENTS/<id>`; `network` plug dropped (no runtime download). Runtime delivery: cpu venv in base, CUDA as `faster-whisper-cuda` component. **nvidia-gpu engine verified on hardware 2026-06-13** (auto-selection scores GPU>CPU; CUDA libs resolve via runtime.yaml LD_LIBRARY_PATH). Sideload requires `.comp` files in the same `snap install` (documented in README). Remaining-but-deferred: larger GPU weights (medium/large-v3) + VRAM-aware per-model gating, both gated on T12 sizing (upstream gap #1, design note §4) |
| T27 | Model idle-unload (in-process): `myna-server --sleep-idle-seconds`, `adapter.unload()` releasing model weights + GPU memory, idle monitor in the server | done | Claude/Charles | T14a | **Done + wired into the snap:** `--sleep-idle-seconds N --idle-action unload`; `unload()` on both adapters (drop ref + gc; NeMo also `torch.cuda.empty_cache()`); `LifecycleService` + `idle_monitor` (`myna/server/lifecycle.py`). Engine `server` scripts pass `--sleep-idle-seconds "$(modelctl get sleep-idle-seconds)" --idle-action unload`; install hook default 300s. Frees the **weights** (the bulk: ~1.1 GB of 1.4 GB for NeMo); process + CUDA context (~300 MB) stay (full release blocked — see T28). Re-warm faster than cold; `progress.phase=preparing` covers the UX. Verify via `nvidia-smi` |
| T28 | snapd socket activation: snapd owns the UDS, daemon starts on first connect and **exits after idle**, relaunched on next connection — full process/VRAM release at idle | blocked (upstream `modelctl run`) | Claude/Charles | T14b, T27, T26 | **Server mechanism done + dev-testable** (`serve_unix` serves on a `systemd_socket()` fd; `--idle-action exit`; verified on a pre-bound socket; runs under `systemd-socket-activate -- .venv/bin/myna-server …`). **Blocked for the snap:** `modelctl run` forks the server and doesn't pass fd 3, so the activation handoff can't reach `myna-server` (design note §4). Snap ships T27 in-process unload instead. Unblock = upstream `modelctl run` change (`syscall.Exec` or forward the listening fd + `LISTEN_PID`). Wake-cost note (Charles, 2026-06-14): a few seconds, not great → GPU parsimony via T27 prioritised for now. **Revive policy (recorded 2026-07-06, hazard review):** IE115 connections are now persistent (T47), and exit-on-idle kills open connections — when this unblocks, the policy must be either *exit only with zero open connections* (plus a server-side idle-connection close so a lazy client can't pin the process forever) or *clean close while idle is benign, client reconnects*; either way the WS close-code semantics must be explicit before shipping |
| T53 | Sync the snaps with upstream modelctl's improved multi-model support | done (2026-08-19) | | T15 | Farshid (2026-07-08, testbed review — he also verified whisper on the NVIDIA GPU): "minor things need to be updated on the modelctl side as we've improved the support for multiple models; nothing concerning". Get the specifics (version/changelog), update whisper/nemotron/qwen snaps' modelctl wiring, and check whether it touches the `ws+unix` status-protocol gap (design note §4.2), the `modelctl run` fd-passing blocker (§4.4/T28), or the one-model-per-process assumption (T48). Same review withdrew the vRAM-gating upstream ask (design note §4.1) — load failures surface on the wire instead (T31). **Hold-off (2026-07-08, Farshid follow-up):** *"another minor breaking change coming — maybe hold on to that for a few more days"*; don't sync yet. The same follow-up detailed engine selection as device-**attribute** matching (GPU vendor/model/NVIDIA compute-capability/AMD microarch; CPU flags/features/CPUID for e.g. Intel OpenVINO), *not* VRAM capacity — folded into `configuration-api.md` §3.6. **Done (2026-08-19):** all seven snaps synced v2.0.0-beta.1 → v2.0.0-beta.12. Changes: `cli`→`modelctl` tarball restructure (beta.4), runtime.yaml `name` field (beta.5), pciutils dropped (lscompute, beta.6), model `id`→`name`/`alias` + non-standard capabilities dropped (beta.8/beta.9), `socket.path`→`ws.unix-socket` (beta.12 entrypoints — §4.2 gap **resolved upstream**, `status` now reports the UbuSTT socket), lint-package wired into CI (`dev/lint-packages.sh`). §4.4/T28 unchanged: `run` still forks without fd passing at beta.12. T48 untouched. Linter also caught two pre-existing bugs: audio8's non-schema `license:` field and sherpa's 57-char engine summary |
| T32 | `nemotron-snap`: package the Nemotron adapter as an inference snap (one snap per family), sibling of whisper-snap | done | Claude/Charles | T09, T15 | `nemotron-snap/` mirrors whisper-snap: `nvidia-gpu` engine, **`nemo-cuda` runtime component** (myna[nemotron] = NeMo + torch + CUDA pip tree — multi-GB), `.nemo` model component (`model-streaming-multi`), `att-context-size` dial exposed as engine config, idle-unload wired (T27). Adapter restores a local `.nemo` via `restore_from`. **Verified in strict confinement on hardware 2026-06-14:** NeMo+torch+CUDA load, `LD_LIBRARY_PATH` resolves first try, `.nemo` restores, server serves. Benign warnings (no action): pydub/ffmpeg (we feed raw PCM, never decode files) and joblib serial-mode (confinement blocks its shm probe). Socket activation blocked (T28) so it ships in-process unload |
| T29 | Residency **default policy** (adaptive): define and implement the v1 default everyone runs under — idle-unload timeout, per-engine warm-vs-cold strategy (whisper full-exit cheap-wake vs NeMo keep-warm), with memory-pressure / power-source awareness as the growth path | todo | | T27, T28 | **Name this first** — the dev toggles (T30) and any future user control are deviations *from* this baseline. 95% of users live here and never touch a knob, so the default is the product. Eventual user-facing control exposes *intent* ("keep dictation instantly ready" vs "free memory when idle"), never mechanism, and lands in UD129 (desktop Settings) scope, designed later from evidence not guesses. **Coupled decision (2026-07-06, hazard review):** the client's capture ring-buffer depth (T20/T21, audio-adapter-api §6) must cover the worst-case cold load this policy tolerates — decide the idle-unload default and the buffer depth together, or pre-ready speech is lost. **Reinforced (2026-07-15):** the client ring has a single sizing driver — the cold-load window (Kind 1); steady-state/processing buffering is the inference backend's concern, and the ring's resident memory (~KB–MB) is ~1000× smaller than the weights, so the memory lever here is residency, not ring depth (`audio-adapter-api.md` §6) |
| T30 | Development/test lifecycle toggles: deterministic controls to force residency states (unload-now, pin-resident, set idle-timeout, force-cold-wake) so engineering can test scenarios in the wild; hidden, not user-facing UI | todo | | T27 | **Two shapes, do not conflate:** (a) *test control* — imperative, ephemeral, forces exact states for engineering; (b) *power-user out* — declarative, persisted config (modelctl/IE108 scope), undocumented-but-reachable. Caveat for the "gather data to design the UI later" rationale: it collides with the offline/privacy posture (no content, no default phone-home), the only reachable population is power users (a biased sample for a mainstream UI), and reachable-but-hidden is still a production contract (Hyrum). Treat UI-informing as a weak secondary benefit; justify the work on (a) |

## Workstream C — Transport (IE114 wire protocol)

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T16 | Prototype WebSocket-over-UDS client/server implementing the session contract; PCM binary frames in, JSON events out | done | Claude/Charles | T02 | `myna/core/transport_ws.py`; contract tests parametrized over loopback + ws-uds; wire protocol documented in module + ADR, feeds T18 |
| T17 | Access-control spike: socket permissions, snap confinement, client identity (polkit?) per IE114 comments | todo | | T16 | Documented recommendation, not necessarily code |

## Workstream D — Desktop client (UD129)

**Taken over by Charles (2026-07-07)**, starting with the **audio adapter as a
Rust crate** (`client/myna-audio`) dropping in behind the orchestrator's existing
`AudioSource` trait (T41) — the seam `docs/audio-adapter-api.md` describes.
Design decisions (2026-07-07):

- **Capture behind an internal `CaptureBackend` trait** — `pw-record`
  subprocess first (known-working, ported from `MicSource`), native
  `pipewire-rs` later behind the same seam (proper device/channel enumeration,
  no fork, graph-side resample). Resolves the §10 backend question.
- **The adapter owns the bounded pre-ready ring** (§6 requirement): `capture()`
  fills it from hotkey press; the consumer simply doesn't drain until `ready`.
  Depth is a constructor param, decided together with T29. Overflow policy is a
  T49 design point (proposal: drop-oldest, surfaced via the stats tap, never
  block the capture path).
- **No in-crate DSP.** Real filtering (noise suppression etc.) is PipeWire
  filter-chain territory, upstream of our capture node; the crate provides an
  **observation/stats tap** (RMS, peak, clipping — things a UI wants), the §8
  hook grown into a real API. VAD stays out (push-to-talk: the hotkey is the
  VAD). This keeps the meeting-prep "DSP stays in PipeWire + model" position.

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T20 | Audio Adapter: PipeWire capture → bounded in-memory buffer → `PcmChunk` stream; VAD optional first pass | superseded (→ T49–T52) | Claude/Charles | T01 | **Seeded the seam, then re-planned as the Rust crate (T49–T52) when Charles took over D (2026-07-07).** What it produced: `MicSource` (`myna.testbed.sources`) — live PipeWire capture via `pw-record --raw` → `PcmChunk` stream, memory-only, `stop()`-terminated; `dev/dictate.py` working push-to-talk demo verified live; `docs/audio-adapter-api.md` draft. The meeting-prep shape decisions stood (library not daemon; DSP stays in PipeWire + model; VAD dropped — the hotkey is the VAD); the unfinished remainder (bounded ring, node wiring, fake-capture unit tests) moved into T49–T52 |
| T21 | Session controller: hotkey press/release driving the `DictationState` machine, wiring audio → `SttSession` → text output | done | Charles | T16, T51 | **Done (feature 003-desktop-injection).** Shipped as the Rust `client/myna-desktop` `DesktopController`: a per-utterance loop over the unchanged `run_dictation` (capture-at-press, push gated on `ready`), driven by a `Trigger` (portal hotkey / stdin stand-in / scripted), routing `Final`→injector and liveness→indicator, with the `DictationState` table enforced (illegal transition = panic, tested). Push-to-talk asserted: no capture while Idle (SC-004). Activation is `org.freedesktop.portal.GlobalShortcuts` hold-to-talk (`Activated`→Press deduped, `Deactivated`→Release) via `ashpd`; default `Super+D`, rebindable in the desktop's own dialog. Capabilities `input_format` negotiation is a recorded follow-up (MVP uses the default 16 kHz s16le mono the shipped adapters accept). Contract: `docs/desktop-injection.md`. |
| T22 | IBus `TextInjector` backend (commit-only MVP) + activity indicator; end-to-end dictation demo against the inference snap | done | Charles | T14, T21 | **Done (feature 003-desktop-injection).** `IbusInjector` speaks the IBus wire protocol over `zbus` (pure Rust, no FFI/subprocess — R1): registers a component/engine, becomes the active global engine, commits via the engine's `CommitText` signal (commit-only), restores the prior engine on end (idempotent). Focus-out → end safely (suppress further commits, SC-007); password/PIN content-type → refuse (`SecureField`, SC-008). Activity indicator: a GTK4 overlay (`ui-gtk`) with distinct per-state visuals + AT-SPI labels + `notify-rust` error fallback, on the GTK main thread bridged to the tokio controller. Verified: GVariant shapes + handshake against the live daemon, full register→activate→commit→restore against an isolated daemon (gated `ibus_hw`). The manual spoken run into a focused GNOME editor is the on-hardware acceptance. Contract: `docs/desktop-injection.md`. **R9 preedit partials landed 2026-07-27 (opt-in):** `myna-desktop --preedit` renders streaming `Unstable` hypotheses in the field's IBus preedit region (`UpdatePreeditText`, underlined, replaced per delta, cleared by commit; same secure-field/focus-loss guards as commit). Default stays commit-only pending the UD136 in-field-hypothesis design call; live acceptance is dictating into a GTK editor against `myna-server --streaming`. Same day's live test exposed two text-assembly bugs, fixed: streaming committed deltas were stripped of whitespace (injectors concatenating them produced "Thisis notworking…") — the 008 I2 contract is now **verbatim concatenation** (deltas carry natural whitespace; the controller joins defensively, never doubling), and unstable deltas restated already-committed words (now the uncommitted remainder only, so preedit reads as a continuation). |
| T23 | Post-processing pass 1: capitalization/punctuation gap analysis per model (Nemotron has it native; Whisper mostly; what remains?) | todo | | T07 | Decide what post-processing the MVP actually needs before building it |
| T49 | **Audio-adapter API v2**: revise `docs/audio-adapter-api.md` from discussion draft to settled contract | done | Charles | T41 | **Done 2026-07-07.** The doc is now the settled v2 contract: `CaptureBackend` seam (`CaptureSpec` + sync never-blocking `Producer::push`/`finish`, callable from a realtime callback; backends: `ScriptedBackend` T50 / `PwRecordBackend` T51 / `PipeWireBackend` T52); **overflow decided: drop-oldest** + `AudioStats::dropped` duration on the tap (blocking would stall capture; failing turns a slow load into a user-facing error); ring depth **default 10 s provisional**, one decision with T29; stats tap = `watch::Receiver<AudioStats>` (RMS/peak/clip linear full-scale + captured/dropped durations), updated at capture time so the meter moves during a cold load; T33 room reserved but no `Encoding` enum yet (serde shape is golden-tested against Python — both languages change together); consumer traits (`AudioSource`/`CaptureStream`/`CaptureError`/`StopHandle`) **move to `myna-core`** (executed in T50), orchestrator re-exports; filtering = PipeWire filter-chain (rnnoise example), no in-crate DSP; `capture()` = the hotkey press, polling deferred until `ready` — no two-phase API. v1's §10 questions all resolved (appendix table) |
| T50 | **`myna-audio` skeleton crate**: `AudioSource` impl = `CaptureBackend` (trait) + bounded ring + stats tap; fake backend + unit tests | done | Charles | T49 | **Done 2026-07-07.** `client/myna-audio`: `CaptureSource` (builder: `ring_depth` default 10 s provisional/T29, `chunk` 100 ms, `target`, `channels`, required `backend`) = `CaptureBackend` → sync never-blocking `Producer::push`/`finish` → whole-frame re-chunker → per-chunk stats (`watch<AudioStats>`: RMS/peak/clip + captured/dropped durations, updated at capture time) → drop-oldest `Ring` → `CaptureStream` (drop-guard: dropping the stream trips stop + discards the ring). Consumer traits **moved to `myna-core::capture`** (`AudioSource`/`CaptureStream`/`CaptureError`/`StopHandle`); orchestrator re-exports, zero call-site changes. `ScriptedBackend` (permanent fixture: Silence/Bytes/Wait/Fault steps, `unavailable()`, finished-probe) drives the behavioral suite: hold-then-drain-everything, graceful stop, abort-reaches-backend, overflow drop-oldest+tap accounting, fault = drain→one `Err`→end, unopenable device, level tracking, whole-frame flush — 14 crate tests. **Drop-in proven**: `CaptureSource`+`ScriptedBackend` through the *unchanged* T41 `run_dictation` runner completes a scripted session (orchestrator dev-dep test) — the mock audio adapter for orchestrator work exists. 75 workspace tests, clippy clean |
| T51 | **`pw-record` subprocess backend** (port of `MicSource`) + live verification | done | Charles | T50 | **Code done + mic-verified 2026-07-07; one spoken run remains.** `PwRecordBackend` (`myna-audio::pw_record`): `pw-record --raw` per the negotiated format (PipeWire's graph link does the resample/downmix — §7 for free), `--target` node selection, 250 ms bounded reads (stop promptness §5), kill-on-stop with bounded stderr harvest; an *unrequested* EOF = `DeviceUnavailable` fault with the child's stderr attached (pw-record records until killed, so EOF means the device/daemon vanished); `spec.channels` rejected → T52, non-s16 width rejected (T33). 5 module tests over an `sh` stand-in command (endless-producer stop/kill, premature-EOF fault + stderr, spawn failure, channel/format rejection). **Runner now captures at press** (stream opened immediately, only the *push* parks on `ready` — the §6 client half, replacing T41's hold-the-stream gate; safe for lazy WAV sources, correct for live mics) + `AudioSource for Box<dyn AudioSource>` so the CLI picks mic-vs-clip at runtime. `myna-dictate --mic [--target <node>]` wired, prints a per-utterance `captured/dropped` readout. **Verified live** against `myna-server --adapter fake` with the real default mic (Razer Kiyo Pro): press → `loading model…`/`ready` gate → scripted transcript → `captured 3.0s, zero drops`, clean exit; 80 workspace tests + clippy clean. **UX-robustness fixes (2026-07-13, shaken out on a second machine `charles@framework` with no working capture source):** (1) a capture fault no longer collapses into a silent `(aborted)` — new `Input/OrchestratorInput::CaptureFailed { message }` lands in a visible `Failed { code: "capture_failed" }` (still `SendAbort` upstream, nothing committed), and the CLI prints `✗ audio capture failed: … (no mic? check wpctl status / --target)`; (2) an empty `Done` transcript renders `✓ (no speech detected)` instead of a bare `✓`; (3) `AudioStats::session_peak` (session-wide max, not the wire-golden'd `myna-core` shape) drives a `⚠ input was near-silent` hint (>0.5 s window, peak < 0.01 ≈ −40 dBFS) to separate a muted/wrong node from a quiet talker. One caveat left: an EOF read *after* the stop flag is set is classed as a graceful stop, so a fast device-EOF can still be masked as an empty transcript if the client stops first — tighten if it bites. **Remaining (needs a human voice + a live capture source):** one spoken run against `--adapter whisper --model base` asserting a non-empty, exact transcript — `myna-server --adapter whisper --model base --socket /tmp/myna.sock` then `myna-dictate --socket /tmp/myna.sock --mic`. On `framework` the default source is absent (`wpctl status` shows an empty Audio→Sources); needs a mic present/selected (or `--target <node.name>`) before this can close. **Closed 2026-07-19:** the spoken run passed; the subprocess backend was subsequently retired on branch `002f` (feature T033). |
| T52 | **Native `pipewire-rs` backend** behind the same `CaptureBackend` seam | done | Charles | T51 | **Done via the spec-kit flow (feature `002-native-pipewire-backend`, 2026-07-15).** `PipeWireBackend` (`myna-audio::native`): a dedicated PipeWire main-loop thread owns a capture `Stream`; the `process` callback pushes PCM straight into the ring (never blocks), a ≤100 ms stop-poll timer honors graceful-stop/abort within the §5 promptness bound. Requests **S16LE at the negotiated rate/channels** so the graph resamples/downmixes (§7, FR-003); node selection by stable `node.name` via `PW_KEY_TARGET_OBJECT` + `DONT_RECONNECT` (a chosen device that vanishes faults; a *bogus* target falls back to default — documented platform limitation, pw-record did the same); **channel pick/downmix** = request `max(idx)+1` graph channels then interleaved-S16 select+average to the negotiated layout. **Live device enumeration** is a sibling API `InputDevices` (registry watch on its own loop thread, `list()` + `watch::Receiver<Vec<InputDevice>>`); `myna-dictate --list-devices` shows it live (verified: two internal mics, then AirPods Pro appearing). `myna-cli --mic` switched to the native backend. **Verified on live PipeWire (this machine):** 10 env-gated integration tests (`MYNA_PIPEWIRE_TESTS=1`, `tests/pipewire_hw.rs`) — default capture + graph-side 48 kHz/stereo conversion + prompt stop + clean abort + named-target select + 4-ch channel-select + enumerate/observe-add-remove + capture-path watermark — all green; plus channel-mix unit tests; full workspace + clippy clean. Constitution: TDD throughout, dual-env integration suite (skips offline, runs unchanged on VM/hardware), watermark baseline, `.workshop/` Workshop definition added (Principle IV; validated by `workshop launch myna` + in-env `cargo build`/`cargo test`), privacy/offline preserved. **Remaining (shared with T51's close-out — a human voice):** one spoken run through `--adapter whisper` asserting an exact transcript; then delete `pw_record.rs` (feature T033, staged last so `--mic` never breaks on `main`) |
| T55 | **Bring the rest of the toolchain under Workshop** — extend `.workshop/myna.yaml` with the Python testbed extras (`whisper`, `nemotron`) and a CUDA/GPU SDK so the whole build+test+bench matrix runs in the canonical environment, not just the Rust audio crate | todo | Charles | T52 | **Follow-up (opened 2026-07-19).** The initial `.workshop/` definition (T52/feature 002) deliberately scoped to what `myna-audio` needs (Rust + uv + the in-project `pipewire` SDK) and CI runs the Rust suite + offline `pytest` there (`.github/workflows/ci.yml`). Next: (a) an in-project or store SDK that `uv sync --extra whisper --extra nemotron` (torch/CUDA/NeMo — multi-GB; decide cache-mount strategy), (b) a CUDA/GPU SDK + a GPU-runner CI lane for the model adapters and `dev/matrix.py`, (c) fold the snap-build deps (`dev/prepare.sh`, snapcraft) in if we want reproducible snap builds under Workshop. Keep it composable — one SDK per concern, opt-in per workflow. |
| T56 | **GNOME Shell indicator accessibility (screen-reader announcements)** — the `myna-shell` extension's HUD pill (feature 004) conveys dictation state purely visually (icon + colour + a content-free status label); a screen-reader user gets no proactive cue as state advances. Spec + build an AT-SPI/live-region announcer so state transitions (listening / finishing / error) are spoken. | todo (spec later) | Charles | T22 | **Opened 2026-07-20**, carried forward through the 2026-07-30 HUD redesign (top-of-panel ribbon → bottom-center pill) — the gap is unchanged and still deliberately out of scope. The indicator is deliberately non-focusable chrome (X11/SC-001), and its `St.Label` text updates silently — Orca won't announce transitions, and it's unverified whether AT-SPI even surfaces the chrome. **Likely shape:** drive announcements from the state transitions in `extension.js` (the *stable* side of the `view.js` seam, so it's independent of whichever view is active), not from a view — e.g. an accessibility notification / live-region utterance per `stateToDescriptor` change, content-free (constitution V: state labels only, never transcript text). Also revisit reachability of the label to AT and whether the Rust `myna-desktop` GTK indicator (T22) needs the parallel treatment. Spec it out before building. |

| T57 | **Ship the orchestrator as a snap** — package `myna-desktop` + `myna-dictate` as the strictly-confined `myna` snap; expose the backend socket to confined clients (T14c content-share half); CI snap build | done (human acceptance left) | Claude | T21, T22, T14b | **Done 2026-07-23 (feature `005-myna-orchestrator-snap`, branch `005-myna-orchestrator-snap`).** `myna-snap/` (core24, strict, **no `network` plug** — every boundary is a Unix socket or the session bus): apps `myna` (daemon, portal hotkey + `--dbus` publisher by default), `myna.toggle`, `myna.install-shortcut` (dconf-based — GNOME schemas are invisible inside the base), `myna.testbed`. **T14c resolved for confined clients:** each inference snap now slots `$SNAP_COMMON/run` as a *writable* content share (`ubustt-socket`) — the case snapd's content interface explicitly supports named sockets for; `myna:backend` plugs it (socket at `$SNAP_DATA/backend/run/ubustt.sock`). Identity/polkit access control remains T17; one backend at a time until T48. **Confinement fixes in the client:** IBus address discovery searches the real home via `/etc/passwd` when `$SNAP` is set (+unit tests); `--install-shortcut` writes `/snap/bin/<instance>.toggle` not a revisioned exe path. **Packaging findings:** (1) the `gnome` extension is unusable for Rust builds — its `LD_LIBRARY_PATH` points at the core22 gnome-46-2404-sdk whose older glibc breaks every noble-linked build tool; GTK4 is staged from noble instead; (2) snapcraft's `rust` plugin env-validation is chicken-and-egg with rustup 1.29 (`rustup dump-testament` fails before a toolchain exists) — nil `rust-deps` part + explicit toolchain instead; (3) a confined PipeWire client needs the full staged set (lib + `libpipewire-0.3-modules` + `libspa-0.2-modules` + `pipewire-bin`'s `client.conf` + `SPA_PLUGIN_DIR`/`PIPEWIRE_MODULE_DIR`/`PIPEWIRE_CONFIG_DIR`) AND a compat symlink `$XDG_RUNTIME_DIR/pipewire-0 → ../pipewire-0` (snapd scopes XDG_RUNTIME_DIR; the desktop-launch pattern — wrappers do it). **Verified live, confined** (polkit session install, no sudo): testbed WAV round-trip through the shared socket (exact transcript, whisper small); `--list-devices` over the pipewire plug; IBus connect; `org.myna.Dictation` owned with live State/Error props; portal bind succeeds; control-mode `myna.toggle` cycle; dconf write. **Left for human acceptance:** the literal Super+D press (no uinput access for synthesis) and a spoken injection run; nemotron/qwen rebuilds with the slot (yamls done); autostart-on-login (snapd user daemons still experimental); store registration of the name `myna` (free as of tonight). CI: `.github/workflows/snap.yml` builds + smoke-tests the snap on client/packaging changes. |
| T58 | **UD136 trigger alignment**: default shortcut choice (we ship Super+D; UD136 suggests Super+J after a clash audit — Super+V taken by clipboard managers), KEY_DICTATE/XF86Dictate hardware-key support when present, Escape-ends-transcription | todo (discuss) | Charles | T21 | UD136 trigger section: hardware dictate key primary where available; a clash-free default combo; user-rebindable (shipped via the portal dialog, T21); opt-in push-to-talk (shipped) and double-tap-Ctrl (hard — mutter-level, not MVP). **Escape-to-end is excluded from MVP** by review (the STT client doesn't own key focus; the focused widget may have its own Escape handling). Decision needed: keep Super+D or move to the design's Super+J before MVP ships |
| T59 | **Silence auto-stop**: end transcription automatically after ~15 s of no voice input (UD136) | todo | Charles | T52 | UD136: "Transcription should end automatically after 15 seconds of no voice input" (benchmarks: Windows/macOS ~30 s, Android ~5 s — reviewers want it tuned from real-world performance, so make the value configurable and measure). Client-side energy gate on the existing `AudioStats` tap (RMS/peak) in `myna-audio` — no model VAD; the hotkey stays the primary turn signal. Watch the interaction with T29 residency: a long pause shouldn't idle-unload the model mid-utterance |
| T60 | **Typing cancels transcription** (UD136): typing while STT is active immediately cancels the session | todo (spike) | | T21 | Wayland feasibility spike first: the dictation client doesn't own key focus, so it can't observe typing directly — needs compositor/portal support or a toolkit/IBus-side signal. Likely excluded from MVP on the same grounds as Escape (review thread), but record what a confined client *can* observe (IBus is our one privileged seam — it may see surrounding-text/key events) |
| T61 | **Sound cues**: on/off/error chimes on state transitions, user-configurable (UD136 state-cue table) | todo | | T22 | UD136 pairs every visual state cue with a sound (recording-on chime, off chime, error chime); Windows/macOS ship them on by default; reviewers want a toggle. No audio-*out* plumbing exists in `myna-desktop` today — likely libcanberra/PipeWire playback from the orchestrator on `DictationState` transitions, driven next to the indicator updates. Content-free by construction (constitution V untouched); error chime pairs with T62's notifications |
| T62 | **Error-state UX mapping** (UD136): the design's explicit failure list → wire codes + user-facing notifications | todo | | T31, T22 | UD136's list: no microphone available; mic permission denied; speech model not installed; unsupported language; inference backend unavailable; no compatible text target focused; secure field (we already refuse — T22 `SecureField`); plus runtime errors (buffer full, engine unresponsive, target field closed mid-session). Each needs a **T31 wire code** (disposition on the wire) mapped to a user-facing ephemeral notification (design has this as an open question mark — propose one). This is T31's forcing consumer: the taxonomy must distinguish these cases for the client to render them |
| T64 | **In-field partials: preedit styling + default** (design meeting): with `--preedit` live (2026-07-27/28), decide (a) whether in-field unstable partials graduate from opt-in to default, and (b) what *differentiating formatting* is actually achievable — the UD136 spec body draws an uncommitted-hypothesis style, but on GNOME/Wayland the text-input-v3 path strips IBus preedit attributes (underline attr is honored by GTK's own preedit rendering only where the app/toolkit supports it; ibus ChangeLog: "GNOME Wayland uses text-input version 3 which deletes the preedit style"). Empirical facts for the meeting: preedit replace/commit semantics work end-to-end (probe-verified); styling is app/toolkit-dependent, so a cross-app consistent "unstable look" may need the shell extension (feature 004 chrome) instead of in-field styling | todo (discuss) | Charles | T22, T08 | Raised from live testing of the R9 preedit work; see `docs/desktop-injection.md` §Streaming preedit |

## Workstream E — Spec feedback

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T18 | IE114 update proposal: audio-push model, WebSocket transport, event vocabulary (progress/final/done/error), lifecycle signal (from T26) | done (draft, for review) | Claude/Charles | T16, T26 | **Draft delivered 2026-06-17:** `docs/IE115-proposal.md` — discrete anchored comments (one per IE115 section) + position paragraph, for Charles to paste into the IE115 Google Doc as comments. Folds in the deviations doc, the implemented versioning/handshake (T35), the progress/final/done/error vocab (T36), capabilities (T24), and the `preparing` phase (T26). **Committed (2026-06-14):** audio-push model + WebSocket-over-UDS transport (working prior art in `transport_ws.py`) + the progress/final/done/error vocab. Deliverable: the written proposal for Farshid. **Error model carved out to T31** — it was the genuinely-unresolved third and shouldn't block the transport/vocab draft. **2026-06-17:** IE115 landed and independently validates the transport + push direction here; the IE115-specific reconciliation (versioning, event-name alignment, deviations note) is split into Workstream F (T34–T37) so this task stays the IE114-targeted update and the IE115 decisions get their own owners |
| T31 | Stable error-code taxonomy for IE114: enumerated codes with semantics (terminal vs recoverable, client vs server fault, retryable), replacing the ad-hoc adapter strings (`unsupported_audio_format`, `inference_failed`) | todo | | T18 | The part IE114 itself flagged incomplete. Two codes exist in adapter code, not a spec. Needs the full set + meaning before T18 can claim a complete error model. **Requirement recorded 2026-07-06 (hazard review): the disposition must ride the wire** — an additive field on the error frame (e.g. `error.recoverable: bool`), never a client-side code→disposition table; three independent copies of that table existed (Rust `classify_error` string-match, the lossy `_ERROR_TO_IE115` map, the ws server's terminal-always behavior) and were guaranteed to drift. The Rust FSM's speculative `Recoverable`/`Advisory` branches were **deleted** the same day (unreachable over either real dialect — every error is terminal on the wire today); reintroduce retry semantics only when the wire flag exists. See `ie115-wire.md` §6 |
| T26 | Spec decision: add a session-lifecycle signal so clients show "loading model…" distinctly from "transcribing" | resolved (2026-06-14) | Claude/Charles | T18 | **Decision: yes, add a lifecycle signal** — `progress` conflates "model loading, nothing happening yet" with "transcribing, almost done", and the cold-load measurements (0.9–2.2 s whisper, more for NeMo) make that gap real. **Scoped, though:** the audio-push model collapses IE114 comment [h]'s `starting/listening/transcribing` trichotomy — the client owns capture so "listening" is redundant and "transcribing" == `progress`. So **one** new phase ("preparing"/model-loading), not a 3-state FSM (that would repeat the over-spec'ing the vocab simplification rejected). **Implemented as `progress.phase` field** (`preparing`/`transcribing`, default transcribing) — not a new event; adapters tag the load-heartbeat `preparing`, `dev/dictate.py` shows "loading model…" distinctly; field round-trips and is forward/backward compatible across the wire |
| T19 | Performance contract proposal: latency SLOs grounded in measured testbed numbers, per hardware tier | todo | | T12 | Draft for IE114/UD129 |
| T24 | Capabilities-discovery API sketch (models, languages, punctuation support) | done | Claude/Charles | T18 | `myna.core.capabilities.Capabilities` (models, languages, `input_formats`, punctuation, translation) + wire codec; `SttService.capabilities()` / `SttClient.capabilities()` served over **both** transports (loopback direct; WS answers a `capabilities.query` message, parametrized contract test proves wire parity); all three adapters populate it (whisper multilingual/punctuation; nemotron en-only/native-punct; fake trivial). `dev/capabilities.py` queries a live snap. **Folds in the audio-format advertisement** (`input_formats`): the service states what PCM it accepts, the client delivers it, and the adapters now **reject** off-format audio instead of resampling — the `np.interp` blocks are gone from both adapters (symmetric with the existing channels/width rejection; conversion is the client's job under audio-push). Provisional vocab → feeds T18 |
| T54 | **Backend-snap configuration API spec** — how the desktop (Settings/orchestrator/power user) reads + changes backend-snap config (model/engine selection, residency policy, tuning) and who owns that boundary; requires inference-snap-side API changes | todo (discuss) | Charles | T24, T48, T53 | **Draft 2026-07-12:** `docs/configuration-api.md` — team-discussion proposal. Separates three layers (provisioning/snapd · configuration/modelctl · session-params/wire) and the two model-selection verbs (session *request* vs config *selection*). Positions: config stays off the transcription socket; CLI first (modelctl) then a UI consuming the same schema; a polkit-gated/root-daemon **broker** for unprivileged writes (the meeting's "external root daemon"); **unprivileged read, brokered write**; engine = device-presence auto + manual override (post-VRAM-gating, answers Charles→Farshid); residency exposed as *intent* (T29). **Central required snap-side change: a machine-readable config-schema discovery** (`modelctl describe-config` — type/domain/default/scope/restart-required), the write-side sibling of capabilities (T24). Open: owner (device eng?), broker shape, provisioning home, restart-vs-live-reconfigure, schema authority, relationship to T48. **Concrete `describe-config --format=json` strawman + provisioning-progress event shape in Appendix A (whisper-snap, real keys).** Bring to the team |
| T33 | **Team discussion (undecided):** sample *encoding* in the audio format — should `AudioFormat`/`input_formats` carry int16-vs-float32, so int16→float32 moves to the client (capture-native or edge-convert) and adapters only reinterpret, never convert? | todo (discuss) | | T24 | **No decision yet — bring to the team.** Finding (2026-06-16): `AudioFormat` has no encoding field today (width only; wire is implicitly S16LE, float32 not expressible). At the raw-frame API **all the adapters want the *same* thing** — float32 normalised [-1,1] 16k mono — and each does the identical `int16→float32/32768` (whisper/nemotron/qwen-c). int16 ingestion exists only at file/stdin decode boundaries we don't use, and converts to float internally anyway. **Implications pull both ways:** uniform target ⇒ a single wire encoding (keep s16le, or switch to f32le + convert once in `MicSource`) settles it without per-model negotiation; and since ASR universally wants float, a full capabilities *encoding-negotiation* axis looks premature (solves a divergence that doesn't exist). Open question = which wire encoding + where the one conversion lives (adapter today vs source/edge per the audio-push invariant). Don't over-build until a model actually diverges. Analysis: this session's grilling. **Position to bring to the team (recorded 2026-07-06, hazard review — dissolves T33 without moving code):** keep **s16le on the wire** (capture-native; half the bytes of f32 for the same information); add an `encoding` discriminant to `AudioFormat` for future-proofing only; reclassify int16→float32 as decode-side *reinterpretation*, not client-owed conversion — the no-resample invariant is about rate/channels, where information is destroyed, and a widening reinterpretation destroys nothing |

## Workstream F — IE115 reconciliation

IE115 (`IE115-spec.txt`) is a WebSocket transcription API modelled on OpenAI's
Realtime API. It ratifies our two pivots away from IE114 — **audio-push** (the
client streams PCM up, no `pipewire-node-name`) and **WebSocket** (the forced
consequence of audio-push: SSE can't carry a client→server audio stream). The
work here is *not* a rebuild — our `myna.core` is already closer to IE115 than
to IE114 — it is renaming/negotiation plus stripping the OpenAI speech-to-speech
baggage (conversation-item object graph, 24 kHz, base64-in-JSON, server VAD,
`obfuscation`/`usage`, voice/tools/instructions) that a local transcriber
doesn't want. See `docs/IE115-deviations.md` for the full keep/strip/modify
rationale.

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T34 | Deviations note (`docs/IE115-deviations.md`): per-feature keep/strip/modify position vs OpenAI Realtime, each justified by an invariant or a measured number | done | Claude/Charles | — | Landed + committed 2026-06-17. Six push-backs (no conversation model; 16 kHz via capabilities not fixed 24 kHz; binary frames not base64; server VAD optional not mandatory; drop `obfuscation`/`usage`; strip s2s config) + conscious decisions on what IE115 drops (segments/score, prompt, model-loading phase, reconnect). This is the team-facing artefact; IE115's own "Deviations from OpenAI Realtime API" section lists only two trivial ones |
| T35 | Protocol versioning: handshake-negotiated `protocol_version` (client states in `session.start`, server echoes in `session.created`) + a versioned event-vocabulary set | done | Claude/Charles | T34, T16 | **Done 2026-06-17.** `myna.core.protocol` (`PROTOCOL_VERSION="1"`, `SUPPORTED_PROTOCOL_VERSIONS`, `is_supported`). **Decided field-in-handshake, not subprotocol token** — versioning must be transport-agnostic (the invariant), so it travels in band in `session.start` and works over loopback/ws/future transports, not just WS. Client declares `protocol_version`; server validates → terminal `transcription.error(unsupported_protocol_version)` on mismatch (feeds T31), else acks `session.created` echoing the served version (captured on `_WsSession.protocol_version`). Missing version = compatible (pre-versioning clients). Single number versions the whole contract incl. the event vocab (additive event types are breaking — `event_from_wire` rejects unknown). `capabilities.query` kept (T24). Tests: `tests/test_protocol_version.py` (3, ws-specific — loopback can't disagree); contract suite green over both transports |
| T36 | Event-vocabulary reconciliation: keep `transcription.*` vs adopt OpenAI `conversation.item.input_audio_transcription.*` names; reconcile our `progress.snippet` (liveness, uncommitted) with IE115 `delta` (incremental *committed* text) | done | Claude/Charles | T34 | **Decided 2026-06-17 — keep flat `transcription.*`.** No code rename; the decision *is* the deliverable, documented in `core/events.py` with the IE115/OpenAI → ours mapping (delta→`progress` *snippet-as-liveness, not committed*; completed→`final`+`done`; failed→`error`; single error channel). `progress.phase=preparing` (T26) preserved — IE115 has no model-loading event. Committed-delta stream deferred to streaming (T08): we emit no committed deltas yet, so don't build the semantics until a model needs them. Vocab versioned via `PROTOCOL_VERSION` (T35) |
| T37 | Session lifecycle/config envelope: align `SessionConfig`/`session.start` with an IE115-shaped `session.update`/`session.created` exchange, stripped of s2s fields (voice, tools, instructions, output_modalities, create_response) | done | Claude/Charles | T34, T24 | **Decided 2026-06-17 — keep flat config, add the `session.created` ack (via T35).** No nested `session.audio.input.transcription` envelope: that nesting only exists to co-locate s2s output config (voice/speed) we don't have. Existing fields (language, output_language, prompt, timestamp_granularity) cover transcription; decision documented in `core/session.py`. No `turn_detection` field — turn detection is client-driven via `session.finish`, never server VAD (deviations §1.4). `session.created` handshake reply landed in T35. **Superseded for the wire (2026-07-02, T44):** the team overruled the flat-config pushback for compatibility, so the IE115 *dialect* (T44–T47) speaks the nested envelope on the wire; the internal `SessionConfig` stays flat and a codec translates. This task's decision stands for the *internal* vocab |

### IE115 wire dialect — hands-on implementation (from 2026-07-02)

Decided (Charles, 2026-07-02): implement IE115 on **both** the Python
`myna-server` and the Rust client ourselves — not waiting for Ivano — to get
concrete experience feeding the open design questions (error taxonomy, overload
signal, STATUS shape, object graph). Built as a **selectable wire dialect**: the
flat `myna.core` vocab stays the semantic core (contract tests untouched); a
codec translates flat ↔ OpenAI-nested on each end; dialect chosen by **shape-sniff**
(first frame `session.update` = IE115, `session.start` = internal). This cashes in
the wire-agnostic FSM (T40) and unblocks the Rust IE115 client (T43) without
Ivano's server. Contract in `docs/architecture/ie115-wire.md`.

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T44 | IE115 wire-dialect design note: exact frame contract (session/config/event mapping, STATUS liveness, error mapping, base64/binary hatch) + open questions | done | Claude/Charles | T34, T36, T37 | `docs/architecture/ie115-wire.md` (2026-07-02). Pins every frame both directions, realises the `events.py` mapping, flags 8 open questions code will force (dialect selection→**shape-sniff, decided**; format completeness; `done` semantics; STATUS shape; overload/lag; object-graph minimalism; `include` tokens; `prompt` placement). Takes the position `model_loading`-as-error is wrong → `STATUS{loading}` instead |
| T45 | Python `myna-server`: IE115 codec (`myna.core.wire_ie115`) + shape-sniff dialect dispatch in `transport_ws`; internal vocab untouched; wire-parity test suite mirroring `tests/test_contract.py` | done | Claude/Charles | T44 | **Done 2026-07-02.** `core/wire_ie115.py` (`Ie115Encoder`/`Ie115Decoder`, config + append codecs); `transport_ws` shape-sniffs the first frame (`session.update`=IE115, `session.start`=internal, `capabilities.query`=caps) and dispatches to a shared `_pump_session`; `WsUnixIe115Client` added. Server emits `session.created`/`session.updated`; binary + base64 `append` accepted; STATUS additive; synthesises `item_id`/`content_index`; `done` synthesised client-side on close. `tests/test_ie115_dialect.py` — 15 tests (codec units + end-to-end parity over the fake adapter; the lossy `adapter_crash`→`server_error` mapping is asserted as T31 evidence). Full suite 119 green. Smoke-tested against live whisper (binary + base64, exact transcript) |
| T47 | **Persistent IE115 connections** (OpenAI multi-commit shape, decided 2026-07-06): server no longer closes after a terminal — each `commit` is answered by its own `completed` and the connection stays open; mapping change `final`↔`delta` (committed, append-only), `done`↔`completed` (per-utterance `item_id`); clients close themselves after their commit's `completed`; close-before-`completed` decodes as a `connection_closed` **error**, never a synthesised `done` (kills the silent-truncation hazard) | done | Claude/Charles | T45, T43 | **Done 2026-07-06.** Python: `wire_ie115` codec + multi-utterance loop in `transport_ws._handle_ie115` (per-utterance adapter session; early-bailing adapter drains to the commit boundary; missing-terminal guard so a compat client can't hang). Rust: stateless `decode_frame`, done-synthesis removed (close-before-terminal → FSM `connection_closed` failure), client closes after `completed`. Internal dialect unchanged (one-shot, documented). Tests: multi-commit e2e (py), IE115 e2e vs live server (rs); wire note §4/§7.4 amended. Our client stays connect-per-press (one commit per connection — valid degenerate case); cloud backends may hold connections open later, behind `BackendClient` |
| T46 | IE115 loopback demo + append micro-benchmark: Rust client ↔ Python server over IE115 across whisper/nemotron/qwen; measure per-`append` CPU/alloc for binary vs base64 (review comment `[h]`) | done | Claude/Charles | T45, T43 | **Done 2026-07-02.** Rust `myna-dictate --dialect ie115` ↔ Python IE115 server verified end-to-end across all three families (whisper/AED, nemotron/transducer, qwen-c/LLM-decoder) — exact transcripts, binary and `--base64-audio` paths identical, proving the dialect is genuinely model-agnostic. Append micro-bench (100 ms 16 k mono s16le chunk): base64+JSON = **1.35× wire inflation** and ~16 µs CPU/chunk (9 µs enc + 7 µs dec) vs ~0 for binary → +11 KB/s + ~160 µs/realtime-second. Negligible per session, real at fleet scale — recorded in `docs/architecture/ie115-wire.md` §5 as the argument for binary-default |
| T48 | **Backend discovery & model selection** — the missing subsystem between workstreams B, D, and G: three snaps, three sockets, each serving one fixed model chosen out-of-band via `modelctl`, and nothing owns *which sockets exist, what's behind each, who chooses*. Design the enumeration/selection layer above the orchestrator (likely: capabilities-query each known socket; map model→socket; the session controller picks) | todo | | T24, T21 | Surfaced by the 2026-07-05 architecture review. The session-level half is fixed (2026-07-06): the IE115 server now **rejects** a `session.…transcription.model` it doesn't serve (`model_not_available` → `invalid_request_error/invalid_parameter`) instead of silently transcribing with whatever it has — so a compat client can probe, but nothing *selects* yet. Keep one-model-per-process (clean residency semantics); this task owns the layer above, plus the sunset question's sibling: whether the internal dialect retreats to loopback/testbed-only once IE115 is the team wire (recorded intent, 2026-07-06 — avoid a permanent double surface of tests/error-mapping/docs on the socket) |
| T63 | **canonical/whisper-snap interop fixture**: run the colleagues' Go IE115 adapter + Collabora WhisperLive backend as a live streaming IE115 peer; measure partial-revision traffic shape, commit→final latency, multi-`completed` behavior, WER on the real corpus; feed results into the T08 streaming-mode spec | todo | Charles | T43 | Reviewed 2026-07-26 (`reference/whisper-snap`; snap packaging in PR #2 `build/snap-packaging`). Their snap: strict, modelctl components, engines — but TCP loopback + `network` plug (no UDS content share), unpinned git-`main` WhisperLive build, no idle-unload, and a **snap-name collision** (`whisper` — settle ownership early). Wire deltas vs our dialect: base64-only appends (our client: `--dialect ie115 --base64-audio`); single commit per connection; `session.created` gated on backend-ready; `model.loaded`/`model.unloaded` liveness (≈ our STATUS, model-centric); **no `item_id`** on delta/completed; partial **revision encoded as an empty `completed`** (reset) + full re-send; mid-utterance `completed`s at segment boundaries (our client treats the first post-commit `completed` as terminal — expect truncation on long clips); suspected ~15 s idle-wait in the finalize path. Its streaming partials make it the reference fixture for the streaming protocol *before our server grows partials* — the experiments here decide what the wire's committed/unstable discriminant must be. **Observed 2026-07-26 (first live run):** their own debug client **panics** (`context deadline exceeded`, hardcoded 180 s overall timeout) waiting for `session.created` while the container downloads/loads a model over a slow network — their wire has **no liveness signal before `session.created`** (`model.loaded` is only sent once the backend is fully ready), so a slow load is indistinguishable from a dead server and the failure surfaces as an opaque client-side timeout. Direct empirical validation of our STATUS{loading} + server-speaks-first greeting; cite in the streaming spec |

## Workstream G — Orchestrator subsystem (Rust)

The client-side coordinator: it owns the **two-region async FSM** from
`docs/architecture/ie115-lifecycle.md` — the per-connection session track
(CREATED→ACTIVE→FINALIZING→DONE) running orthogonally to model residency
(UNLOADED→LOADING→RESIDENT) — with the accept-gate (`ACTIVE ∧ RESIDENT`),
commit-drain (COMMIT ≠ done; wait for `done`), pre-ready-audio drop, and
terminal-vs-recoverable error mapping. The FSM is the deliverable; everything
else is plumbing around it.

**Buildable now via mocks (our earlier work):** the existing Python `myna-server`
(ws+unix, fake/whisper/nemotron/qwen adapters) stands in for the IE115 inference
snap; a WAV source over `corpus/real` for the audio adapter; stdin for the
hotkey; stdout for the injector. Design decisions (2026-07-01, Charles): keep the
FSM **wire-agnostic** and speak the *existing* `myna.core` wire first (real
end-to-end runs against a running Python server today), with **IE115 event names
as P4**; enhance the Python fake adapter to emit the full `STATUS` sequence so
the residency gate is honestly exercised; small `core/orchestrator/cli`
workspace; **rigorous** FSM (all async edge cases, not a thin loop); Python
server kept as a test dependency (Rust integration tests spawn it).

Stack: Rust (nightly present), tokio + tokio-tungstenite + serde. Honors the
invariants: never persist audio, bounded in-memory buffering, no content logged.

| ID | Task | Status | Owner | Depends on | Notes / acceptance |
|---|---|---|---|---|---|
| T38 | **P0** — Cargo workspace scaffold + `myna-core` crate: `AudioFormat`/`PcmChunk` (per `docs/audio-adapter-api.md` §2), event/session/protocol types + serde wire codec mirroring `myna.core`, `PROTOCOL_VERSION="1"` | done | Charles | — | **Done 2026-07-01.** `client/` workspace (`myna-core` + `myna-orchestrator`/`myna-cli` stubs); tokio not yet pulled (T39). `myna-core` covers audio/events/session/protocol/control with a JSON codec verified against **golden frames captured from Python `myna.core`** (compared as parsed JSON values — key order/whitespace differ, structure matches; both ends parse JSON). 25 unit tests green, clippy clean. Includes the new `ready` phase (T42) |
| T39 | **P1** — `BackendClient` trait + WS-over-UDS impl speaking the **existing** `myna.core` wire (`session.start`→binary PCM→`session.finish`; parse `session.created` + `transcription.*`) | done | Charles | T38, T14a | **Done 2026-07-01.** `myna-orchestrator::backend`: `BackendClient` trait → `BackendHandle` split into a cheap-clone `BackendSink` (audio/finish/abort up) + `BackendEvents` receiver (down), decoupling the FSM from the wire via channels so it can push audio and read events concurrently. `WsUnixBackend` (tokio + tokio-tungstenite over `UnixStream`) does the handshake (declares `protocol_version`, awaits `session.created`, maps a rejection error to `BackendError::Rejected`), then a pump task bridges the split WS to the channels (terminal-event/close/abort teardown; commit-drain: sink-drop keeps reading). Integration test spawns the real `myna-server --adapter fake` on a UDS and asserts the full scripted session round-trips (26 tests green, clippy clean). Also **added `--adapter fake` to the server CLI** (serves the permanent fixture over the wire, zero deps — reused by T42). Test launches the venv server binary **directly** (not `uv run`, which orphans a grandchild) so cleanup is reliable |
| T40 | **P2** — the FSM (centerpiece): two orthogonal regions + accept-gate, commit-drain, pre-ready drop, terminal/recoverable error mapping (enum; taxonomy provisional pending T31). Fake-backend fixture emitting the full `STATUS{loading→ready→transcribing}` + edge cases | done | Charles | T38 | **Done 2026-07-01.** Split into a **pure synchronous FSM** (`myna-orchestrator::fsm`) — `(state, input) -> (state', actions)`, no I/O — and a thin async **driver** (`::driver::run_session`) that only pumps client inputs + backend events through it. Two orthogonal regions: session (`Active→Finalizing→Done`/`Failed`/`Aborted`) × residency (`Unloaded→Loading→Resident`), with the accept-gate (`Active ∧ Resident`), commit-drain (`Finalizing` is non-terminal; only `done` completes), pre-ready drop (`AudioDropped{NotResident}`), residency **re-lapse** (idle-unload mid-session re-closes the gate — open item #4), and a provisional `ErrorDisposition` enum (`Terminal`/`Recoverable`/`Advisory`) via `classify_error` (only `not_ready`→advisory + `overloaded`→recoverable mapped; rest terminal, pending T31 — **deleted 2026-07-06**: unreachable string-match guesses; every error is terminal until T31 puts the disposition on the wire, see the T31 row). Fixture: `backend::fake::FakeBackend` (Rust mirror of the Python `FakeAdapter`, permanent regression asset) — scripted, model-free, emits the full `STATUS` liveness + `happy_path`/`commit_drain`/`mid_stream_error`/`slow_ready` scripts; `WaitForFinish` step stages §3C precisely. **Edge cases 3A/3B/3C each have deterministic pure-FSM tests** (17 FSM tests) + 4 async driver tests over the fixture; clippy clean. Wire-agnostic — the FSM never touches a socket, so IE115 names (T43) are a backend swap, not a rebuild |
| T41 | **P3** — boundary mocks + demo bin: `AudioSource` trait + `WavFileSource` (real-time paced, `corpus/real`), `Trigger` (stdin), `TextSink` (stdout); wired into the `myna-cli` orchestrator binary end-to-end against the real Python server | done | Charles | T39, T40, T20 | **Done 2026-07-01.** Three boundary traits, each with a mock, matching the audio-adapter API so real impls drop in: `audio::AudioSource` (§3 signature: `format()` + `capture()→Stream`, `CaptureError` §4) + `WavFileSource` (minimal RIFF parser, real-time pacing, graceful `StopHandle` per §5); `trigger::Trigger` (+`StdinTrigger` two-Enter flow / `ScriptedTrigger`) for the T21 hotkey; `sink::TextSink` (+`StdoutSink` / `CollectingSink`) for the T22 injector. `runner::run_dictation` composes source→FSM driver→sink for one utterance (clean end→`EndOfAudio`/finalize, fault→`Abort`; negotiates the source format into the config). `myna-dictate` binary wires them for push-to-talk (`--socket`/`--clip`/`--corpus`), Release→graceful-stop via `select!`. The runner **gates capture on the first `ready`** (holds the pump until residency, `notify_one`-based, aborts on session end so a load-time error can't hang) — so a paced/replayable source can't drain into a closed accept-gate during a slow cold load (the client half of ie115-lifecycle §3A; pairs with T42's adapter-emits-`ready`). **Verified end-to-end against a live Python `myna-server`** — `--adapter fake` (`tests/dictation_e2e.rs`, skip-friendly like T39) *and* `--adapter whisper --model base` manually (exact real-speech transcript, zero dropped audio). 29 orchestrator tests + clippy clean. Independent of the audio-adapter crate (the T49–T52 `myna-audio` crate replaces `WavFileSource` behind the same trait) |
| T42 | Adapters emit `ready` after load, before consuming audio, so the residency accept-gate opens over the wire (fake **and** every real adapter) | done | Charles | T02 | **Done 2026-07-01 — was a correctness bug, not just mock fidelity.** Manual `myna-dictate`→whisper runs surfaced a **deadlock**: the whisper/nemotron/qwen adapters emitted `preparing` heartbeats during load but only emitted `transcribing`/`final` *after* receiving audio, while the client's accept-gate (T40) drops audio until it sees `ready` — so no audio ever flowed and the session finalized empty. Per *fix-the-adapter-not-the-harness*: all three real adapters now `emit(TranscriptionProgress(phase=PHASE_READY))` right after `_load_model_with_heartbeat`, before the `async for chunk in audio` loop; the fake adapter's `default_script` opens with `preparing→ready` too. Also re-exported `PHASE_READY` from `myna.core` (was defined in `events` but not surfaced). Vocab (the `ready` phase) had already landed. Second half in T41: the runner now **holds capture until the first `ready`** so a slow cold load can't drain a paced source into the closed gate. Regression: `fake.rs::runner_gates_all_audio_until_ready` (probe backend asserting no audio precedes `ready`); verified live end-to-end against `whisper --model base` (exact transcript, zero drops). 103 py + 28 orch tests green |
| T43 | **P4** — IE115 `BackendClient` (Rust): a second backend speaking IE115 names (`session.update`/`input_audio_buffer.append`/`conversation.item.input_audio_transcription.*`/`STATUS`) behind the wire-agnostic FSM; `--dialect ie115` on `myna-dictate` | done | Charles | T40, T45 | **Done 2026-07-02.** `backend::ws_unix_ie115::WsUnixIe115Backend` — a second `BackendClient` behind the *unchanged* FSM (the trait boundary held: only the backend swapped). Sends `session.update` eagerly, PCM as binary by default / base64 `append` with `.base64_audio(true)`, `input_audio_buffer.commit` on finish; `Decoder` maps STATUS→progress, `completed`→final, `error`→terminal, synthesises `done` on close. `--dialect internal\|ie115` + `--base64-audio` on `myna-dictate` (generic run loop over both backends). 5 unit tests + workspace clippy clean; verified against live whisper/nemotron/qwen-c (see T46) |

---

T33 (audio sample-encoding) is the audio half of this reconciliation — IE115's
fixed 24 kHz mandate makes the "negotiate format via `input_formats`, don't
hardcode" position concrete. T31 (error-code taxonomy) is unchanged by IE115:
IE115's `error` has `type`/`code`/`message` but, like IE114, defines no code
set — same gap.

---

## Measured findings (synthetic fixture tier, 2026-06-14)

First model×hardware matrix on the synthetic espeak corpus (13 clips, commit-on-finalize).
Latency is content-independent and **trustworthy**; WER is **directional only** and, as
point 3 shows, actively misleading across architectures.

| config | WER% | median finalize | cold load | note |
|---|---|---|---|---|
| `nvidia-gpu/small` | 11.07 | 0.150s | 0.91s | only config meeting both accuracy + UD129 latency |
| `cpu/small` | 11.07 | 1.685s | 2.21s | p90 3.4s > UD129 1–2s target — `small` not viable on CPU for live |
| `cpu/tiny` | 37.27 | 0.295s | 0.51s | fast but inaccurate |
| `nemotron` (default ctx) | 59.04 | 0.027s | — | native transducer — fastest by far; synthetic WER unreliable (point 3) |

1. **Native transducer wins latency decisively.** Nemotron finalize ~0.027s (each frame
   processed once, no end-of-utterance re-decode) vs AED Whisper 0.15s (GPU) / 1.7s (CPU).
2. **Whisper CPU vs GPU WER is identical** (11.07%, bit-for-bit per category): GPU is a pure
   latency play, not accuracy.
3. **Synthetic audio cannot fairly compare architectures.** Nemotron scores 59% WER on
   espeak yet transcribes real human speech perfectly (verified via `dev/dictate.py`). espeak
   is severely OOD for models trained on real speech; Whisper's web-scale training masks this.
   → real WER claims need a recorded-speech corpus (**T25, now critical path**); synthetic tier
   stays valid for plumbing + latency only.
4. Nemotron is **English-only** (`non-english-german` 100%, as predicted).

## Real-corpus finding (LibriSpeech tier, 2026-06-14)

First run on the **real recorded-speech** tier (T25, `corpus/real/`), Nemotron,
commit-on-finalize, vs the synthetic espeak tier under the *same model and code path*:

| tier | clips | mean WER% |
|---|---|---|
| real (LibriSpeech dev-clean, clean read speech) | 6 | **0.0** |
| synthetic (espeak, quiet + long-form) | 3 | **44.6** |

The gap is the whole point of T25: the synthetic-tier WER (44.6%, incl. 100% / empty
output on the pangram) is an artefact of espeak being out-of-distribution, **not** a
model deficiency — the same model is flawless on real voice. WER claims must come from
the real tier; the synthetic tier remains valid for plumbing + latency only. (Accents
still uncovered — clean read speech only; Common Voice/VCTK is the follow-up.)

## Milestones

1. **M0 — Contract verified (done, 2026-06-12):** T01–T02. Fake adapter +
   harness over loopback; contract tests green.
2. **M1 — Real audio, real model:** T03–T07. One real candidate measured
   end-to-end on fixture audio.
3. **M2 — Streaming matrix:** T08–T11. Streaming candidates compared;
   evaluation matrix produced.
4. **M3 — Spec convergence:** T16, T18, T19, T12. IE114 updated with
   transport, events, and measured performance contract.
5. **M4 — End-to-end dictation:** T14a–T14c, T20–T22. Hotkey → speech → text
   in a GNOME app via the whisper snap.
6. **M5 — Orchestrator loop (Rust):** T38–T42. Trigger → WAV → transcription
   through the wire-agnostic FSM against the real Python server; IE115 wire (T43)
   follows once Ivano's server lands.
7. **M6 — Dual-mode streaming:** T08, T63. Streaming alongside batch,
   user-configurable and hardware-tier gated. Whisper local-agreement and
   Parakeet chunked commit are shipped; sherpa is packaged; nemotron native
   streaming and the concluding report remain.
