1. The Complementarity Rule: Speech for Intent, Agents for Syntax
Effective voice coding does not mean reading curly braces and semicolons aloud. Instead, voice acts as a high-bandwidth channel for architectural intent while coding agents translate that intent into clean, type-checked diffs.
Quick Navigation
Push-to-Talk or Silero VAD slices 16kHz PCM audio cleanly, rejecting cooling fans and mechanical keyboard clicks before inference runs.
Dictionary glossaries, active editor buffers, and lightweight LLMs strip filler words and restore exact repo identifiers and camelCase symbols.
Clean transcripts enter Claude Code, Antigravity, Codex, or an Obsidian +Inbox/ watcher to produce plans, code diffs, or tickets.
Architecture: Microphone → VAD Gate → ASR + Glossary → Cleanup LLM → Agent / Vault
2. Acoustic Setup, VAD Gating & Dictation Engine Matrix
Transcription accuracy depends just as much on microphone proximity and silence detection as on model size. Choose an engine tier that matches your latency target, privacy boundary, and operating system.
| Engine / Tool | Execution Mode | Latency & Cost | Developer Vocabulary Support | Best Engineering Fit |
|---|---|---|---|---|
mlx-whisper / parakeet-mlx |
100% Local Apple Silicon MLX | 180ms – 450msFree / OSS |
224-token initial_prompt + custom mlx-lm cleanup pass |
Air-gapped proprietary repos, custom Python hotkeys, privacy-critical Macs. |
Handy (cjpais/Handy) |
100% Local Tauri v2 + Rust | 200ms – 500msFree / MIT OSS |
GGUF Whisper & Parakeet V3 + optional Ollama post-processing | Cross-platform (macOS, Windows, Linux) offline push-to-talk with Silero VAD. |
| Superwhisper | Local-First + Optional Cloud | 200ms – 600msFree / Pro Paid |
Custom Modes, SQLite history, and CLI/MCP vocab self-sync |
Agents auditing past dictations to auto-patch misheard repo terms. |
| Wispr Flow | Cloud TLS 1.2+ Stream | 150ms – 350msBasic / Pro $12–15 |
Active IDE buffer context + voice file tagging + snippet triggers | Multi-device speed, Cursor/VS Code symbol awareness, zero local GPU setup. |
Claude Code /voice & Codex F8 |
Native Agent CLI Built-In | 250ms – 500msIncluded in Plan |
Claude primes on repo + Git branch; Codex uses F8 two-way voice | Zero-install terminal sessions (requires local mic; not plain remote SSH). |
| Talon + Cursorless | Local Grammar + AST Engine | < 80msFree / Beta |
Full spoken phonetic alphabet, symbol grammars, AST node targeting | Hands-free RSI recovery and surgical edits without touching a keyboard. |
Microphone Physics, Silero VAD & Barge-In Mechanics
Keep a dynamic cardioid mic or headset boom 5–10cm off-axis from your mouth. Distant laptop array mics pick up fan roar and tactile switch clicks that corrupt word boundaries.
Legacy WebRTC VAD checks raw energy and misfires on mechanical keys. Compact neural Silero VAD (~2MB ONNX) scores 32ms chunks in 1ms with robust noise rejection.
Require 350–500ms of continuous silence before closing an utterance so thinking pauses survive. Full-duplex agents increment an epoch ID on barge-in to cancel stale outputs.
Fn, Left Option, or Spacebar in /voice) wins on predictability: you never leak off-hand remarks or coughs into an active agent terminal. Reserve continuous Silero VAD for full-duplex conversational modes, walking capture memos, or hands-free RSI setups paired with a hardware mute switch.
3. Annotated Workspace: Voice-to-Agent HUD & Vocabulary Panel
Modern voice coding pairs a lightweight acoustic capture HUD with live IDE context awareness and custom replacement dictionaries so spoken prompts land cleanly in your agent terminal.
verifyJwt.ts (feat/oidc-pkce)
src/auth/verifyJwt.ts and verify that the redisKeyTtl stays fixed at 60s."jay double u tee" → JWT"cube control" → kubectl"pkce" → PKCE • "verifyJwt.ts""redis key ttl" → redisKeyTtlsk_live_... → [REDACTED_TOKEN]mlx-community/distil-whisper-large-v3
Log: ~/.voicebridge/logs.sqlite (24h purge)
Control Legend & Inspection Points
1 Trigger & Silero VAD Telemetry: Confirms whether Push-to-Talk or automatic VAD is active, showing live speech probability and trailing pause tolerance.
2 Active IDE Context Lock: Reads open file paths, visible symbols, and Git branch names to bias transcription toward exact project terminology.
3 Live Stream Latency Budget: Tracks end-to-end audio capture, inference, and formatting latency; keep under 500ms so voice feels instantaneous.
4 Backtrack & Filler Normalization: Strips verbal hesitation and honors self-corrections such as "no wait" or "scratch that" before pasting.
5 Target Agent Harness Router: Injects the finalized prompt directly into Claude Code, Antigravity, Codex CLI, or your Obsidian inbox buffer.
6 224-Token Glossary & Alias Table: Displays active Whisper initial_prompt budget usage alongside deterministic regex symbol replacements.
7 Privacy Fence & Secret Scrubber: Automatically disables cloud context scraping inside .env buffers or sensitive repos and masks spoken secrets before logging.
4. Spoken Spec-to-Plan Prompting for Coding Agents
Rambling voice prompts force coding agents to guess file boundaries and invariants. Structuring a 30-second spoken recording into five crisp slots turns conversational speech into an unambiguous engineering specification.
Color-Coded Formula: The 5-Slot Spoken Engineering Spec
[1. Context & Mode] [2. Goal & Behavior] [3. Target Files & Symbols] [4. Invariants & Non-Goals] [5. Verification Gate]
Example spoken aloud into claude (Plan Mode), Antigravity, or Codex:
"Enter Plan Mode first, do not edit yet."
"Add idempotency key deduplication to webhook ingestion so duplicate Stripe retries return 200 without double-charging."
"Start in src/webhooks/stripeHandler.ts and reuse our existing Redis cache helper."
"Keep the existing signature middleware untouched and expire keys after 24 hours."
"Finish by running the webhook Vitest suite and showing the exact diff plan."
Raw Voice Dump vs. Normalized Spec Block
# Spoken transcript (22 seconds at 175 WPM) -> cleaned automatically by local Qwen / Superwhisper Mode:
"Plan first before editing. Add idempotency-key deduplication to the Stripe webhook handler in
src/webhooks/stripeHandler.ts using our existing Redis helper. Preserve current HMAC signature
verification unchanged, set a 24-hour TTL on processed event IDs, and verify with Vitest."/voice + Plan Mode
Toggle /voice in Claude Code (Claude.ai Pro, Max, Team, or Enterprise) and hold Spacebar while speaking. Claude automatically passes your repository and active Git branch names as speech recognition hints.
Pro pattern: Press Shift+Tab into Plan Mode before speaking so long multi-file specs are reviewed as a blueprint before a single file changes.
In Google Antigravity, dictate architectural specs directly into the planner or spawn parallel investigator subagents to check impacted modules while you keep speaking follow-up constraints.
In OpenAI Codex CLI, pair external dictation (Wispr Flow, Superwhisper, or Handy) for exact CLI flags and ~/.codex/config.toml paths, or toggle F8 for conversational voice steering.
Spoken Routing Cues Across Agent Harnesses
| Spoken Opening Cue | Target Mode | Why This Phrase Prevents Wasted Edits |
|---|---|---|
| "Plan first, do not edit" | Claude Code Plan Mode | Locks write tools until you inspect proposed files, types, and migration boundaries. |
| "Fan out across modules" | Antigravity Subagents | Triggers parallel read-only investigators to map call sites before touching code. |
| "Read-only audit only" | Codex CLI Sandbox | Confines execution to dry-run checks and diff inspection before applying patches. |
5. Custom Code Vocabularies & Whisper Prompt Priming
Out-of-the-box speech models trained on podcasts routinely mangle programming jargon. Priming the decoder glossary, wiring replacement rules, and letting agents audit mis-transcriptions eliminates manual typo fixes.
# Priming mlx-whisper with a strict <= 224-token codebase glossary + syntax formatting cues
import mlx_whisper
GLOSSARY = (
"Stack glossary: kubectl, PostgreSQL, FastAPI, TypeScript, Zod, Pydantic, Supabase, "
"useEffect, verifyJwt, redisKeyTtl. Syntax cues: camelCase, snake_case, =>, !=, ||."
)
result = mlx_whisper.transcribe(
"prompt_clip.wav",
path_or_hf_repo="mlx-community/distil-whisper-large-v3",
initial_prompt=GLOSSARY,
)
Whisper caps initial_prompt at 224 tokens; longer lists are silently truncated from the front. Audio longer than 30 seconds also loses initial priming unless you re-inject prompts per chunk (e.g., carry_initial_prompt=True in faster-whisper).
Keep only the top 30–40 repo-specific acronyms, database tables, and package names inside your active prompt budget.
Superwhisper exposes a local CLI and MCP server (superwhisper vocab, history, mcp). Connect it to Claude Code or Antigravity so your agent scans recent transcripts against repo symbols and writes missing replacement rules automatically.
On Wispr Flow, enable IDE context awareness so visible file tabs automatically bias variable spelling in Cursor and VS Code.
6. Voice-Driven PR Diff Walkthroughs & Terminal Triage
Code review is predominantly a reading and reasoning task. Speaking aloud as you scan a pull request lets you direct agent pre-reviews, classify noisy bot comments, and draft crisp technical replies in seconds.
Tell the agent exactly what to ignore (lockfiles, formatting) and where risk lives ("audit null safety in session.ts and race conditions in checkout").
Pull unresolved comments via gh pr view and speak a triage instruction to group critiques by root cause as Valid, Out-of-Scope, or Invalid.
Dictate your blunt architectural rationale aloud and let the agent translate it into a polite, file-and-line-cited GitHub PR response.
Terminal Triage Pipeline: Spoken Focus Prompt + GitHub CLI
# 1. Fetch clean diff and open comments, then dictate focus areas via /voice or Whisper HUD
gh pr diff 418 --patch > /tmp/pr_418.diff
gh api repos/{owner}/{repo}/pulls/418/comments --jq '.[] | {path, line, body}' > /tmp/pr_418_comments.json
# 2. Spoken instruction pasted into Claude Code (/code-review) or Antigravity:
# "Walk through pr_418.diff ignoring generated snapshot updates. Validate every comment in
# pr_418_comments.json against the real files, rank findings by severity, and draft a concise
# technical reply explaining why the partition key already prevents duplicate reads."
When a 400-line test log fails, resist typing manual grep commands. Speak the symptom directly: "Ignore the lint warnings, inspect why test_auth_refresh timed out on CI, and tell me if the mock clock drifted."
For terminal apps over SSH where native /voice lacks mic access, use local system dictation (Handy, Superwhisper, or Wispr Flow) to paste cleanly into your tmux pane.
When using spoken output modes (such as Codex F8 voice conversation), configure client-side Silero VAD barge-in so interrupting a verbose explanation immediately halts audio playback and cancels pending tool calls via epoch IDs.
Always confirm destructive shell commands (git reset, rm, DB migrations) visually before pressing Enter.
--force or destructive glob wildcards before execution.
Spoken Review Triage Taxonomy
Verified bug or missing guard confirmed directly in source lines; dispatch patch immediately.
Legitimate refactor unrelated to PR goal; convert by voice into a follow-up GitHub issue.
False-positive linter or bot note; dictate a brief explanation citing existing invariants.
7. Voice-Memo-to-Obsidian +Inbox/ & GitHub Pipelines
Your sharpest architectural ideas rarely happen at your desk. An automated capture pipeline turns off-screen voice memos into linked Obsidian vault notes and actionable GitHub issues before you sit back down.
# macOS folder watcher (fswatch) -> mlx-whisper -> Obsidian +Inbox/ & Claude Code vault organizer
fswatch -0 ~/Sync/VoiceMemos | while IFS= read -r -d "" audio; do
base=$(basename "$audio" .m4a)
mlx_whisper "$audio" --model mlx-community/whisper-large-v3-mlx \
--output-format txt --output-dir ~/Vault/+Inbox/
claude -p "Read ~/Vault/+Inbox/${base}.txt using CLAUDE.md rules: clean fillers, add YAML \
frontmatter, inject [[wiki-links]], file into 01_Projects or 03_Resources, and create any \
requested GitHub issues with gh issue create --label voice-capture."
doneCLAUDE.md
Place a CLAUDE.md file at the root of your Obsidian vault specifying your folder taxonomy (+Inbox/, 01_Projects/, 03_Resources/), mandatory YAML frontmatter keys (date, type, tags, status), and bidirectional [[wiki-link]] lookup rules.
Alternatively, native Obsidian plugins such as Whisper or ReWrite let you record and post-process directly inside the editor.
Include a spoken routing cue at the start of walking memos—such as "Note plus GitHub task"—so the classifier saves rich research prose into Obsidian while filing concise, label-tagged acceptance criteria via gh issue create.
Keep processed raw audio archived outside Git repositories so binary .m4a clips never bloat source history.
8. Five Copy-Paste Recipes for Voice Coding & Capture
Practical blueprints you can deploy today—spanning zero-cloud Apple Silicon push-to-talk, agent-driven dictionary syncing, Obsidian inbox sorting, and multi-channel PR triage.
| Recipe & Goal | Recommended Stack | Core Command / Pattern | Guardrail & Output |
|---|---|---|---|
| 1. Air-Gapped Local Mac PTT Zero-cloud dictation in IDE |
mlx-whisper + mlx-lm + sounddevice |
mlx_whisper.transcribe(buf, path_or_hf_repo="mlx-community/ distil-whisper-large-v3") |
100% offline GPU memory; never leaves Mac; scrubs secrets pre-paste. |
| 2. Cross-OS Offline Dictation Linux / Windows / Mac desktop |
Handy (cjpais/Handy) + Parakeet V3 |
Enable Silero VAD + bind global PTT hotkey + optional local Ollama formatter |
Zero subscription fee; Metal/Vulkan accelerated; clean clipboard insert. |
| 3. Self-Healing Code Dictionary Auto-fix misheard symbols |
Superwhisper CLI/MCP + Claude Code | superwhisper history --limit 30Agent compares logs to repo symbols & updates superwhisper vocab |
Eliminates recurring typos across team-specific packages & acronyms. |
| 4. Spoken Spec-to-Plan Sprint Fast multi-file architecture |
Claude Code /voice or Wispr Flow + Plan Mode |
Hold Spacebar in Plan Mode; speak 5-slot spec; review diff plan before edits | No unreviewed file writes; 3x faster architectural context transfer. |
| 5. Walking Memo-to-Backlog Phone memo to Obsidian & GitHub |
Voice Memos + iCloud + fswatch + gh CLI |
Sync .m4a → local Whisper → claude -p sorts +Inbox/ & files gh issue |
Idempotent archive move prevents duplicate GitHub tickets on rerun. |
Copy-Paste Snippet: Local Push-to-Talk Pipeline with Silero VAD & Secret Redaction
import re, mlx_whisper, pyperclip
# Assumes 16kHz float32 audio clip captured via sounddevice and gated by Silero VAD (>= 0.5 prob)
SECRET_PATTERNS = re.compile(r"(sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16})")
GLOSSARY = "Kubernetes, kubectl, PostgreSQL, FastAPI, Pydantic, Antigravity, camelCase, snake_case"
def dictate_to_clipboard(wav_path: str) -> str:
res = mlx_whisper.transcribe(
wav_path,
path_or_hf_repo="mlx-community/distil-whisper-large-v3",
initial_prompt=GLOSSARY,
)
cleaned = SECRET_PATTERNS.sub("[REDACTED_SECRET]", res["text"].strip())
pyperclip.copy(cleaned) # Paste manually without firing auto-Enter in terminals
return cleaned9. Offline Privacy Zones, Secret Scrubbing & Team Etiquette
Bringing a live microphone into software engineering introduces real privacy, credential-hygiene, and workspace-etiquette stakes. Build hard local boundaries before dictating around proprietary repositories.
Cloud dictation tools (such as Wispr Flow over TLS 1.2+ or native IDE voice endpoints) offer rapid cross-device sync and optional 24-hour or zero audio retention. However, regulated healthcare, fintech, or pre-announcement codebases often prohibit external audio or screen-context uploads altogether.
Rule: Route sensitive projects exclusively through offline engines (mlx-whisper, Handy, or Superwhisper Local Mode) with cloud context reading disabled.
Accessibility-powered context awareness inspects active window buffers to spell variable names accurately. Always exclude password managers, .env buffers, browser admin consoles, and private terminal key dumps from window scraping.
Never speak API keys, passwords, or customer PII aloud while dictation history logging is enabled; apply automated regex redaction before saving transcripts to SQLite or Obsidian.
Security Rule: Local Redaction & Retention Policy
# Recommended ~/.voicebridge/privacy.yaml policy for engineering laptops
default_mode: local_mlx # Never route audio offsite by default
excluded_apps: ["1Password", "Bitwarden", "Keychain Access"]
excluded_file_globs: ["**/.env*", "**/*secret*", "**/*.pem", "**/credentials.json"]
history_retention_hours: 24 # Auto-purge local SQLite dictation logs daily
scrub_before_persist: true # Mask API tokens before Obsidian or log write- Prefer Short Bursts: Speak focused 10–20 second specs via Push-to-Talk in shared rooms rather than continuous monologue.
- Visual Mute Tally: Use a hardware mic mute button with an LED indicator before joining standups or pairing calls so voice triggers never hijack team meetings.
- Acoustic Isolation: Use quiet linear switches or headset noise-gating so typing corrections never leak into open-mic VAD streams.
- Combine Paradigms: Pair Talon Voice + Cursorless for hands-free window switching and AST token jumps with AI prompt dictation for multi-file feature authoring.
- Foot-Pedal PTT: Map a USB foot switch to your dictation hotkey so hands and wrists rest neutrally while dictating long specs or PR reviews.
- Reduced Keystroke Load: Speaking 150+ WPM intent slashes repetitive typing strain by over 70% on documentation and test scaffolding.
10. End-to-End Voice Synergy: From Walking Thought to Merged PR
Voice workflows shine brightest when capture, transcription, vocabulary normalization, agent planning, and knowledge archiving link into one continuous pipeline across mobile and desktop tools.
At the end of every sprint, run an agent pass over your recent transcripts (via Superwhisper MCP or local logs) comparing corrected commit symbols against raw speech text. Newly coined service names and flags flow automatically back into Step 2 so accuracy compounds every week.
Because spoken specs launch faster than typed prompts, pair voice-first agent loops with isolated sandboxes or Git worktrees. When you dictate two competing approaches, agents can prototype and benchmark both safely in parallel while you review the winners aloud.
Weekly Cross-Tool Cadence
| Cadence | Pipeline Trigger | Automated Outcome |
|---|---|---|
| Real-Time (Desk) | Hold Fn / Spacebar /voice |
5-slot spoken specs enter Plan Mode; agents execute tests in isolated sandboxes. |
| On-the-Move | Watch / Phone Voice Memo | iCloud + fswatch transcribes locally, links Obsidian notes, and opens draft issues. |
| Friday Review | superwhisper vocab + MCP |
Agent diffs recent commits against voice logs and updates custom code dictionaries. |
11. Troubleshooting Clinic & Pre-Flight Checklist
Fast, reliable fixes for the seven most common voice-coding bottlenecks, followed by a six-point readiness check before shipping sensitive work by voice.
| Symptom & Failure Mode | Root Cause | 10-Second Production Fix |
|---|---|---|
| End of glossary ignored | Whisper initial_prompt exceeded the 224-token ceiling and truncated silently. |
Trim list to your top 30–40 active repo identifiers; verify token count before injecting. |
| Jargon breaks after 30s | Whisper 30-second window boundary dropped your initial glossary priming. | Enable carry_initial_prompt=True in faster-whisper or re-seed every 25s chunk. |
| Prompt cuts off mid-thought | VAD silence timeout is too short for natural architectural pauses. | Increase Silero VAD trailing hysteresis window to 400–500ms or switch to Push-to-Talk. |
| Keyboard clicks trigger mic | Legacy energy-based WebRTC VAD treats mechanical switch clacks as speech. | Swap to ONNX Silero VAD (threshold 0.55) and use close-mic cardioid pickup. |
Claude /voice unavailable |
Running inside remote SSH or routed via raw API / Bedrock / Vertex keys. | Authenticate with Claude.ai plan locally, or use local Handy / Superwhisper / Wispr Flow. |
| Garbled symbol or CLI flag | Raw ASR transcribed spoken punctuation or package acronyms phonetically. | Add explicit syntax cues to glossary and map rules (e.g., "cube control" → kubectl). |
| Clipboard overwrites old paste | Fast consecutive dictations raced clipboard injection before IDE read it. | Disable auto-Enter in terminals; serialize paste calls with a 150ms clipboard lock delay. |
6-Point Pre-Flight Voice Workflow Readiness Checklist
□ 1. Acoustic Gain & Mute Tested: Close-mic level peaks cleanly between -12dB and -6dB, and your hardware mute toggle displays clear LED state.
□ 2. 224-Token Stack Glossary Seeded: Top repo directories, database models, CLI verbs, and casing conventions sit inside your active prompt budget.
□ 3. Privacy Fence & Scrubbing Verified: Regulated repos default to offline mlx-whisper or Handy, and .env / password apps are excluded.
□ 4. No Auto-Enter in Live Shells: Dictation pastes text cleanly into terminal panes without submitting unreviewed destructive commands.
□ 5. Plan Mode Guard Active: Long spoken multi-file prompts land in Plan Mode or isolated Git worktrees for structural review before applying edits.
□ 6. Capture Deduplication Verified: Voice-memo-to-Obsidian +Inbox/ watchers archive processed audio so reruns never open duplicate GitHub issues.