#40 feat(hands-free): put the side panel away and talk to the page through a mic you can drag anywhere

[Anonymous]

Originally created by: imshaikot

Closes [#39].

Hands-free mode. The side panel folds away into a mic on the page, and the user talks to the page directly: words stream in above the mic as they are said, a ring runs before sending, and a press discards, stops a run or opens an approval. A second's hover fans out an arc menu, a long press drags the mic anywhere, and hold to talk listens only while left Control is held.

Shape

• Speech runs in an offscreen document (dictation.html, reason USER_MEDIA, a new offscreen permission, Chromium only). With the panel closed there is nowhere else: the background has no DOM, and recognition in a content script would run on the page's origin — a prompt per site, and none at all under a Permissions-Policy that forbids the mic. The document exists exactly while the mic should listen, so creating it starts the mic and closing it stops it; recognition working there was probed before anything was built on it.
• Chrome runs one recognizer at a time. Measured: starting a second one aborts the first. So the panel and the mic never listen together (panelOpened ends hands-free, and the mic stays off while any panel's run port is connected), and an aborted the page did not ask for means another page took the mic: it yields rather than fighting for it.
• The background decides (src/lib/bridge/hands-free.ts). The mic listens for one tab: the one in front of a focused window, whose conversation has no run and no approval waiting, with the link up and no panel open. Every tab's orb gets a view, sent only when it changed, and only the listening tab receives transcripts. An instruction goes through the same instruct the panel's port takes (runCommand), so it lands in that tab's own conversation and takes the fast path as a typed one would; a finished turn's reply comes back through onTurnSettled.
• The orb (src/lib/handsfree/) is plain DOM in a closed shadow root, like the rail and the toast, with the panel's tokens, easing and token animation spelled out. The layout is pure (geometry.ts): the arc is centred on the way out from whichever edges the orb is against and always swept clockwise, so dragging it to another edge turns both the arc and the order it deals its items. Captions and the approval card go on whichever side fits, and the card's arrow follows the orb.
• Hold to talk reopens the dictation page as dictation.html?hold, which holds no microphone until the key is down. On release it calls stop(), not abort(), so the words already spoken are finished, and the orb sends the moment the page says held. The key is left Control, matched by event.code === 'ControlLeft': the same physical key on every layout, present on every platform, and inert on its own in every browser. Because Control is a modifier, a hold engages only after 250 ms with nothing else pressed, and any other key, a click, a wheel or a blur cancels it and discards what was heard.
• Only where speech works (speech-support.ts). A Firefox build compiles the check to return false (confirmed in the bundle). Brave is denied by brand: it has the recognizer constructor and no service behind it, so a plain feature check would pass it. Any other fork is written off by browsentic/speechService after a network failure before its first transcript, and works always wins. The panel hides the detach button and /hands-free behind it, and the background ends hands-free if it ever finds itself on anywhere unsupported.
• A-Eye skips overlays. The rail, the toast and the orb carry data-browsentic-overlay. A closed shadow root retargets a hit to its host, so the lens used to outline the host and pick it — and a click on one right after hovering the page picked the last page element hovered. It now draws nothing there and picks nothing, for the user's pick and the agent's alike.
• Only trusted events can open the mic or answer an approval. A page cannot reach inside a closed shadow root, but it can synthesize a keydown on its own window; the hold key and the approval buttons both check isTrusted.

State: browsentic/handsFree and browsentic/dictation in storage.session, so a restarted browser never comes back with a microphone on; browsentic/orbPosition, browsentic/pushToTalk and browsentic/speechService in storage.local. The socket protocol is unchanged at 19.

Verified, and not

Driven over CDP in headless Chrome for Testing 153, with a fake microphone, so Google's real speech service answered — it transcribed the test tone as words:

• detach from the header button and from /hand-free + Enter, and the fold captured frame by frame with every animation paused and seeked;
• the offscreen page starting and reaching listening, transcripts reaching the orb and streaming in, the ring, and the instruction arriving as a turn in the tab's conversation; a press during the ring discards it;
• the menu opening after a second's hover, turned at the right edge and in a corner; a long-press drag storing its position;
• working, approval, muted and needs-mic states; the approval card placed, arrowed and answered with a trusted click; a reply streaming in;
• A-Eye raised by the agent with the orb on screen: nothing outlined over it, and a click on it picked nothing;
• hold to talk with real key events: the page reopened in hold mode, Ctrl+C left the mic closed, a hold listened, and letting go sent;
• real Brave, headless on a scratch profile: no detach button, and a hands-free flag forced into storage was wiped at once;
• a service written off mid-session: the mic leaves with a toast saying why.

Not verified: a real voice in a headed browser; an approval raised by a real agent run end to end; Firefox at runtime, beyond the bundle check above.

Gates

1735 tests pass, coverage floors hold (shared lib 59.6% against a 37% floor), yarn compile and yarn daemon:compile are clean, and both builds pass. Each of the seven commits type-checks and passes the lib and DOM suites on its own. web-ext lint on the Firefox build: 0 errors and 14 warnings, where main has 11; the three new ones are the orb's escaped innerHTML templates, the same pattern the rail and the toast use.

Found along the way, not fixed here

• Vitest serves import.meta.env.FIREFOX as the string "false", which is truthy, so every if (import.meta.env.FIREFOX) in the repo takes the Firefox branch under unit test. speech-support.ts keeps its decision in the pure capableOf() so it can be tested; the other call sites are untouched.
• The toast's font: … inherit shorthands are invalid CSS, so the browser drops them and its text falls back to defaults. The orb had the same mistake, fixed there.
• Hold to talk is heard in the top frame only, because the content script runs in no other frame, so the key goes unheard while focus is inside an embedded frame (Google Docs' editor, for one). Documented as a limit.

Commits

• 3f5fbfc fix(a-eye): skip every element Browsentic draws into a page
• f9f22ef feat(voice): know whether this browser can transcribe at all
• f4f488f feat(hands-free): lay out the mic orb
• 9c63360 feat(hands-free): listen through an offscreen page once the side panel is closed
• c313970 feat(hands-free): draw the mic orb in a closed shadow root
• f0f34c4 feat(extension): detach the side panel into the hands-free mic
• ac4093a docs(hands-free): the feature page, which browsers get it, why left Control

🤖 Generated with Claude Code

[Anonymous]

Ticket changed by: imshaikot

• status: open --> closed