CMU's SoundBubble uses headset mics to create virtual fingertip microphone

SoundBubble, developed by Carnegie Mellon University researchers, creates a 'virtual microphone' that tracks a user's fingertip using an array of microphones mounted on an XR headset or smart glasses. The system calculates sound propagation delays to each microphone based on the headset's existing hand tracking, then aligns audio streams from the fingertip location while suppressing noise from elsewhere. In tests with 10 participants, the system achieved a 94.7% overall true positive rate, with particular strength in detecting ad hoc touch (97.5%) and passive widget interactions (97.1%). Simulations suggest an eight-microphone glasses-style array could retain most performance, pointing toward future AR glasses applications.

How SoundBubble isolates fingertip audio

• Uses time-of-arrival differences: sound reaches each headset-mounted microphone at slightly different times depending on fingertip distance
• Leverages existing 3D hand tracking from headset cameras to know precise finger position in space
• Applies delay-and-sum beamforming: shifts individual microphone streams to align fingertip-origin sounds in phase before combining them
• Off-axis sounds from other locations fail to align and are attenuated through destructive interference
• Demonstrated detection of sounds 20 dB below background noise levels including music, cafe noise, and speech
• Eight-microphone glasses-style array simulated to retain near-equivalent performance to 16-microphone setup; four microphones showed significant degradation

Why it matters

The technique could enable passive, uninstrumented objects to become interactive controls in AR environments without adding electronics, potentially simplifying hardware requirements for XR interfaces and reducing the sensor burden on users.