InvisibleButtons

What it is

I started this after seeing Holo, a macOS project that locates a tap by measuring the time difference of arrival between multiple microphones. Most Windows laptops only expose a single mono capture stream though, and the few that expose more usually run hardware or driver level noise suppression that wrecks the inter channel phase relationship that trick depends on. So that approach just does not work on Windows, and I had to find another way in.

Instead of locating the tap in space, I treat it as a timbre problem. A tap to the left of the microphone travels through a different amount of chassis and desk material than one to the right, so it ends up with a measurably different spectral envelope. I capture that envelope with MFCCs (mel frequency cepstral coefficients) and train a small scikit-learn classifier to tell the two apart, all from one microphone, on stock Windows audio.

How it works

On every tap, the pipeline runs like this:

mic → adaptive onset detector → 0.5s window → MFCC features → novelty gate → classifier → gesture → action

I split detection into a sensitive front end and a strict back end, because those two jobs actually pull in opposite directions.

Finding the tap. Comparing loudness against a threshold measured once at startup falls apart the moment the room changes. Turn on a fan and the detector either fires nonstop or, once you raise the threshold to cope, goes deaf. So instead it tracks the room continuously and triggers on the two things that actually characterise a tap: a jump in band limited energy, and a simultaneous jump in spectral flux, which measures how much the spectrum changed rather than how loud it got. Steady noise, however loud, barely moves the flux at all. The comparison is against the previous 70 milliseconds or so, not a long term average, and that is what tells a tap apart from a room simply getting louder.

Deciding it was a tap. A two class model has to answer "left" or "right" about anything you hand it, so without help a cough or a keystroke would get classified as a keystroke press. I put two gates in front of it. One is a novelty gate, which measures how far the current window sits from the calibrated taps in feature space and rejects anything further away than the calibration set is from itself. The other is an optional background noise class, recorded by typing, talking and moving around for a minute so the classifier can just answer "that was nothing" directly. On synthetic room benchmarks the front end went from 64 false triggers down to zero while a fan spun up, and the novelty gate rejects thuds, speech, keystrokes, claps and music (sounds that used to get classified as a confident left or right) without losing a single genuine tap.

Design notes

Loudness is thrown away on purpose. Tap windows get peak normalised before feature extraction, because how hard you tap varies far more between two taps on the same side than the left and right difference ever does. Amplitude is a trap, so the model works on timbre alone.

dB is measured against each window's own peak. Most of a 0.5 second window is near silence after the tap has died away. If you convert that to dB against a fixed reference, the silence ends up encoding the room's noise floor instead of the tap, so a model calibrated in a quiet room would collapse to one constant prediction the moment the room got noisier. Clamping the range below each window's own peak keeps the features focused on the tap itself, accuracy holds up to roughly sixteen times the calibration noise floor.

The audio callback does almost nothing. MFCC extraction and prediction take tens of milliseconds, far too long to run inside a PortAudio callback without causing dropouts. So the callback just copies blocks into a ring buffer and queues an event, and a worker thread does the actual work.

Warm up happens before the stream is trusted. librosa's numba kernels JIT compile on first use. Until that finishes, the engine meters audio but refuses to trigger, because a tap detected mid compile would stall the worker for tens of seconds and the app would look like it had frozen.

The app

Five screens, all driven by the same live audio engine.

• Tap Zones. Bind a key to each side, watch the live confidence, fire a zone by hand.
• Macro Bindings. Build multi step pipelines (keystroke, text, delay, shell command, app launch, sound) triggered by single, double or triple taps.
• Sensor Calibration. Live oscilloscope, surface presets, detection gates, tap event log.
• Calibrate & Train. Record samples, fit the model, read the cross validated accuracy.
• Preferences. Startup behaviour, dependency checks, profile import and export, reset.

Under the hood, Electron spawns one long lived python -m src.daemon process and talks to it over stdin and stdout in JSON lines. Requests carry an id and get a reply back, while meter levels, detected taps and training output just arrive as events whenever they happen. The renderer never touches Node directly, everything goes through a typed bridge exposed in the preload script. Since anything printed to stdout would corrupt that protocol stream, the daemon claims real stdout for protocol frames and points Python's own stdout at stderr instead.

Stack

Python underneath (librosa for the MFCCs, scikit-learn for the classifier, sounddevice and PortAudio for capture), wrapped in an Electron and vanilla JS desktop shell. I tested it with a synthetic tap generator, basically a damped sum of resonances, low and slow for the left side, higher and faster for the right, so I could exercise the detection pipeline, the JSON bridge and the UI itself without needing a microphone or another person around to tap on command.