Audio Mirror is a browser-based interactive music project: movement in front of a camera becomes sound.

Try the MVP at franky.metalbox-store.org. It runs on both computers and phones — open it in a browser and allow camera access.

This is an old build, kept here as a memory. It now serves as my weather station.

How it works now

Pose recognition uses MediaPipe Pose Landmarker for Web, a browser and JavaScript task from Google’s MediaPipe Tasks Vision package.

Processing happens entirely in the client-side browser. The engine combines Rust compiled to WebAssembly with TypeScript, so pose data is processed where the camera is running rather than being sent away for server-side inference. This keeps the experience usable on a PC or a phone and allows the system to scale without turning camera processing into server load.

The current application is an MVP. It plays audio files from self-hosted S3-compatible storage, while the rest of the project is self-hosted as well.

From movement to MIDI

Pose data is fed side by side into a spiking neural network (SNN) and a liquid state machine (LSM). Their output weights are mapped to MIDI signals. Rather than relying on a traditional BPM counter, the system decides the frequency of musical events itself.

Audio Mirror can be connected to Ableton Live or another DAW as a MIDI device, so its movement-driven output can control external instruments and production workflows.

From hardware installation to browser

Before the browser version, Audio Mirror ran as a hardware setup built around a Raspberry Pi, camera, screen, Coral Edge TPU accelerator, and a model quantized to run on the TPU. The model ran inside Python code, with an MQTT event bus and a Kotlin Multiplatform processing engine.

Looking for collaborators at the 2026 ADE Hackathon

Audio Mirror is looking for two people to help turn the MVP into a stronger experience:

• Audio designer — someone with musical taste who can shape the sound, musical direction, and how Audio Mirror responds to movement.
• UI designer — someone with visual taste who can make the experience intuitive and remove technical information that does not help a final user.

Any development help, feedback, or advice would be highly appreciated — even a short conversation can help move the project forward.

View the project on Bitbucket. The project is publicly available only for the duration of the hackathon.

Once finished, the project will be deployed to its mirror/staging URL: franky-staging.metalbox-store.org.

Hackathon challenges

Audio Mirror satisfies the Stability AI challenge requirements. The plan is to integrate Stability AI in place of the current sample playback.

It also satisfies Colin Benders’ quest: there is no BPM setting, because the system determines the frequency of musical events from movement itself.

The project is still comparing the LSM and SNN approaches, so this is a good moment for collaborators to influence both the experience and its direction.

Timeline

April 8–20, 2026

• Added the Rust liquid state machine workspace, now compiled to WebAssembly for browser use, along with its build workflow and loader.
• Began running the LSM alongside the SNN.
• Developed pose features, temporal embeddings, reservoir processing, readout logic, diagnostics, and benchmarks.
• Added LSM prediction and activity visualisations, then fixed the WebAssembly import.

December 20–26, 2025

• Fixed the MediaPipe integration and added Docker, CI/CD, and error logging.
• Improved the camera and activation-panel layout, including responsive mobile behaviour.
• Added audio sample playback driven by SNN neurons and audio-file support.

December 19, 2025

• Started the browser-based prototype with a camera and pose pipeline.
• Added pose derivatives and spike encoding for the spiking neural network.
• Implemented a winner-take-all SNN and its activation visualisation.

August 29, 2025

• Cleaned up tracking and added tests.
• Added an aggregate task named sliding.
• Added sliding parameters — not finished.
• Identified logging sliding data in the database and MQTT as future work.

August 26–28, 2025

• Defined the derivatives and removed the ones that should not remain.
• Added a runtime to take pose data, calculate, store, filter, and smooth it.
• Covered the runtime with tests.
• Extracted tracking hyperparameters into config.yaml — in progress.

August 25, 2025

• Defined Hungarian pose tracking.
• Moved derivative metrics from the Kotlin Multiplatform repository.
• Defined per-frame derivatives; not finished yet.

August 24, 2025

• Moved pose metrics from an abandoned Kotlin Multiplatform repository to Rust — partially.
• Decided to include illumination tracking.
• Defined a roadmap.

August 23, 2025

• Set up pose tracking on a Raspberry Pi from the Mirror repository.
• Experimented with FMOD.