A browser kiosk that reads a gesture and a word, then brews a real espresso

Context

The client was launching a new espresso machine and wanted a stand experience that put the product in the visitor's own hands: walk up, make a gesture, say a word, and the new machine brews you a real cup — a hands-on proof anyone could do in seconds on the show floor. Two constraints shaped everything. It had to run on an ordinary tablet at the stand, with nothing to install. And it had to drive the actual machine in front of the visitor — pour a real espresso, not play a video of one.

Problem

The headline is a real-time perception problem wired to a physical outcome.

• Read a person, a gesture, and a word at once — live. Notice when someone steps into position, recognize the Italian "perfetto" hand gesture, and hear them say "perfetto" — all from a single tablet camera and microphone, in real time, in a loud hall.
• Coach, don't just judge. A visitor who almost has it should be nudged along, not failed. The recognition has to feed a forgiving, staged loop — "keep working," "almost there" — rather than a binary pass/fail.
• Reach a real machine from a web page. The payoff is physical: the instant of success has to cross over to the espresso machine an arm's length away and start the pour — from the browser, reliably.
• Run unattended, all day. There's no operator on a stand. It has to attract, detect, run, celebrate, dispense, and reset itself for the next person — on whatever tablet is on the table.

Approach

The whole thing is one browser page, installed to the tablet's home screen as a PWA — no native app, no store. Two MediaPipe Tasks Vision models run on the live camera feed: a pose landmarker that notices when a visitor is present and centered, and a gesture recognizer that tracks the hand to catch the "perfetto" pinch and its motion. In parallel, speech recognition listens for the spoken "perfetto"; for noisy rooms where the browser's recognizer struggles, we carried an on-device Whisper path compiled to WebAssembly as a fallback.

A small state machine orchestrates the ritual end to end — attract loop → visitor detected → intro → the gesture-and-voice moment → success → dispense → a "perfetto" photo the visitor can keep — and tears itself back down when they walk away or time out, so it runs all day with no one minding it. The coaching is staged across attempts: it escalates from say perfetto to keep working to almost there, each with its own audio, so a near-miss is encouraged instead of rejected.

The dispense is a direct Bluetooth write. On success the page connects to the machine over Web Bluetooth and writes the maker's own brew command to its BLE characteristic — the integration we were engaged to build against their protocol — so the new machine pours, in front of the visitor, started by their gesture.

Outcome

A visitor walks up, makes the "perfetto" gesture, says the word, and the new machine brews them a real espresso — start to finish in seconds, on an ordinary tablet, with no app and no attendant. A product launch becomes something you do with your hands instead of something you watch: a single browser page that sees, listens, coaches, and reaches through Bluetooth to make a physical machine pour — the kind of software-meets-hardware piece that doesn't fit one box.