Intuitive
Pinch to pick a window up. Make a fist to shut it. Turn a knob in the air for the volume. Gestures your hands already know.
Hand-gesture window control for Linux
Reach out and grab it
HoloTouch lets you control the windows on your desktop with your hands in the air. A webcam watches them, and windows move, resize, close, scroll and switch.
Move a window
Thumb + index pinch
Pinch over a window and carry it.
A hand is not a mouse
Most attempts at spatial control map your hand to the pointer and stop there: a few awkward gestures, laid on top of an interface built for a keyboard and mouse. HoloTouch starts from what hands already do.
Space first
It uses the room around you. Tip two fingers at the screen to scroll, or touch a fist to your chin and every window is laid out as a card.
Extensible
Replace what the pie menu holds, add macros you fingerspell, and record your own hands to teach it your alphabet.
Quick and consistent
Thirty frames a second on a laptop CPU, with an ordinary webcam. A slow hand does not jitter and a fast one does not lag.
The closest you will get to Tony Stark tossing windows around.
Fifteen gestures
Each is shown the way the control panel's guide shows it: a hand, and what it acts on.
Move a window
Thumb + index
Pinch over a window and carry it. A quick pinch brings it to the front.
Resize
Second hand pinches too
While one hand holds a window, pinch with the other and pull them apart.
Maximize or minimize
Flick a held window
Flick it up and let go to maximize it, down to minimize it.
Send to a workspace
Hold at a screen edge
Carry a window to the left or right edge of the screen and wait there.
Click
Point, thumb down
Point up, thumb out, then tap the thumb down onto your middle finger. Twice is a double click; keep it down to drag.
Pie menu
Thumb + pinky
Move toward an item and let go to pick it: apps, windows, music.
Close a window
Fist, held still
Hold a fist over the window until the ring completes.
Scroll
Two fingers up
Tip the two fingers at the screen to scroll down, straighten them up to scroll up: the further, the faster.
Volume and brightness
Claw, turned
Grip a knob in the air and turn it. Right hand: volume. Left: brightness.
Enter
Open palm, swept right
Sweep an open hand quickly to the right to press the Enter key.
Window switcher
Fist to chin
Every window becomes a card. Point at one and pinch to go to it.
Camera
Peace sign, both hands
Hold both up until the ring completes to open the camera and take a photo.
Dictate
Thumb and pinky out
Hold up the letter Y and speak. Let go, and what you said is typed where the keyboard is.
App launcher
Thumb, index and pinky out
Hold up I love you, then fingerspell an app's name until it is the only one left, or a macro's letters.
New tab
Clap twice
With a web browser in front, clap your hands twice to open a new tab in it.
Eleven hand shapes, read by rule
From 21 landmarks a hand, HoloTouch measures how far the thumb is from each fingertip and how curled each finger is, in palm lengths. Hand-written rules turn that into a pose.
An open palm. Swept fast to the right, it presses Enter.
The eleventh is a hand at rest, which does nothing. Each pose has separate thresholds for entering and leaving, and must hold for a moment before it counts, so a pose does not flicker.
An overlay that reads on anything
A see-through, click-through layer shows where your hands are and what each gesture is about to do. Every shape pairs cream with ink, so it reads over light and dark windows alike.
Idle
Small and pale once a hand has made no gesture for a while.
Ready
An open ring. The desktop shows through it.
Pinching
The ring tightens and a dot grows as the fingers close.
Pinched
The dot is full: the hand has hold of something.
- Sky is your left hand.
- Coral is your right hand.
- Butter is progress: a hold filling up.
- Mint is done, and running.
- Lilac opens further.
How it works
Three kinds of process: a tracker that finds hands, a core that turns them into gestures and acts on windows, and an optional control panel that starts, stops and watches the core.
-
1
Tracker
Camera to landmarksThe webcam is read at 1280×720, 30 frames a second. MediaPipe finds up to two hands, 21 landmarks each, and a face mesh for the chin. It runs in a process of its own, so inference never stalls the overlay.
-
2
Core
Landmarks to gesturesEach hand keeps its identity when hands cross. A 1€ filter smooths a slow hand hard and a fast hand lightly. The engine starts the matching interaction, and one runs at a time.
-
3
Desktop
Gestures to windowsxfwm4's windows are changed with standard EWMH requests. Clicks and keys go through XTEST, and scrolling through a virtual mouse for smooth wheel events.
-
4
Overlay
What you seeA fullscreen, transparent Qt Quick window sits above everything and draws the cursors, frames, menus and rings. Nothing samples or blurs the desktop.
Misread hands do no damage
- A hand must be in view and relaxed for a moment before its gestures count.
- A pinch that begins while the hand is moving fast is ignored.
- A grab shorter than 300 ms is undone.
- Only application windows can be changed, never the desktop or a panel.
- 15
- gestures
- 21
- landmarks a hand
- 30
- frames a second
- 352
- tests
Get started
HoloTouch runs on a laptop CPU with an ordinary webcam.
What it needs
- Linux with an X11 session, xfwm4 and a running compositor
- A webcam
- Python 3.12
A few tools are optional, each for one feature: smooth scrolling, volume, dictation and the rest. holotouch doctor reports which are present.
$ git clone https://github.com/justanotherinternetguy/HoloTouch $ cd HoloTouch $ uv sync $ uv run holotouch doctor $ uv run holotouch panel
holotouch panel- Open the control panel: start and stop, the gesture guide, and practice.
holotouch run- Start tracking and the overlay.
holotouch run --dry-run- Practice on two stand-in windows. Your real ones are left alone.
holotouch doctor- Check the machine, and measure the tracking rate.