Machine Learning

GestureKey

PythonOpenCVMediaPipeScikit-learnPyQt6

Real-time hand gesture detection system that translates webcam input into OS-level keyboard shortcuts. Clean layered architecture with an integrated ML pipeline — from raw camera frames to system actions, running as a background tray app.

The Problem

I wanted to control my computer during presentations and recordings without touching the keyboard — a gesture-based shortcut layer running in the background. The core challenge was making detection fast and stable enough to be usable in real conditions: variable lighting, slight camera movement, and the natural imprecision of human hand positions.

Key Decisions

MediaPipe for landmark detection instead of a custom CV pipeline

MediaPipe returns 21 normalized 3D landmarks per hand in under 10ms on CPU. Building an equivalent from scratch with OpenCV would have taken months and produced an inferior result. The right engineering decision here was to treat hand detection as a solved problem and focus the original work on the classifier and the gesture engine on top.

Random Forest over a neural network for gesture classification

A CNN needs thousands of images and a GPU to train in a reasonable time. But MediaPipe already extracts semantic landmarks — the relative positions of 21 joints — so the raw pixel data is irrelevant. On that structured input, a Random Forest trained on 200 samples per gesture reaches 98%+ accuracy and runs inference in under 2ms. A neural network would have added complexity, training time, and deployment weight with no accuracy gain.

Temporal stabilizer using a majority vote window

Frame-by-frame predictions are noisy: if one finger moves a pixel, the classifier might briefly output the wrong class. Firing an OS shortcut on every raw frame prediction would make the app unusable. The stabilizer keeps a rolling buffer of the last N predictions and only emits an action when the same gesture wins the majority vote for the full window. This eliminates flicker without adding perceptible latency.

PyQt6 for the tray app instead of Tkinter or Electron

The app needs to run silently in the background, intercept global keyboard shortcuts, and inject OS-level key events — all of which require native system access. PyQt6 integrates cleanly with the OS tray APIs and provides proper global hotkey hooks. Tkinter has no tray support. Electron would have been absurd for a background utility: a 150MB Chromium instance to replace a 4MB native binary.

Outcome

GestureKey runs stably as a background process without impacting CPU or memory noticeably. The model generalizes well across different lighting conditions and users who weren't in the training set. The project sharpened my understanding of where ML adds value — not in replacing classical CV, but in the classification layer where structured features already exist.

Highlights