MotionKey

MotionKey is an Android app and training pipeline for personalized hand-gesture shortcuts. A user records a few examples in front of the phone, a Python service trains and quantizes a small classifier, and the resulting LiteRT model runs entirely on the device.

The current actions are taking a photo, starting or stopping video recording, and advancing a teleprompter. Gesture names and motions are user-defined; the system does not ship with a fixed gesture vocabulary.

Key capabilities

• Personalized gestures: up to three user-defined gestures, each mapped to a local action.
• Offline recognition: after training, camera processing, classification, and action dispatch do not require a network connection.
• Landmark-only training data: MediaPipe converts camera frames to hand landmarks on the phone; training uploads CSV landmark sequences rather than images or video.
• False-trigger protection: training requires an explicit background class, and inference uses confidence, debounce, latch, cooldown, and neutral-arming gates.
• Reproducible optimization: the repository includes scripts and committed results for fp32-to-int8 size, accuracy, and latency comparisons.

Project status and supported scope

MotionKey is an experimental application intended for development, evaluation, and extension rather than production deployment.

The current implementation supports one visible hand, one person, a front camera, and up to three custom gestures plus a system-managed background class. It targets Android 12/API 31 or newer and is tested primarily on a Pixel 4. Clear indoor lighting and a stationary, portrait-oriented phone produce the best results.

The training server is synchronous, single-user, and unauthenticated. It binds to loopback by default and must not be exposed directly to the public internet.

Technology stack

Component	Technologies
Android app	Kotlin, Jetpack Compose, CameraX
Hand tracking	MediaPipe Hand Landmarker
On-device inference	LiteRT, XNNPACK
Trainer	Python 3.11, NumPy, TensorFlow
Training API	FastAPI, Uvicorn
Tests	pytest, JUnit, Android instrumentation tests

Architecture

flowchart LR
    subgraph device["Android device · capture and offline inference"]
        camera[Front camera] --> landmarks[MediaPipe<br/>hand landmarks]
        landmarks --> recorder[Recording library<br/>CSV sequences]
        landmarks --> pipeline[Gesture pipeline<br/>LiteRT + trigger gate]
        model_store[(Installed int8 model<br/>and class metadata)] --> pipeline
        pipeline --> actions[Device actions<br/>photo · video · teleprompter]
    end

    subgraph host["Training host · Python 3.11"]
        api[FastAPI session API] --> dataset[(Per-build<br/>gesture dataset)]
        dataset --> trainer[TensorFlow<br/>1D-CNN trainer]
        trainer --> quantizer[int8 quantization]
    end

    recorder -->|upload landmarks| api
    quantizer -->|download model + labels| model_store

Loading

The training host is needed only when a model is built or rebuilt. Once the app installs model_int8.tflite and labels.json, recognition and actions stay on the Android device. See the architecture notes for the full data path and design rationale.

Requirements

Area	Requirement
Trainer and server	Python 3.11, pinned in .python-version
Android build	JDK 17+, Android SDK 35, about 2 GB free disk space
Device	Android 12/API 31+ with a front camera; arm64 recommended
Device connection	adb and USB debugging, or a trusted local network
First build	Network access for Python, Gradle, Android, and MediaPipe dependencies

Run ./scripts/check_env.sh to inspect the local toolchain and get targeted installation hints.

Quick start

Run commands from the repository root unless a step says otherwise.

1. Set up the trainer and server

./scripts/setup.sh
./scripts/check_env.sh

setup.sh creates trainer/.venv, installs the trainer and server dependencies, and runs the Python test suite.

To exercise the full ML pipeline without a phone:

./benchmarks/run_host_pipeline.sh

This synthesizes a dataset, trains a model, quantizes it, and writes results to benchmarks/results/host-<arch>/.

2. Start the training server

./scripts/serve.sh

The server listens on 127.0.0.1:8000 by default. Leave it running while the phone uploads recordings and downloads the trained model.

3. Build and install the Android app

In a second terminal:

cd android
./gradlew test
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
cd ..

The first Gradle build downloads the MediaPipe Hand Landmarker bundle into the app's assets directory. The downloaded file is excluded from version control.

4. Connect the phone to the server

The recommended development connection is an ADB reverse tunnel over USB:

./scripts/connect_pixel.sh
adb reverse --list

The second command should include tcp:8000 tcp:8000. Recreate the tunnel after unplugging the device, rebooting it, or restarting ADB.

For a trusted LAN connection, explicitly expose the server and configure the app with the host machine's LAN address:

BIND_HOST=0.0.0.0 ./scripts/serve.sh

Then set the app's server URL to http://<host-ip>:8000. The service has no authentication, so do not use this mode on an untrusted network.

5. Train and use gestures on the device

1. Open Teach, select Teach a new gesture, enter a name, and choose an action.
2. Record at least four examples of each custom gesture. Eight are recommended.
3. Record at least three Prevent accidental triggers clips. Include ordinary movement and frames with no hand; six clips are recommended.
4. Select Build model. The app creates a new server session, uploads the landmark recordings, trains the model, and installs the model and metadata.
5. Open Control and select Start gesture control.
6. Keep your hand lowered or out of frame until the status changes to Ready, then perform a trained gesture.

The model continues to work if the server is stopped or the phone is placed in airplane mode. Renaming a gesture or changing its action updates local metadata and does not require retraining. Adding or deleting gestures, or changing their recordings, requires a rebuild.

Training data and model pipeline

Each frame contains 64 values: a hand-presence flag followed by the x, y, and z coordinates of 21 MediaPipe landmarks. Frames without a detected hand are represented explicitly rather than dropped.

The trainer performs the following stages:

1. Resample recordings to 15 frames per second.
2. Extract 24-frame windows with a five-frame stride.
3. Normalize each sequence while preserving motion across frames.
4. Apply augmentation and feature standardization.
5. Train a small 1D convolutional classifier.
6. Export fp32 and post-training int8 LiteRT models.
7. Store preprocessing parameters and the class/action table in labels.json.

Dataset readiness is enforced independently by the app, API, and trainer:

Class	Minimum	Recommended
Each custom gesture	4 recordings	8
Background / Prevent accidental triggers	3 recordings	6

The background class is always model output index 0 and has no action. This lets empty frames and ordinary movement resolve to a defined no-op instead of a custom gesture.

Recognition behavior

Recognition uses overlapping 1.6-second windows. A gesture fires when the same class wins twice consecutively with at least 0.85 confidence. The class then latches until the hand returns to a different or background state, and a 1.5-second cooldown limits the trigger rate.

Every recognition session starts disarmed. Two consecutive neutral windows are required before the trigger gate receives predictions. This prevents opening or resuming the recognition screen from immediately firing a gesture that is already being held.

The pipeline reports explicit states such as collecting, low confidence, confirming, cooldown, held gesture, and missing action mapping. The production decision logic has no Android dependency, so recorded landmark sequences can be replayed through it in JVM tests.

Training API

FastAPI exposes interactive API documentation at http://127.0.0.1:8000/docs while the server is running.

Method and path	Purpose
POST /sessions	Create an isolated training session
POST /sessions/{id}/gestures/{name}	Register gesture display and action metadata
POST /sessions/{id}/demos/{name}	Upload landmark CSV recordings
GET /sessions/{id}/readiness	Return per-class dataset counts and requirements
POST /sessions/{id}/train	Train and quantize a session dataset
GET /sessions/{id}/model	Download the session's int8 model
GET /sessions/{id}/labels	Download preprocessing and class metadata
DELETE /sessions/{id}	Remove a session and its stored data
GET /health	Return service and training-requirement information

Server state is stored under server/storage/, which is excluded from version control. There is no automatic retention or multi-user isolation beyond session directories.

Repository layout

Path	Contents
android/	Android app, JVM tests, instrumentation tests, and bundled synthetic model
trainer/	Dataset processing, model training, quantization, evaluation, and pytest suite
server/	FastAPI session and training service
benchmarks/	Host/device benchmark scripts and committed result files
scripts/	Environment setup, server startup, and device connection helpers
docs/	Detailed architecture and supporting documentation
.github/	Continuous integration and dependency-update configuration

Development and validation

Run the relevant checks before opening a pull request:

trainer/.venv/bin/ruff check .
trainer/.venv/bin/ruff format --check .
(cd trainer && .venv/bin/python -m pytest -q)
(cd android && ./gradlew test)

Additional device and pipeline checks:

Check	Command
Android instrumentation tests	cd android && ./gradlew connectedDebugAndroidTest
Synthetic host pipeline	./benchmarks/run_host_pipeline.sh [label]
On-device classifier latency	./benchmarks/run_device_latency.sh
End-to-end camera pipeline latency	./benchmarks/run_device_pipeline.sh
Held-out device recording evaluation	./benchmarks/run_device_eval.sh

The current automated suites contain 106 passing Python tests with two expected failures, plus 465 Android JVM tests per build variant. CI runs Python linting, formatting, tests, a server import check, shell syntax checks, and Android JVM tests.

Benchmarks

The bundled model and committed host results use synthetic landmark recordings. They validate the pipeline and provide reproducible optimization measurements; they are not evidence of real-world recognition accuracy.

Metric	fp32	int8	Change
Classifier size	116.9 KiB	39.7 KiB	2.9× smaller
Synthetic holdout accuracy	1.0000	1.0000	No measured loss
Host classifier latency p50	0.005 ms	0.002 ms	2.5× faster

On the reference Pixel 4, the int8 classifier with XNNPACK measured 0.141 ms p50 and 0.159 ms p95. The live camera-to-classifier pipeline measured 82.5 ms p50 and 144.5 ms p95 on an empty-scene path. See benchmarks/results/ for environment details and raw reports. Latency depends on the device, instruction set, and whether MediaPipe is detecting or tracking a hand, so rerun the scripts on the intended target.

Troubleshooting

Problem	Resolution
Python or TensorFlow installation fails	Use Python 3.11 and run ./scripts/setup.sh; do not reuse a virtualenv created with another Python version.
Gradle cannot find the Android SDK	Set ANDROID_HOME or create android/local.properties with sdk.dir=/absolute/path/to/Android/sdk.
MediaPipe model download fails	Check network/proxy access and rerun Gradle; the build error includes a manual curl command.
adb reports no device	Enable USB debugging, use a data-capable cable, authorize the computer, and rerun adb devices.
The app cannot reach 127.0.0.1:8000	Start the server, then recreate the tunnel with ./scripts/connect_pixel.sh and verify adb reverse --list.
Android blocks cleartext communication	Use the debug build for local HTTP training; release builds retain Android's default cleartext restriction.
Build model is disabled	Record the missing examples listed by the readiness panel, including background clips.
Recognition does not fire	Wait for Ready, keep the full hand in frame, return to neutral between gestures, and review the on-screen recognition state.
Recognition fires on ordinary movement	Add more varied Prevent accidental triggers recordings and rebuild the model.

Known limitations

• Recognition runs only while the app is in the foreground.
• Only one hand is detected and classified at a time.
• The available action catalog is currently limited to photo capture, video recording, and teleprompter advancement.
• Photos are captured from the analysis stream and are 480×640 on the Pixel 4; video records at 720p on the reference device.
• Captures are stored in app-private storage and contain no audio. They can be opened, shared, or deleted from MotionKey, but there is no editing workflow.
• The bundled fallback model is trained on synthetic motion and is intended for smoke tests and benchmarks. Useful recognition requires personal recordings.
• Confidence and gating defaults have not been tuned through a broad field study.
• The training API has no authentication, resource quotas, concurrent job management, or automatic session cleanup.

Contributing and security

See CONTRIBUTING.md for setup and pull-request guidance. Read SECURITY.md before changing the server's network exposure or working with real landmark recordings.

License

MIT