Spell it with Sign Language: An Asl Typing Game with MediaPipe

Abstract

This talk introduces a Python-based typing game that helps users practice American Sign Language (ASL) fingerspelling using real-time hand tracking and gesture recognition. Players "type" letters by signing them into a webcam, with MediaPipe used for hand landmark detection and scikit-learn for static sign classification. For dynamic letters like "J" and "Z," we use a second model built with PyTorch to capture motion patterns from sequential data. We'll introduce ASL fingerspelling for context, then walk through our development pipeline—from data collection (both self-recorded and public datasets) to training, evaluation, and game design. A demo will showcase the game, which has been tested by students in educational settings. We’ll share performance metrics (~88% static accuracy at 15 FPS), deployment lessons, and a public GitHub repo to support reproducibility. If you're into computer vision, accessibility, or building fun Python projects, this talk is for you.

Description