Search Results for "gesture"

Seamless Communication is a research project focused on building more integrated, low-latency multimodal communication between humans and AI agents. The motivation is to move beyond “text in, text out” and enable direct, live, multi-turn exchange involving language, gesture, gaze, vision, and modality switching without user friction. The system architecture includes a real-time multimodal signal pipeline for audio, video, and sensor data, a dialog manager that can decide when to act (speak, gesture, point) or query, and a cross-modal reasoning layer that fuses perception with semantic context. The research prototype includes components for visual grounding (understanding when a user references something in view), gesture recognition and synthesis, and turn-taking mechanisms that mirror human conversational timing. ...

...The project combines Jupyter or Colab notebooks with browser-based visual demos that allow users to see trained models operating in real time. Many experiments involve tasks such as image classification, object detection, gesture recognition, and simple generative models. The models are typically trained in Python using TensorFlow and then exported for interactive demonstrations in a web environment using JavaScript and TensorFlow.js. Because the project focuses on experimentation rather than production systems, it acts as a sandbox where developers can explore machine learning concepts and observe model behavior. ...

...These pipelines can run on a wide variety of platforms including mobile devices, desktop systems, web browsers, and embedded edge devices. MediaPipe is widely used in computer vision and multimedia applications such as hand tracking, face detection, pose estimation, object recognition, and gesture analysis. The framework includes prebuilt solutions that developers can quickly integrate into applications as well as lower-level APIs that allow custom pipeline construction.

Yoha is a browser-based hand tracking engine designed to enable real-time gesture recognition and interaction using standard webcams, making it accessible for web applications without specialized hardware. Built using JavaScript and TensorFlow.js, it runs directly in the browser and performs inference on-device, eliminating the need for server-side processing. The engine is capable of detecting 21 two-dimensional hand landmarks, allowing developers to build applications that respond to gestures such as pinching or forming a fist. ...

Douyin-Bot is a Python automation project for interacting with the Douyin mobile app through Android device tooling. It was built as an experimental bot that captures phone screenshots, analyzes visible content, and performs automated app actions. The project uses Python and ADB to connect desktop-side logic with a mobile device. Its original goal was playful and experimental, focused on browsing and identifying content based on computer vision and face analysis. It is best understood as a...

NEWS: PolyRec is now on GitHub. Please visit https://github.com/cluelab/polyrec PolyRec is a unistroke gesture recognizer suitable for fast prototyping of gesture-based applications. The recognizer uses a nearest neighbor approach, and requires a small number of samples for each class. The similarity between two gestures is calculated through a three steps procedure: -firstly, each gesture is approximated to a polyline, in order to extract its main movements; - then, the two polylines are aligned to obtain an equal number of segments from both of them; - lastly, the distance is found by summing the contribution of each pair of segments. ...

Inspired by interfaces from sci-fi movies like Iron Man, Gesture Analytical Sonic Interface (GASI) is a proof of concept of a Webcam gesture (Kinect like) and Voice recognition based computer interface, constraining itself to only components included in average laptops (A simple webcam and microphone, no Kinect)