Search Results for "speak"

JoyAI-VL-Interaction is an open real-time video-language interaction system built around an 8B-scale vision-first model. It is designed to watch a webcam or livestream continuously and decide whether to speak, stay silent, or delegate a harder task. Unlike turn-based assistants, it focuses on event-driven interaction where timing matters as much as answer quality. The repository releases the model, training recipe, time-aligned interaction data, and deployable system together. Its system includes inference, WebUI, ASR, TTS, and background-agent services running on vLLM-based infrastructure. ...

...The motivation is to move beyond “text in, text out” and enable direct, live, multi-turn exchange involving language, gesture, gaze, vision, and modality switching without user friction. The system architecture includes a real-time multimodal signal pipeline for audio, video, and sensor data, a dialog manager that can decide when to act (speak, gesture, point) or query, and a cross-modal reasoning layer that fuses perception with semantic context. The research prototype includes components for visual grounding (understanding when a user references something in view), gesture recognition and synthesis, and turn-taking mechanisms that mirror human conversational timing. ...

...Because the model is trained with a “large-margin learning” objective over many synthesized and natural speech samples, it gains robust control over expressive attributes, and can perform iterative editing: e.g. you could record a line, then ask the model to “make it sadder,” “speak slower,” or “change accent to X.”