Download
MAI-UI is a cutting-edge open-source project that implements a family of foundation GUI (Graphical User Interface) agent models capable of interpreting natural language and performing real-world GUI navigation and control tasks across mobile and desktop environments. Developed by Tongyi-MAI (Alibaba’s research initiative), the MAI-UI models are multimodal agents trained to understand user instructions and corresponding screenshots, grounding those instructions to on-screen elements and generating sequences of GUI actions such as taps, swipes, text input, and system commands. Unlike traditional UI frameworks, MAI-UI emphasizes realistic deployment by supporting agent–user interaction (clarifying ambiguous instructions), integration with external tool APIs using MCP calls, and a device–cloud collaboration mechanism that dynamically routes computation to on-device or cloud models based on task state and privacy constraints.
Features
- Natural language to GUI action generation for mobile/desktop interfaces
- Multimodal grounding of text and screenshots for UI understanding
- Support for direct user interaction and clarification workflows
- MCP tool integration for extended API-level operations
- Device–cloud hybrid execution to balance privacy and performance
- Models at multiple scales (from lightweight to large-capacity variants)
Project Samples
