Search Results for "android input device"

...The project works by using Android’s accessibility API to extract structured UI state (as XML) from the device, which is then fed to a large language model (LLM) like OpenAI’s models for decision-making, and actions are executed via the Android Debug Bridge (ADB). This approach bypasses expensive vision-based models and provides faster, cheaper automation with fine-grained interaction capabilities (for example, tapping buttons, typing text, navigating screens).

ADK (Android Device Key) Python is a reference implementation by Google for working with Android attestation keys in Python. It facilitates the integration of Android attestation features into backends or systems that require verification of device identity and integrity. This is especially important in high-security applications where verifying that a device is genuine and uncompromised is critical.

Open-AutoGLM is an open-source framework and model designed to empower autonomous mobile intelligent assistants by enabling AI agents to understand and interact with phone screens in a multimodal manner, blending vision and language capability to control real devices. It aims to create an “AI phone agent” that can perceive on-screen content, reason about user goals, and execute sequences of taps, swipes, and text input via automated device control interfaces like ADB, enabling hands-off completion of multi-step tasks such as navigating apps, filling forms, and more. Unlike traditional automation scripts that depend on brittle heuristics, Open-AutoGLM uses pretrained large language and vision-language models to interpret visual context and natural language instructions, giving the agent robust adaptability across apps and interfaces.

Chatbox is a cross-platform desktop AI client designed to give you a fast, polished, and private way to work with modern language models. It runs locally on Windows, macOS, and Linux, keeping your conversations and data stored on your own device. Chatbox acts as a unified interface for popular LLMs like ChatGPT, Claude, Gemini, and local models via Ollama, making it easy to switch providers without changing tools. Built with an ergonomic UI, it’s optimized for long sessions, prompt...

CogAgent is a 9B-parameter bilingual vision-language GUI agent model based on GLM-4V-9B, trained with staged data curation, optimization, and strategy upgrades to improve perception, action prediction, and generalization across tasks. It focuses on operating real user interfaces from screenshots plus text, and follows a strict input–output format that returns structured actions, grounded operations, and optional sensitivity annotations. The model is designed for agent-style execution rather...

...Developed by Tongyi-MAI (Alibaba’s research initiative), the MAI-UI models are multimodal agents trained to understand user instructions and corresponding screenshots, grounding those instructions to on-screen elements and generating sequences of GUI actions such as taps, swipes, text input, and system commands. Unlike traditional UI frameworks, MAI-UI emphasizes realistic deployment by supporting agent–user interaction (clarifying ambiguous instructions), integration with external tool APIs using MCP calls, and a device–cloud collaboration mechanism that dynamically routes computation to on-device or cloud models based on task state and privacy constraints.

...It can provide controlled access to Calendar, Contacts, Location, Maps, Messages, Reminders, and Weather. The menu bar interface lets users enable individual services and approve the required macOS permissions. A bundled command-line server communicates over standard input and output with clients such as Claude Desktop, Claude Code, Cursor, and Amp. Calendar and reminder tools can create items, while other tools support searches, directions, travel estimates, message history, and current conditions. The app itself does not collect or store user data, although connected AI clients may send tool results off device. ...

49Agents is an open-source “agentic IDE” that reimagines how developers interact with multiple AI agents, terminals, and development environments by placing everything onto a single infinite, zoomable canvas. Instead of relying on traditional tab-based workflows, it provides a spatial interface where terminals, editors, Git views, and monitoring tools coexist as movable panes, enabling users to manage complex multi-agent systems visually and intuitively. The platform is designed to work...