Search Results for "keyboard"

Open Interface is a cross-platform application that allows users to control their computers using large language models (LLMs). By sending user requests to an LLM backend, it determines the necessary steps and executes them by simulating keyboard and mouse inputs. The system can adjust its actions based on real-time feedback, providing a self-driving computer experience.

The Self-Operating Computer Framework is an innovative system that enables multimodal models to autonomously operate a computer by interpreting the screen and executing mouse and keyboard actions to achieve specified objectives. This framework is compatible with various multimodal models and currently integrates with GPT-4o, o1, Gemini Pro Vision, Claude 3, and LLaVa. Notably, it was the first known project to implement a multimodal model capable of viewing and controlling a computer screen. The framework supports features like Optical Character Recognition (OCR) and Set-of-Mark (SoM) prompting to enhance visual grounding capabilities. ...

...Screenshots of the mirrored window provide visual input, while Apple's Vision framework extracts visible text and tap-ready coordinates. macOS CGEvents supply taps, long presses, drags, scrolling, keyboard input, and iPhone Mirroring shortcuts. After performing an action, the agent captures the screen again to verify the actual resulting state. The harness can be installed as an agent skill for systems such as Claude Code or Codex. Its editable helper workspace also lets an agent create missing automation functions during execution.

GELab-Zero is an open-source “GUI Agent” framework aiming to automate interactions with graphical user interfaces (GUIs), combining both the agent model and all supporting infrastructure — including inference, input orchestration, and GUI automation logic — in a plug-and-play package that runs locally, without cloud dependencies. The idea is to let developers or users harness an AI agent that can simulate clicking, typing, reading UI elements, and interacting with apps in a human-like way...

...Its purpose is to serve as a baseline or reference implementation so researchers or developers can see how to build agents that operate in real-time, visual environments (e.g., games, browser apps) via pixel observations and keyboard/mouse actions. Under the hood, this starter agent implements a version of the A3C (Asynchronous Advantage Actor-Critic) algorithm, adapted for the specific challenges of Universe environments (e.g., network latency, VNC streaming, asynchronous observations). The repo includes modules like train.py, worker.py, model.py, a3c.py, and envs.py to support training, parallel worker management, policy/critics, and environment wrappers.