Search Results for "keyboard"

Open Interface is a cross-platform application that allows users to control their computers using large language models (LLMs). By sending user requests to an LLM backend, it determines the necessary steps and executes them by simulating keyboard and mouse inputs. The system can adjust its actions based on real-time feedback, providing a self-driving computer experience.

Maestro is a cross-platform desktop application designed for power users to orchestrate and manage fleets of AI agents and project workflows from a keyboard-centric interface. It provides a high-performance experience for running multiple agent sessions in parallel, integrating with tools such as Claude Code, OpenAI Codex, and other agent tooling to automate tasks, perform unattended execution, and organize long-running work flows. Users can collaborate with AI to draft specifications, break tasks into playbooks, and run them sequentially or concurrently in isolated contexts, each with clean session history. ...

The Self-Operating Computer Framework is an innovative system that enables multimodal models to autonomously operate a computer by interpreting the screen and executing mouse and keyboard actions to achieve specified objectives. This framework is compatible with various multimodal models and currently integrates with GPT-4o, o1, Gemini Pro Vision, Claude 3, and LLaVa. Notably, it was the first known project to implement a multimodal model capable of viewing and controlling a computer screen. The framework supports features like Optical Character Recognition (OCR) and Set-of-Mark (SoM) prompting to enhance visual grounding capabilities. ...

...Users can resume, fork, compact, search, rewind, and switch models while preserving session history through DSH services. File references, image attachments, slash commands, questionnaires, and keyboard-driven input are built into the interface. Long conversations use event-driven rendering, virtualization, and bounded caches to control terminal and memory costs. The plugin also supports themes, MCP, skills, subagents, agent presets, and VS Code terminal workflows.

GELab-Zero is an open-source “GUI Agent” framework aiming to automate interactions with graphical user interfaces (GUIs), combining both the agent model and all supporting infrastructure — including inference, input orchestration, and GUI automation logic — in a plug-and-play package that runs locally, without cloud dependencies. The idea is to let developers or users harness an AI agent that can simulate clicking, typing, reading UI elements, and interacting with apps in a human-like way...

...Its purpose is to serve as a baseline or reference implementation so researchers or developers can see how to build agents that operate in real-time, visual environments (e.g., games, browser apps) via pixel observations and keyboard/mouse actions. Under the hood, this starter agent implements a version of the A3C (Asynchronous Advantage Actor-Critic) algorithm, adapted for the specific challenges of Universe environments (e.g., network latency, VNC streaming, asynchronous observations). The repo includes modules like train.py, worker.py, model.py, a3c.py, and envs.py to support training, parallel worker management, policy/critics, and environment wrappers.