Search Results for "computer use"

The Self-Operating Computer Framework is an innovative system that enables multimodal models to autonomously operate a computer by interpreting the screen and executing mouse and keyboard actions to achieve specified objectives. This framework is compatible with various multimodal models and currently integrates with GPT-4o, o1, Gemini Pro Vision, Claude 3, and LLaVa. Notably, it was the first known project to implement a multimodal model capable of viewing and controlling a computer screen....

SWE-agent turns LMs (e.g. GPT-4) into software engineering agents that can resolve issues in real GitHub repositories. On the SWE-bench, the SWE-agent resolves 12.47% of issues, achieving state-of-the-art performance on the full test set. We accomplish our results by designing simple LM-centric commands and feedback formats to make it easier for the LM to browse the repository, and view, edit, and execute code files. We call this an Agent-Computer Interface (ACI).

ACT-CV provides an easy to use interface between visual objects on the computer screen (or another video source) and a cognitive modeling tool.

The April project defines a set of C++ objects (World, Actor, Sensor, Reflex, Brain, Actuator, Event) that help simulate various environments. The world has an amount of energy that the user may use to create Actors and components while Actors may also create other Actors from their own energy. The code is structured in several layers: - april-core simply defines the library and the objects - april-gui builds on the concepts in april-core, giving a visual interpretation to the...

The Guiey project is an attempt to develop a system for graphical user interface scripting and testing in Java. Guiey will use a computer by seeing, typing, and clicking - just like a human user.