Search Results for "visual"

...Notably, it was the first known project to implement a multimodal model capable of viewing and controlling a computer screen. The framework supports features like Optical Character Recognition (OCR) and Set-of-Mark (SoM) prompting to enhance visual grounding capabilities. It is designed to be compatible with macOS, Windows, and Linux (with X server installed), and is released under the MIT license.

...It aims to create an “AI phone agent” that can perceive on-screen content, reason about user goals, and execute sequences of taps, swipes, and text input via automated device control interfaces like ADB, enabling hands-off completion of multi-step tasks such as navigating apps, filling forms, and more. Unlike traditional automation scripts that depend on brittle heuristics, Open-AutoGLM uses pretrained large language and vision-language models to interpret visual context and natural language instructions, giving the agent robust adaptability across apps and interfaces.

android-action-kernel is an open source Python library designed to let AI agents control and automate native Android applications running on real devices or emulators. It fills a gap in automation tooling by focusing on mobile-first workflows where traditional browser or desktop-based automation doesn’t work; such as logistics, gig work, field operations, and other industries reliant on phones or tablets. The project works by using Android’s accessibility API to extract structured UI state...

airda(Air Data Agent) is a multi-smart body for data analysis, capable of understanding data development and data analysis needs, understanding data, generating data-oriented queries, data visualization, machine learning and other tasks of SQL and Python codes.

The universe-starter-agent repository is an archived OpenAI codebase designed as a starter reinforcement-learning agent that can interact with and solve tasks in OpenAI’s Universe environment platform. Its purpose is to serve as a baseline or reference implementation so researchers or developers can see how to build agents that operate in real-time, visual environments (e.g., games, browser apps) via pixel observations and keyboard/mouse actions. Under the hood, this starter agent implements a version of the A3C (Asynchronous Advantage Actor-Critic) algorithm, adapted for the specific challenges of Universe environments (e.g., network latency, VNC streaming, asynchronous observations). ...

This is a software agent based computing program that will enable researchers and other users to tap in computing power of machine available by sharing work load on the fly with zero configuration on network & resources A self organizing agent program that will understand network and its resource. where as the only job left to researcher is to split up jobs in several chunks of programs either parallel or sequential jobs and go issue the job (A visual Modeler or Scripting support need to be yet designed) Software agents will automatically manage the rest or resource management, sharing , cloning of tasks etc. new resources can be added and removed from the system on fly; in layman terms the project will create an agent program that enable sharing & execution of program among all the available resources whether it be desktop, laptop, pda . thereby one can accelerate research to the very extent of resource availability with out bothering about anything... ...

Pythia is a natural language question answering system, which uses Speech Recognition and Text To Speech technologies to communicate with the user.