Audience

Researchers in need of a tool to enhance AI agents' interaction with graphical user interfaces through advanced screen parsing techniques

About OmniParser

OmniParser is a comprehensive method for parsing user interface screenshots into structured elements, significantly enhancing the ability of multimodal models like GPT-4 to generate actions accurately grounded in corresponding regions of the interface. It reliably identifies interactable icons within user interfaces and understands the semantics of various elements in a screenshot, associating intended actions with the correct screen regions. To achieve this, OmniParser curates an interactable icon detection dataset containing 67,000 unique screenshot images labeled with bounding boxes of interactable icons derived from DOM trees. Additionally, a collection of 7,000 icon-description pairs is used to fine-tune a caption model that extracts the functional semantics of detected elements. Evaluations on benchmarks such as SeeClick, Mind2Web, and AITW demonstrate that OmniParser outperforms GPT-4V baselines, even when using only screenshot inputs without additional information.

Integrations

Ratings/Reviews

Overall 0.0 / 5
ease 0.0 / 5
features 0.0 / 5
design 0.0 / 5
support 0.0 / 5

This software hasn't been reviewed yet. Be the first to provide a review:

Review this Software

Company Information

Microsoft
Founded: 1975
United States
microsoft.github.io/OmniParser/

Videos and Screen Captures

OmniParser Screenshot 1
Other Useful Business Software
MongoDB Atlas runs apps anywhere Icon
MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
Start Free

Product Details

Platforms Supported
Cloud
Training
Documentation
Videos
Support
Online

OmniParser Frequently Asked Questions

Q: What kinds of users and organization types does OmniParser work with?
Q: What languages does OmniParser support in their product?
Q: What kind of support options does OmniParser offer?
Q: What type of training does OmniParser provide?

OmniParser Product Features

OmniParser Additional Categories