Midscene
Vision-based AI framework for cross-platform UI automation tasks
...Instead of relying on traditional selectors, DOM structures, or accessibility attributes, it uses a vision-first approach where screenshots are analyzed by visual-language models to identify interface elements and perform actions. It allows developers to automate interactions on web applications, desktop software, and mobile devices without needing platform-specific automation logic. Developers can describe tasks such as clicking buttons, filling forms, or extracting information, and the system interprets these commands to interact with the interface accordingly. Midscene.js includes SDKs, scripting options, and integration capabilities that allow automation workflows to be written in JavaScript, TypeScript, or YAML-based scripts. ...