A Java-based visual compiler for GUI recognition and automation. The screens are described in an XML file which contains the definitions of lexemes and the tokens that comprise them. Upon a successful match of a screen, user-defined code is executed. Within the scope of this code, the user is capable of extracting data from the screen, interpreting it, and driving the GUI accordingly. The demonstration example reads the value of a calculator, displays it for the user, and enables him to enter a number which is then inputted into the calculator by pressing the appropriate keys.