"Watch What I Do" Chapter 17

Figure 1. Triggers in programming mode with the desktop blanket, which appears as small dots, covering the Macintosh windows. The user is making a selection from the Triggers Menu.


Guiding Automation with Pixels to Achieve Data Access

Richard Potter

Triggers is a programming system that shows how simple pattern matching applied to the pixels on a computer screen can effectively access data that is otherwise hidden inside an application program and unavailable to other programming by demonstration systems. Triggers invokes operators in applications by simulating keyboard and mouse actions, and accesses data through the pixel representations on the computer screen. Triggers extends the record/playback style popularized by keyboard macros. Triggers shows that pixel-based device-level algorithms exist, are understandable, can be easily implemented, and can allow a programming system to process data in situations where it would otherwise be impossible.


Uses and Users

Application domain: Any Macintosh application

Intended users: Researchers

User Interaction

How does the user create, execute and modify programs?
The user demonstrates condition-action rules. The user records the conditions by outlining pixel patterns (bitmaps) on the computer screen that Triggers is to locate. Normal macro recording is used to demonstrate the actions, except that mouse actions can be specified in relation to pixel pattern locations. Rules can be single-stepped or allowed to fire continuously. Triggers' rules are displayed as icons. Individual rules can be opened to display each of the condition-action steps as icons. Rules and steps may be reordered by dragging icons. Additional steps can be inserted into existing rules. Limited modifications of existing steps are possible.

Feedback about capabilities and inferences:
When running, Triggers optionally highlights the areas of the screen being searched for pixel patterns. Execution can be paused during any pixel pattern test for inspection. Highlighting icons gives a visual trace of program execution.


Inferencing: No inferencing.

Program constructs: The condition-action rules are implicit conditionals. Continuous trying of the rules is an implicit while loop. Boolean flags within rules allow for more complex program constructs.


Machine, language, size, date:

Macintosh. Implemented in 12,000 source lines of C on top of the Think C class library, 1991.

Notable Features

By transparently accessing data in otherwise closed applications, Triggers proves that pixel-level data access is a viable partial solution to the data access obstacle. While macros have long proved the value of simulating the keyboard and mouse, which the user touches, Triggers proves the value of analyzing the pixel information on the computer screen, which the user sees. The surprising number of tasks that can be readily automated by Triggers encourages exploration of new user interface widgets and techniques that will enable pixel-based programs to be specified by novice users.

back to ... Table of Contents Watch What I Do