"Watch What I Do" Chapter
17
Figure 1.
Triggers in programming mode with the desktop blanket, which appears as small
dots, covering the Macintosh windows. The user is making a selection from the
Triggers Menu.
Chapter
17
Triggers:
Guiding Automation with Pixels to Achieve Data Access
Richard Potter
Triggers is a programming system that shows how simple pattern matching applied
to the pixels on a computer screen can effectively access data that is
otherwise hidden inside an application program and unavailable to other
programming by demonstration systems. Triggers invokes operators in
applications by simulating keyboard and mouse actions, and accesses data
through the pixel representations on the computer screen. Triggers extends the
record/playback style popularized by keyboard macros. Triggers shows that
pixel-based device-level algorithms exist, are understandable, can be easily
implemented, and can allow a programming system to process data in situations
where it would otherwise be impossible.
Triggers
Uses and Users
Application domain: Any Macintosh application
Intended users: Researchers
User Interaction
How does the user create, execute and modify programs?
The user demonstrates condition-action rules. The user records the conditions by outlining pixel patterns (bitmaps) on the computer screen that Triggers is to locate. Normal macro recording is used to demonstrate the actions, except that mouse actions can be specified in relation to pixel pattern locations. Rules can be single-stepped or allowed to fire continuously. Triggers' rules are displayed as icons. Individual rules can be opened to display each of the condition-action steps as icons. Rules and steps may be reordered by dragging icons. Additional steps can be inserted into existing rules. Limited modifications of existing steps are possible.
Feedback about capabilities and inferences:
When running, Triggers optionally highlights the areas of the screen being searched for pixel patterns. Execution can be paused during any pixel pattern test for inspection. Highlighting icons gives a visual trace of program execution.
Inference
Inferencing: No inferencing.
Program constructs: The condition-action rules are implicit
conditionals. Continuous trying of the rules is an implicit while loop. Boolean
flags within rules allow for more complex program constructs.
Implementation
Machine, language, size, date:
Macintosh. Implemented in 12,000 source lines of C on top of the Think C
class library, 1991.
Notable Features
By transparently accessing data in otherwise closed applications, Triggers proves that pixel-level data access is a viable partial solution to the data access obstacle. While macros have long proved the value of simulating the keyboard and mouse, which the user touches, Triggers proves the value of analyzing the pixel information on the computer screen, which the user sees. The surprising number of tasks that can be readily automated by Triggers encourages exploration of new user interface widgets and techniques that will enable pixel-based programs to be specified by novice users.
back to ... 
Table of Contents Watch What I Do