"Watch What I Do" Chapter
A Teachable Graphical Editor
Mondrian is an object-oriented graphical editor that can learn new graphical
procedures through programming by demonstration. A user can demonstrate a
sequence of graphical editing commands on a concrete example to illustrate how
the new procedure should work. An interface agent records the steps of the
procedure in a symbolic form. Machine learning techniques track relationships
between graphical objects and dependencies between the interface operations.
The agent generalizes a program that can then be used on "analogous" examples.
The generalization heuristics set it apart from conventional "macros" that can
only repeat an exact sequence of steps. The system represents all operations
using "storyboards" which depict examples. By bringing the power of procedural
programming to easy-to-use graphical interfaces, we hope to break down the
"Berlin Wall" that currently exists between computer users and computer
Uses and Users
Application domain: Graphical editor
Intended users: Visual thinkers
How does the user create, execute and modify programs?
After selecting the New Command icon, the user demonstrates the procedure on an example. This resulting procedure is displayed as a new icon in the application's palette.
Feedback about capabilities and inferences:
As the user performs the example, Mondrian uses speech to describe its inferences.
The procedure can be viewed as a storyboard.
Program constructs: Procedure calls
Types and sources of information:
Information about properties of graphical objects.
Heuristics for ranking the significance of relationships between objects.
Machine, language, size, date: Macintosh, Common Lisp, Approximately
600KB source, 6 MB image. 1991.
Mondrian's "dominoes" and "storyboards" form a consistent visual language for representing both built-in operations and the results of user demonstrations. It is the only PBD system to use natural language feedback.
back to ... Table of Contents Watch What I Do