"Watch What I Do" Chapter 16


A Teachable Graphical Editor

Henry Lieberman


Mondrian is an object-oriented graphical editor that can learn new graphical procedures through programming by demonstration. A user can demonstrate a sequence of graphical editing commands on a concrete example to illustrate how the new procedure should work. An interface agent records the steps of the procedure in a symbolic form. Machine learning techniques track relationships between graphical objects and dependencies between the interface operations. The agent generalizes a program that can then be used on "analogous" examples. The generalization heuristics set it apart from conventional "macros" that can only repeat an exact sequence of steps. The system represents all operations using "storyboards" which depict examples. By bringing the power of procedural programming to easy-to-use graphical interfaces, we hope to break down the "Berlin Wall" that currently exists between computer users and computer programmers.


Uses and Users

Application domain: Graphical editor

Intended users: Visual thinkers

User Interaction

How does the user create, execute and modify programs?
After selecting the New Command icon, the user demonstrates the procedure on an example. This resulting procedure is displayed as a new icon in the application's palette.

Feedback about capabilities and inferences:
As the user performs the example, Mondrian uses speech to describe its inferences.
The procedure can be viewed as a storyboard.


Program constructs: Procedure calls


Types and sources of information:
Information about properties of graphical objects.
Heuristics for ranking the significance of relationships between objects.


Machine, language, size, date: Macintosh, Common Lisp, Approximately 600KB source, 6 MB image. 1991.

Notable Features

Mondrian's "dominoes" and "storyboards" form a consistent visual language for representing both built-in operations and the results of user demonstrations. It is the only PBD system to use natural language feedback.

back to ... Table of Contents Watch What I Do