Table of Contents Watch What I Do
Many applications, such as GNU Emacs [Stallman 87], have a macro by example facility, but lack a visual representation for the macros. Without a visual representation, it is impossible to review the operations that compose the macro. When there is an error in such a macro, the macro must be demonstrated once again from scratch. If an error occurs in a macro without a visual representation, the system cannot provide a comprehensible error message explaining which step generated the error.
Though visual representations are clearly important for a macro by example facility, many systems omit this component since it is problematic how to statically display commands executed through an application's graphical user interface. We have developed a technique for visually representing such commands. Previously we used a representation, called editable graphical histories, to provide a visual record of commands executed in a session with a graphical editor (Chapter 19). We have extended this technique to represent macros by example, and support the definition and editing of these macros. Here we introduce a macro by example facility that uses editable graphical histories as its visual representation, and discuss the many ways that the macro facility takes advantage of these histories.
The macro by example system described in this paper is implemented as part of Chimera, an editor system with modes for editing 2D illustrations, user interfaces, and text (Chapter 12). Macros can currently be defined in both the illustration and user interface editing modes. All of the examples in this paper are generated from the PostScript output of Chimera and its macro by example facility.
In the next section we discuss how other example-based systems have dealt with the issue of representation. Then we briefly describe editable graphical histories, and in the rest of the paper focus on how they support a macro by example facility.
Representing commands in text-based systems tends to be easier, since the textual commands themselves form a convenient representation. Tinker, a text-based programming by demonstration facility, has a textual audit-trail of steps used in constructing procedures (Chapter 2). To edit the demonstrated procedure, the user can either textually edit these steps or the resulting LISP procedure. Tweedle, a graphical editor with both a WYSIWYG view and a textual code view, allows procedures to be generated in both views [Asente 87]. However, to edit a procedure, the user must be able to understand the code view. In the MIKE UIMS, graphical macros can also be defined by demonstration [Olsen 88]. In this system macros can be defined and edited largely in demonstration mode, but the visual representation of graphical commands is textual.
A programming by demonstration component of SmallStar, a miniature version of the Star user interface, adopts a mixed text and iconic representation for programs (Chapter 5). The system uses a predefined set of icons or pictographs to represent entities on the desktop. The domains for which our system is targeted are more graphical in nature, so prefabricated icons will not suffice. As will be discussed in more depth in the next section, our approach is to generate graphics automatically to represent the operations in the macro. Mondrian, a system developed concurrently with Chimera, has a related visual representation, but differs by using uneditable screen miniatures (Chapter 16).
All the programming by demonstration systems discussed thus far have special operations to start and stop recording events. In our system, operations are always being recorded by an undo/redo mechanism. When users realize that a set of operations that they had performed are generally useful, they can always open up a history window and encapsulate the interesting operations into a macro. A programming by demonstration system named Eager also generates macros from a history (Chapter 9). It constantly monitors the command stream for repeated operation sequences. When a repetitive task is detected, the system presents feedback that indicates the tasks it anticipates, and when users are confident in Eager's predictions, they can have it automatically generate a generalized procedure. However, this procedure has no graphical representation.
Editable graphical histories use several techniques to make a sequence of commands more comprehensible. The panels are graphical, and use the same visual language as the interface itself. Since the user of the system is already familiar with the interface's conventions, it is easy for them to interpret the histories.
Related commands are coalesced into single panels by pattern matching rules. This makes the histories more compact, but also makes them easier to interpret, since instead of showing physical commands the system shows logical commands. For example, the first panel of the history in Figure 1 represents commands that have added a text label to the oval. The two commands move the caret (the software cursor) to the desired position for the text, and then insert the characters. The second panel changes the font of this text and includes both an object selection and Set-Font command. An interactive elaboration facility can be used to expand higher-level panels into their lower-level components.
If we were to shrink down the entire screen to fit in each panel, then it would be difficult to see the changes that the panels represent. Instead the system shows only those objects that participate in the operation, as well as a little scene context to indicate where on the screen the operation was performed. Each panel of the history in Figure 1 contains only a subset of the editor scene or of Chimera's control panel. For example the second panel includes the Text Input widget, in which the name of the new font was typed, as well as the selected text. The history mechanism adds other objects to the panels only to provide context. Objects in the panels are rendered according to their roles in the explanation. Currently the rendering pass subdues contextual objects by lightening their colors. This usually makes it easy to distinguish these objects from those that participate in the operation.
In the second pass, the demonstrated sequence of commands is supplemented with additional information to convert this sequence into a macro. The commands executed in this pass are different than those forming the ordinary application dialogue. This pass includes selecting a set of previously executed commands to encapsulate into the macro, selecting arguments for the macro, generalizing the commands to work in other contexts, and debugging and saving the macro. Splitting macro definition into a demonstrational step and a generalization step was first done by Halbert in SmallStar (Chapter 5). It has the advantage that the demonstrational pass of the macro is purely demonstrational, and certain constructs, such as conditionals and loops, which are difficult to add by demonstration, can be introduced in a separate non-demonstrational pass.
However, unlike SmallStar which had special commands to start and stop recording a macro, commands in Chimera are always being recorded by an undo/redo facility. At any time, users can open up the history window, review the commands executed in a session, or undo and redo some of these commands. They can also select a set of commands to be incorporated into a macro. The history of Figure 1 shows a set of commands that add text to an oval and construct a drop shadow for the oval. Recognizing that drop shadows are necessary for other objects in the scene, we can now extract the relevant panels from the history and turn them into a macro.
First we select these panels using the mouse, and as feedback, the selected panel labels appear in reverse video, as shown in Figure 1. The macroize operation, which is executed next, takes a panel selection, and opens up a new Macro Builder window on these panels. This window initially contains only those panels that were selected in the graphical history.
A panel is added to the beginning of the history, depicting the argument selection. Argument declarations are placed at the beginning of our macros, just as they appear at the beginning of traditional procedures. The resulting panel is the first of the sequence of panels depicted in the Macro Builder of Figure 2. The argument declaration panels show the arguments as they appear before the operations in the macro were invoked, plus additional scene context. They are not just copies of the panels that were used for selecting the arguments. Scene objects that do not exist at the beginning of the macro, such as the oval produced by the copy operation in the third panel of Figure 2, are not plausible arguments, and Chimera will not allow them to be used for this purpose.
In addition to choosing graphical objects to be arguments to a macro, we can also choose graphical properties such as color or line style. To select a graphical property, we can select from the history a widget in which this property is displayed. Widgets can be selected just like any other graphical object. For this macro, we would like the color of the drop shadow to be a second argument. First we locate the panel in which we specified the color. This is the Text Input field of the fourth panel of Figure 2. We select this widget, and give the second argument the name "color". A new argument declaration panel is created, which is the second panel of Figure 2.
Recall that our system chooses rendering styles for panel objects according to the objects' roles in the explanation. However in Figure 2, all objects are rendered in their usual fashion. We automatically revert to this standard rendering when panels are made editable, since these panels become fully editable scenes, and the user may want to query or manipulate the colors and other graphical properties of the objects in these panels. When the panels are restored to their original uneditable state, the original rendering is also restored.
For
example, after selecting the last panel of Figure 2, we execute the
Generalize-Panel command. The window shown in Figure 3 appears, containing the
various generalizations that the system considered plausible in the given
context, with the most likely interpretation selected. The generalizations of
all the operations contained in a panel can be viewed and modified at once.
This panel contains the selection of the drop shadow, and the subsequent
translation of the shadow to lie at the appropriate offset under the original
object. Only one of the built-in interpretations for the selection is valid in
the context of the last panel: that the object selected at this step is the
object created in the third panel.
The Generalize-Panel command need not be executed explicitly for every panel in the macro. Another command can be used to set or reset all panels to their default generalization. When the macro is executed, all panels that have never been generalized are automatically given a default generalization.
To invoke a macro, the user executes a menu command and a macro invocation window pops up on the screen. For the drop shadow macro that we have just defined, this window is shown in Figure 4. The window contains an entry for each of the arguments declared previously. The first argument, "object", is assigned two buttons: one to set the argument and the other to show it. The second argument, "color", is a property argument, and Chimera uses a different technique to set and show property arguments. For each property argument, a copy of the widget used to specify the argument during the original macro demonstration is included in the invocation window. Since the Text Input field of the control panel was originally used to specify the color of the drop shadow, this widget is copied and added to the invocation window. As a default, the widget contains the value specified for this parameter during macro demonstration time. Effectively this parameter is treated as an optional parameter with a default. If the user does not change its value in the macro invocation window, it will default to the value that was used during demonstration.
Figure 4. Window for setting arguments and invoking the drop shadow macro.
At this point we notice a bug in the macro. Though we expect the macro to add a light grey drop shadow to the Bezier, the drop shadow is black. To debug the macro, we return to the original Macro Builder window in Figure 2 and examine the commands that it contains. The bug quickly becomes apparent. Though we changed the fill color of the drop shadow, we never changed its line color. On inspecting the results of this initial test again, it is clear that the circle's drop shadow is incorrect as well since it too has a black line, yet we did not notice a problem at first because the shadow is dark.
Figure 5. Testing the macro. The first row contains a set of test objects. The next row contains the results of invoking the original macro on these objects, using the colors named at the bottoms of the columns. The final row shows the results of invoking the debugged version of the macro.
The history representation provides a means of selecting operations. This is useful for two different steps of macro creation. At any time, the user can scroll through the history, and select out useful commands for a new macro. Accordingly, Chimera needs no additional commands to start and stop macro recording. Later, a user may want to view or change the generalizations associated with a set of panels. Again, the graphical representation can be used to select the appropriate panels.
The macro representation makes it very easy to select arguments. After selecting the checkbox that makes the panels of a macro editable, we can select objects directly in the panels and turn them into arguments. Graphical properties can also be turned into arguments by selecting the widgets that set these properties from the macro panels.
Macros are not always defined correctly the first time, and the histories present an interface for editing commands. New commands can be inserted by invoking additional commands in the editable panels, and executing a command that propagates the changes. Unwanted commands can be removed by selecting panels and deleting them.
The macro system itself often refers to the representation when communicating important information to the user. During the generalization process, the interpretation of a command may refer back to steps made at an earlier point in time. For example, the system often needs to refer to an object that was created in a particular panel, or a measurement that was made at a certain step. Macros can generate run-time errors if they are invoked on objects of the incorrect type. Chimera also uses the macro representation to indicate which panel of the macro generated an error.
In summary, these histories support macro definition in five different capacities: 1. Reviewing macro contents
2. Selecting operations
* to encapsulate in a macro
* to set and view generalizations
3. Selecting arguments
4. Editing macro contents
* inserting operations
* deleting operations
5. Referencing operations
* during generalization
* in error messages
We could also use the panels to specify additional command generalizations when the existing ones fail. For example, one step in a macro might involve reducing a box's width by half. If the system is incapable of inferring the desired intent of this operation, the user might be able to add annotations to the panels that make the intent explicit. Since the panels are editable, the user might be able to use direct manipulation techniques to define a temporal constraint between the width of the box at two different points in time (or panels). The interface for doing this could be similar to Chimera's interface for defining constraints between separate objects.
There are a number of basic ways in which our macro by example facility can be enhanced. Currently when we save macros, we save all of the scene state, which allows us to restore the original graphical representation of macros for subsequent editing. This increases the storage requirements of the macro. We could also provide an option that automatically strips superfluous scene objects from the macro. Not only would this reduce storage, but it might also make the graphical macro representation clearer since it would contain no extraneous objects.
We would also like to expand the generalizations that Chimera is capable of making, so that a greater number of useful macros by example can be defined. Loops and conditionals would also increase the power of our macro facility. We have experimented with using graphical search and constraint-based search (Chapter 12) as iteration mechanisms for graphical macros, but others would be helpful as well.
Finally, it is important to provide a means of representing changes to the macros within the graphical representation, so that changes can be undone. Currently, Chimera can generate histories for macro panels that have been edited, but once the panel changes have been propagated into the macro, or a panel has been deleted, the information is lost.
Intended users: End users
Table of Contents Watch What I Do
