back to ... Table of Contents Watch What I Do

NOTE: Some of the images in this chapter could not be converted properly for use on the web


Uses of Demonstrational Techniques

Brad A. Myers


Garnet is a comprehensive user interface development environment in Lisp for X/11 (Display Postscript and Macintosh versions are in progress). It helps create graphical, highly-interactive, direct manipulation user interfaces. Garnet contains many high-level tools, including the Gilt interface builder [Myers 91d], the Lapidary interactive design tool [Myers 89b], the C32 spreadsheet system [Myers 91a], the Jade dialog box system [Vander Zanden 90], and more to come. Garnet also contains a complete toolkit, which uses constraints [Vander Zanden 91a], a prototype-instance object model, and a new model for handling input [Myers 90c]. The toolkit also contains two complete widget sets, one with the Motif look and feel.

Typical applications created with Garnet include: drawing programs similar to Macintosh MacDraw, user interfaces for expert systems and other AI applications, box and arrow diagram editors, graphical programming languages, game user interfaces, simulation and process monitoring programs, user interface construction tools, CAD/CAM programs, etc. Garnet is in the public domain and is freely available. As of fall, 1992, over 30 projects around the world are using the system regularly. You can get Garnet by anonymous FTP from Change to the directory /usr/garnet/garnet/ and retrieve the README file for instructions. Or you can send electronic mail to Garnet stands for Generating an Amalgam of Realtime, Novel Editors and Toolkits.

One of the important goals of the Garnet project is to allow all aspects of the user interface to be created without conventional programming. In particular, we want to allow the user to draw example pictures to show what the user interface will look like, and then demonstrate how the user interface will respond to inputs from the end user. As a result, demonstrational techniques are widely used in Garnet, mainly in the various higher-level tools. This chapter discusses some of these. Other papers about Garnet discuss the overall design [Myers 90d], the components, the programming style [Myers 92a] [Myers 92e], and there is a complete reference manual [Myers 92b].


The Lapidary user interface tool allows the pictorial aspects of programs to be specified graphically [Myers 89b] [Vander Zanden 91b]. A "Lapidary" is a workman who cuts, polishes and engraves precious stones, and here is a Lisp-Based Assistant for Prototyping Interface Designs Allowing Remarkable Yield. In addition, the behavior of these objects at run-time can be specified using dialogue boxes and by demonstration. In particular, Lapidary allows the designer to draw pictures of application-specific graphical objects which will be created and maintained at run-time by the application. This includes the graphical entities that the end user will manipulate (such as the components of the picture), the feedback that shows which objects are selected (such as small boxes around an object), and the dynamic feedback objects (such as hair-line boxes to show where an object is being dragged). Lapidary is a direct descendent of Peridot (Chapter 6) and extends a number of Peridot's ideas.
(a)                                                                                    (d)  

Figure 1. The workspace window of Lapidary (b), where a node of a graph editor is being created, along with the standard commands (a), object menus (c), and a dialog box for setting constraints on rectangles (d).

In addition, like Peridot, Lapidary supports the construction and use of "widgets" (sometimes called interaction techniques or gadgets) such as menus, scroll bars, buttons and icons. Lapidary therefore supports using a pre-defined library of widgets, and defining a new library with a unique "look and feel." The run-time behavior of all these objects can be specified in a straightforward way using constraints and abstract descriptions of the interactive response to the input devices. Lapidary generalizes from the specific example pictures to allow the graphics and behaviors to be specified by demonstration.

Graphical objects can be created in a number of different ways using Lapidary. As shown in Figure 1, the standard menus provide the usual range of graphical primitives, so objects can be created from scratch.


A central feature of Lapidary that makes it appropriate for creating run-time application graphics is the use of constraints. Constraints allow the designer to specify a relation between a graphic object and other objects in the scene, and have that relation maintained at run-time by the system. If a constraint is one of a standard set, then it can be specified easily using the Lapidary menus (see Figure 1-d). These menus support having objects be connected on their edges or in the middle, with optional offsets. The sizes of objects can also be related. There are different windows showing the constraints for lines and a few other objects. Experience with Peridot demonstrates that these simple types of constraints make up the vast majority of those needed in typical user interfaces.

Sometimes, designers want to use relationships that cannot be created out of these simple choices. In that case, the Custom option is selected, and the designer is allowed to type in an arbitrary Common Lisp expression specifying the constraint using the C32 system (discussed below).

Unlike Peridot, Lapidary currently does not try to infer the graphical constraints. Instead, they must all be specified explicitly using the dialog boxes. With Lapidary, we wanted to concentrate on creating a practical tool that extends the range of interfaces that can be produced, and the constraint inferencing in Peridot was felt to be too risky for the first version. Future Garnet tools will revisit this issue.

In order for the graphical objects to be useful at run-time, the specific constraints must be generalized to work on run-time objects, rather than on the specific example objects used in the editor. For example, in Figure 1, the label on the nodes should change, but still stay centered, as the node is replicated. Thus, the constraints need to be generalized to reference objects indirectly through variables, rather than by using specific object names. To do this, the reference to the object is replaced with an expression that calculates the desired object, and stores it in a special slot. The constraint system then automatically ensures that the constraints change whenever the slot is set with a different object.

It is important to emphasize that Lapidary makes these transformations automatically. The user interface designer never sees any of the code. Even if the designer created custom constraints by typing Lisp code, the references in the expression can be to example objects (selected by pointing at them with the mouse), and the system will convert these references to be general variables where appropriate.

Another way that Lapidary generalizes from the examples is to automatically make copies of objects at run-time. For example, to show the selection in a drawing editor, the designer might draw a single set of selection handles around an example object. However, at run time, multiple objects might be selectable, so Lapidary arranges for the selection handles to be duplicated at run-time if necessary.

Interactive behavior

Although it is useful to prototype the graphic appearance of user interfaces, it is much more useful if the interactive behavior can also be specified easily. Lapidary therefore provides this capability. In order to edit the behavior of objects, or to add behavior to new objects, we have encapsulated a number of kinds of interactive behaviors into "interactor" objects [Myers 90c], each of which has its own dialogue box for specifying properties. For example, to change which mouse button operates a menu, it is only necessary to change the button indicated in the dialogue box.

Often, there will be a specific object that serves as the feedback for an operation. For example, a reverse-video rectangle might move over the items in the menu to show which is the current selection. In other cases, the objects themselves should change to be the feedback. For example, the currently selected item in a menu might be shown in italics. Another use is to have buttons move to cover their shadows (and therefore look more "3-D"), as in the Motif and Garnet look and feels. In this case, the desired changes can be shown by demonstration. To specify the changes by demonstration, first the designer selects the objects that will change, and then hits a button in the dialogue box. The full current state of the selected objects is remembered. Next, the designer edits the objects in whatever way desired, for example to make the string be italic. Then, another button is hit, and Lapidary creates a constraint that will choose between the two values based on whether the object is selected or not. Changes can be made to as many properties as desired, and correct constraints will be created for all of them.


Through the use of demonstrational techniques, Lapidary is able to allow the designer to interactively create far more of the user interface than any other tool. In particular, new widgets and application-specific objects can be created. Demonstrational techniques are crucial since these objects all are parameterized and will change dynamically at run time, so it is only possible to draw examples, not the actual objects to be used. These examples are generalized into named prototypes which the applications can then make instances of at run time.


When the iconic menus in Lapidary are not sufficient for specifying the desired constraints, Garnet provides the C32 spreadsheet program to help enter more complex constraints [Myers 91a]. C32 can also be used stand-alone. It displays and allows the user to edit any kind of object and constraint, no matter how they were created: by hand-coding, by using Lapidary, or by using C32. C32 stands for CMU's Clever and Compelling Contribution to Computer Science in Common Lisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of Creative Constructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, and are Communicated using Columns of Cells that are Constantly Calculated so they Change Continuously and Cancel Confusion.

Figure 2 shows a typical instance of C32. Each column contains a separate object. Rows are labeled with the names of the slots, such as :left, :top, :width, :height, :visible, etc. Since different objects can have different slots, the slot names are repeated in each column. For example, lines have slots for the endpoints (:x1, :y1, :x2, :y2) but rectangles do not. Also, each object's display can be scrolled separately, so each has its own scroll bar. This makes the spreadsheet look somewhat like a multi-pane browser as in Smalltalk.


Figure 2. (a) C32 viewing three objects (b). The scroll bars can be used to see more slots or columns. Changing the window's size will change the number of slots and objects displayed (the number of rows and columns). Field values are clipped if they are too long, but can be scrolled using editing commands. The "F" icon means that the slot value is computed with a formula. All inherited slots are shown in italics and marked with the "I" icon. When a formula is inherited the value is shown in a regular font since it is usually different from the prototype's. The inherited icon is also shown next to the formula icon rather than next to the value.

The spreadsheet cells show the current values of the slots. If a value changes, then the display will be immediately updated. If the user edits the value in the spreadsheet cell, the object's slot will be updated. The "F" icon by some slots in Figure 2 means that the slot value is computed from a formula. Pressing the mouse on the icon causes the constraint expression to appear in a different window. The expression itself can be edited by typing or other techniques.

Use of inferencing

It is sometimes not convenient to read an object into a spreadsheet column just to generate a reference to it. Therefore, a command will place into the current formula a reference to any object in a Garnet window. However, selecting a graphical object does not specify which slot of the object should be referenced. In one mode, the user must type this directly or select a slot from a menu. However, the other mode uses heuristics to guess the slot from the example by looking at the slot being filled and where the mouse is pressed in the selected object. For example, if the slot is :left, and the mouse is pressed at the right of an object, then the reference will be to the right of the object. For the :width slot, however, the same press would generate a reference to the width of the object. Unlike Peridot, C32 does not try to confirm any of the inferences, but rather simply inserts the text into the formula. If the guess is incorrect, it is easy for the user to delete the text and type the correction.

Once a complex formula is created, it will often be needed in a slightly different form for a different slot or a different object. As an example, suppose the user has constructed a constraint that centers an object horizontally with respect to two other objects. Now, suppose the programmer wants to center the object vertically also. The formula could be copied to the :top slot, but all the slot references need to be changed (:left to :top and :width to :height). Therefore, when a formula is copied, C32 tries to guess whether some slot names should be changed. This uses a few straightforward rules based on the slot names of the source and destination slots. Currently, these rules are hardwired into the code. If it appears that slot names should be changed, the user is queried with a dialog box, and if the answer is OK, then the formula is modified automatically. Since this is a more radical change than the inferred slots discussed in the previous section, it seems prudent to require confirmation.

Automatic generalization

Another possibility is that the references in the formula should be generalized into variables. C32 therefore provides a command that will change the entire formula into a function that takes the objects and/or slots as parameters. The user can choose the names for the function and for the variables. C32 will generalize objects, slots, or both.

The intelligent copying and generalizing in C32 helps the user generate correct constraints by example. Without these aids, it is quite common to forget to change one or more of the references when formulas are copied. Generalizing also helps the programmer decrease the size of the code by promoting the reuse of existing formulas.


Gilt is an interface builder that allows dialog boxes and other windows to be created interactively by choosing widgets from a palette and putting them into a window using a mouse [Myers 91d]. Gilt is similar to many other interface builders, including the NeXT Interface Builder, Prototyper from SmethersBarnes for the Macintosh, etc. Gilt stands for the Garnet Interface Layout Tool.

Demonstrational techniques have been added to Gilt in two places: to infer graphical styles from examples, and to infer transformations of data and dependencies to minimize the number of call-back procedures.

Graphical styles in Gilt

In most toolkits, the widgets have many properties that the designer can set, such as the color, font, label string, orientation, size, the minimum and maximum values of a range, etc. Many widgets in the Motif widget set, for example, have nearly 50 different properties that can be set. Most interface builders, including Gilt, provide "property sheets" that allow the designer to specify the desired values. However, it can be quite difficult and time consuming to find and set all of the appropriate properties. To show the magnitude of the problem, many applications contain over 2000 widgets, and the properties for each must be set in a consistent manner. A study has shown that achieving consistency in an interface is a frequently cited problem [Myers 92c].

Another problem for interface designers is laying out the widgets in the window. When the designer places widgets with the mouse, they tend to be uneven and look sloppy. Therefore, most builders provide grids and alignment commands. However, these can be clumsy to use, and they do not ensure that different dialog boxes will have a consistent alignment (for example, that the titles are always centered at the top of the window).

To help solve these problems, Gilt introduces the notions of Graphical Tabs and Graphical Styles into an interface builder, which are more completely described in [Hashimoto 92]. These are based on the styles and tabs in text editors such as Microsoft Word. A "graphical tab" is simply a horizontal or vertical position in the graphics window to which objects can be aligned. A "graphical style" is a named set of properties, which can be applied to widgets. The designer can edit a widget so it has the desired properties, select it, and then define a named style based on it. The values of the properties and the positions of the widgets will be associated with that style name. The style can then be applied to other widgets.

Furthermore, Gilt will try to automatically guess when to apply a style, so the designer does not have to. By guessing the appropriate properties and layout, Gilt makes the user interface design process significantly faster, since users can quickly and imprecisely place widgets, and the system will automatically neaten them. Since the inferencing is based on the styles the user has defined, rather than based on global, default rules, as in earlier systems like Peridot and Druid [Singh 90], the inferred properties and positions are more likely to be correct.

These features in Gilt are classified as "demonstrational" because the user defines a style by example on a particular widget, but the style is automatically generalized so it will work on any of a set of widget types.

A graphical style includes a set of widget properties, and optionally some position information as well. To create a new style, the designer modifies a widget to the desired appearance using the conventional property sheets, selects that widget, and then issues the Define Style command. The designer must then type a style name into the Style editing window that will appear. Gilt compares the widget's current properties with the default values for that widget and copies all that are different. Styles can also include position information. For example, a designer might specify that objects with the Main-Title-Style should use a large bold font, and be centered at the top of the window. The position information for styles can either be with respect to a graphical tab stop, or relative to a previously created object.

Inferring styles

Although the styles mechanism as described above is already quite useful, Gilt goes further and tries to automatically determine when a particular style is appropriate. The style control window (Figure 3) provides three options: no inferencing of styles, styles applied immediately when they are inferred, or a prompt-first mode where the designer is asked if the style should be applied, as in Peridot and Druid [Singh 90]. If the system usually infers the correct style, then the immediate mode will be the most efficient.

Figure 3. The main style control window. This allows styles to be read and written to a file, and style guessing to be turned on and off. Also, the style of the selected object is always echoed at the bottom of the window.

When inferencing is on, Gilt tries to infer a new style whenever a widget is created or moved. The algorithm looks for styles that affect the same type as the widget, and, if the style has a position component, then it checks how close the widget matches the style's position. The types that styles are associated with include strings, button objects (including radio buttons and check boxes), numeric sliders (including both sliders and scroll bars), text input fields, etc. A list is created of all the styles that match, sorted from most likely to least likely.

Any inferencing system will sometimes guess wrong. Thus, it is important to provide appropriate feedback so the users are confident that they are in control and know what Gilt is doing. In immediate mode, the first style on the style list is immediately applied to the graphics, and the name of the style is shown at the bottom of the style control window (Figure 3). The widget will also jump to the inferred position and change appearance. If the inferred style is not correct, the designer can hit the "Try Again" button, which will remove the guessed style and instead apply the next style in the sorted list. This can be repeated until there are no more styles in the list. The "Undo" button can also be hit to remove the guessed style, and return the widget to its original position and properties. In prompt-first mode, the sorted list of all the inferred styles is presented in a window, with the most likely selected. The designer can select a different style, if necessary, and then hit OK or Cancel. When a style is defined, it immediately becomes a candidate for inferencing. This is very useful when a number of widgets will all be created using the same style.

Editing styles

When a style is applied to a widget, either explicitly or inferred, Gilt sets up appropriate pointers and back pointers so that if the style is ever edited, all widgets using that style are immediately updated.

Styles can be edited in two ways. A property sheet can be displayed which shows the current values of the properties for the style, and this can be edited directly. This property sheet has the same format as the ones for the standard widgets. The positions associated with the style can be edited using the appropriate dialog boxes.

Alternatively, the designer can edit the styles in the same way as they were created: by working on example widgets. Whenever a widget is edited that has already been defined to be of a particular style, Gilt pops up a dialog box asking if the edit should change the style itself. The other alternatives are to make the widget no longer belong to the style, or to cancel the change and return the object to its appearance before the edit was attempted.

In the future, we plan to add the ability to have objects use a particular style with exceptions, but this is a complex problem [Johnson 88]. Some of the issues are whether to copy the attributes or retain the link to the original style, what to do to a style when the style it inherits from is changed, and whether to save the inheritance links in the style files, or write out all the style information to each file.

Minimizing call-back procedures in Gilt

Conventional toolkits today require the programmer to attach call-back procedures to most buttons, scroll bars, menu items, and other widgets in the interface. These procedures are called by the system when the user operates the widget in order to notify the application of the user's actions. Unfortunately, real interfaces contain hundreds or thousands of widgets, and therefore many call-back procedures, most of which perform trivial tasks, resulting in a maintenance nightmare. Gilt allows the majority of these procedures to be eliminated [Myers 91d]. The user interface designer can specify by demonstration many of the desired actions and connections among the widgets, so call-backs are only needed for the most significant application actions. In addition, the call-backs that remain are completely insulated from the widgets, so that the application code is better separated from the user interface.

We have observed that many of the call-back procedures are actually used to filter the values from widgets and connect widgets to each other, rather than to perform real application work. By identifying some common tasks that call-backs are used for, and providing other methods for handling the tasks, we have been able to eliminate the need for most call-backs. The tasks can be classified into the following categories:

Gilt provides a standard style of window that allows the filter expressions to be entered. The goal is to minimize the amount of code that needs to be typed to achieve the required transformation. Therefore, much of the filter expression is generated automatically when the designer demonstrates the desired behavior. Other parts can be entered by selecting items from menus. As a last resort, the designer can type the required code. If a call to an application function is necessary in a filter expression, Gilt makes sure that the procedure is called with appropriate high-level parameters, rather than such things as a widget pointer or the string labels. Thus, the call-backs that remain are completely insulated from the user interface.

Gilt tries to automatically pick the appropriate transformation. There are two techniques used to guess what is appropriate. First, the designer can type an example value into the Resulting Filtered Value field at the bottom of the Exported Value Control window (Figure 4-a). In this case, Gilt will try to guess a transformation that will convert the current unfiltered value into the specified value. If none of the built-in transformations is appropriate, then Gilt creates a case statement. The designer can then operate the widget to put it into different states (and therefore to change the unfiltered value), and type the desired filtered value for each case. This allows arbitrary transformations (e.g., converting the German "Fettdruck" or the French "Gras" to :BOLD). The resulting code for the filter is shown in the Filter Expression window.

The second option is used when the designer enters a procedure into the filter expression, and then selects a widget to supply the value to a parameter of the procedure. Here, Gilt tries to find an appropriate transformation so that the widget value will be filtered into the required type of the parameter. A Value Control window will pop up to confirm each transformation, and also to request the designer to specify the transformation if Gilt cannot infer it.

The user can check that the filter expression is achieving the desired result in two ways. First, the interface can be exercised to test the code. Second, the Filter Expression field shows the Lisp code that is being used. In the future, we will be investigating other techniques for showing the transformations that will be usable by non-programmers. For example, the filter expressions might use normal arithmetic expressions, or we might create a special graphical programming language.

Figure 4. (a) The Gilt window that allows the designer to control how values for widgets are filtered. Many of the fields are filled in by Gilt as the designer demonstrates the desired behavior. The Unfiltered Value shows the value as currently provided by the widget before any filtering. The Filter Expression is the Lisp expression to filter the value. The designer can hit the Use Value of Object button to insert a reference to the value of a selected object. The default filter simply copies the original value. The Resulting Filtered Value field shows the final value after the filtering. This field can be edited to show the transformation for the current widget by example. (b) shows the filter expression after a function has been selected from a menu and the widget references have been filled in. (c) shows the additional windows that appear to confirm the transformations that are inferred for the widgets that are referenced in (b).


For enabling and disabling widgets, similar techniques are used. One of the most common dependencies is to enable and disable widgets based on the values of other widgets. To specify this, the designer can operate a widget to have the appropriate value, then enable or disable the dependent widget, and Gilt will fill in the values for the Change my Enable expression. In trying to guess appropriate control expressions for dependent slots, Gilt knows about check boxes and radio buttons being on or off, text fields being empty or having a value, and numbers being zero or non-zero. In addition, if the Change my Enable window is for a set of selectable items (such as a menu or a panel of buttons), the controlling widget can return a list of values, each element of which controls an item.

All the other properties of widgets can be controlled in the same way as enabling. Widgets can be made to be visible and invisible by bringing up a Change my Visible window. Similar windows control properties such as color and font.

To edit the value of any of the filter expressions for a widget, the designer can simply select the widget and bring up the appropriate Control... or Change my... window. The designer can then edit the text of the expression. Alternatively, if the user demonstrates new transformations, these will replace the existing ones as appropriate.


Jade creates dialog boxes from just a list of their contents [Vander Zanden 90]. It uses general rules from graphic design as well as look-and-feel-specific rules in order to create a pleasing presentation. Jade is useful when an application contains a large number of dialog boxes (so that using Gilt would be inconvenient), or when the contents of a dialog box is not known in advance so the dialog box needs to be dynamically generated (so using Gilt would be impossible). Jade stands for the Judgment-based Automatic Dialog Editor.

The demonstrational aspect of Jade is that it will automatically generate the rules that control the layout from examples of the desired picture. An interactive editor is being created that will allow the designer to show the system how the interface should look. This part of the system is still under development.


Garnet contains built-in support for interfaces including gestures. A gestural interface uses the path that the mouse goes through in order to determine the command. For example, the user might draw an "X" over an object to cause it to be deleted, or an "L" shaped motion might mean to create a new rectangle, whereas a circular motion might mean create a new circle. The gestural mechanism in Garnet uses an algorithm that is trainable [Rubine 91a]. This means that the designer gives examples of the desired gestures, and the system uses statistical techniques to match the end-user's gestures against the examples to decide which gesture the end-user is giving.

Lessons Learned

Unlike Peridot, the goal in Garnet is not specifically to investigate demonstrational techniques, but rather to create a usable and efficient collection of tools to create user interfaces. However, we have found that demonstrational techniques are very effective for extending the boundaries of what can be accomplished by direct manipulation. Inferencing is used in most of our demonstrational systems, but it is not particularly sophisticated. In all cases, just one or two rules are needed to decide how and when to generalize. All the techniques are application-specific and ad-hoc, which suggests that some general-purpose toolkit containing demonstrational techniques would probably not be helpful. As with other demonstrational interfaces, the primary problems in Garnet have been how to provide appropriate feedback for the generalizations so the users are comfortable with them, and how to allow editing. In most cases, the technique used currently is just to show the Lisp code that was inferred, and require that the user directly edit the code, but we plan to investigate more sophisticated methods in the future.


The Garnet research is sponsored by the Avionics Lab, Wright Research and Development Center, Aeronautical Systems Division (AFSC), U. S. Air Force, Wright-Patterson AFB, OH 45433-6543 under Contract F33615-90-C-1465, Arpa Order No. 7597.

The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government.

Garnet -- Lapidary, C32, Gilt

Uses and Users

Application domain: User Interface Development Environment

Intended Users: Primarily programmers

User Interaction

How does the user create, execute and modify programs?
Lapidary: Dialog boxes are used to establish graphical constraints on example objects, and the constraints will be generalized and applied to runtime application objects. Also, an example object can be used to demonstrate how an application object's properties should change in response to a mouse click.
C32: The user can copy and paste a formula from an example slot, and it will be generalized so that it will be applicable to the destination slot.
Gilt: To generate filter expressions, the user can give an example of a return value in the proper format and data type. The user can also demonstrate how certain values of one widget should enable or disable another widget. For styles, an example widget with the correct styles is created.

Feedback about capabilities and inferences:
Lapidary: Shows the inference in dialog boxes.
C32: Dialog boxes show the inferences, and the user must hit the "OK" button.
Gilt: When inferring a graphical style, the style is immediately applied, so the object's appearance will change. Also, the style name is displayed. For filter expressions, the inferred Lisp code is displayed.


Inferencing: Garnet applies fixed heuristics based on domain knowledge. Note that the inferencing is based on resulting graphical objects, not on a trace of actions.

Program constructs: Variables


Types and sources of information: Hard-wired into the various programs.
C32: Information associating slots with physical locations on an object. Information about typical copy-paste transformations.


Machine, language, size, date: X/11 and Common Lisp. Various platforms. 1988-present.

back to ... Table of Contents Watch What I Do