back to ... Table of Contents Watch What I Do

NOTE: The images in this chapter could not be converted properly for use on the web


Creating User Interfaces by Demonstration[ ]

Brad A. Myers


Peridot, completed in 1987, was one of the first programs to use programming-by-example with inferencing as part of its user interface. The success of Peridot seemed to inspire many other applications of the programming-by-example idea. This chapter presents an overview on the use of programming by example in Peridot. There are many other articles about other aspects [Myers 86b] [Myers 87a] [Myers 90b] [Myers 91e] and the complete system is described in a book [Myers 88].

Peridot is an experimental tool for creating graphical, highly-interactive user interfaces. One of the primary goals of Peridot is to allow these interfaces to be created by non-programmers. To achieve this, Peridot uses the techniques of Visual Programming, Programming by Example, Constraints and Plausible Inferencing. Peridot was developed as part of the author's Ph.D. research while at the University of Toronto.

Peridot stands for Programming by Example for Real-time Interface Design Obviating Typing, and is a working prototype implemented in Interlisp-D on the Xerox 1109 DandeTiger workstation. Peridot can create many types of interaction techniques (sometimes called "widgets"), which are the low-level components of user interfaces. This includes most kinds of menus, property sheets, light buttons, radio buttons, scroll bars, two-dimensional scroll boxes, percent-done progress indicators, graphical potentiometers, sliders, iconic and title line controls for windows, and many others. Thus, Peridot can create almost all of the Apple Macintosh interface, as well as many new interfaces, such as those which use multiple input devices concurrently. Peridot also created its own user interface.

Two of the most important components of the design for highly-interactive user interfaces are the graphical presentation and the way the graphics change due to the mouse and other input devices. Unfortunately, previous tools for creating user interfaces have not adequately addressed these components. In order to allow the graphical parts of the user interface to be easily specified, Peridot allows the user interface designer to draw a picture of what the user interface should look like using a special drawing package. In order to specify how the user interface should work with the mouse, Peridot allows the designer to move the mouse and toggle its buttons. While this is happening, Peridot creates code for the interface and its connection to actual application programs. The code produced is not simply a transcript of the designer's actions, however, because this would not provide sufficient functionality. For example, a pop-up menu might be designed with a particular set of strings, but the same code should work for any list of strings. Therefore, Peridot creates parameterized procedures, where parts of the interface can depend on the values of the parameters. For example, the size of the menu might be adjusted to exactly fit around the strings. Peridot uses some simple artificial intelligence techniques to infer how the graphics and mouse should change based on the actual values for the parameters.

In order to allow interaction technique procedures to be created in a direct manipulation manner [Shneiderman 83], Peridot has the designer provide example values for each parameter. For instance, when creating a menu, the designer would provide an example list of strings to be displayed in the menu. Using Programming by Example, Peridot generalizes from the given examples to create a general-purpose procedure. Since programs are created using graphical techniques, Peridot can also be classified as a Visual Programming system. Constraints, which are relationships that are maintained automatically by the system, are used in two ways in Peridot. Graphical Constraints tie various graphical objects together so that when one changes the others will be updated automatically. Data Constraints are used to insure that graphical objects are updated whenever any special variables, called Active Values, are updated.

Related Work

Because programming user interfaces is difficult and expensive, there has been a growing effort to create tools to help with this task. Sometimes, the tools are simply a library of low-level procedures that create the components of the interface. There may be procedures that display menus, on-screen buttons, scroll bars, etc. In this case, the library is usually called a "user interface toolkit," and an example of this is the Macintosh Toolbox [Apple 85].

Some user interface tools also help with the creation and management of these low level components. These higher-level tools are often called "User Interface Management Systems" (UIMSs) [Myers 89a] [Olsen 87] [Pfaff 85]. Many UIMSs allow some part of the presentation of the user interface to be specified graphically. For example, Menulay [Buxton 83] allows the designer to place text, graphical potentiometers, iconic pictures, and light buttons on the screen and see exactly what the end user will see when the application is run. The interfaces that Menulay can create are essentially a series of mostly static frames, where the parts that can be manipulated by the user are programmed by hand in advance. Thus Menulay allows the designer, who does not need to be able to program, to lay out the interface on the screen. Modern systems using similar techniques include most interface builders, like the NeXT Interface Builder.

These systems are limited to using graphical techniques for specifying the placement and size of picture and user interface elements (e.g., where menus are located and what type of light button to place where), but Peridot was the first system that allows both the appearance and the dynamic behavior of the elements to be specified in a graphical, non-textual manner.

Peridot supports this by incorporating ideas from a number of Visual Programming and Programming by Example systems.

"Visual Programming" is the use of graphics to create computer programs [Myers 90a]. "Programming by Example" is (by my definition) when the system generalizes from example values provided by the user. Many Visual Programming and Example-Based Programming systems have been designed to provide non-programmers with programming capabilities. For example, Rehearsal World (Chapter 4) uses the metaphor of a stage to allow the user to create teaching programs by connecting together pre-existing "actors" (which are represented by pictures).

The visual programming systems that are most relevant to Peridot also use programming by example. The seminal system is Pygmalion (Chapter 1) which supports programming using icons. SmallStar (Chapter 5) allows the end user to program a prototype version of the Star office workstation. When programming, the user simply enters program mode, performs the operations that are to be remembered, and then leaves program mode. The operations are executed in the actual user interface of the system, which the user already knows. Since the system does not use inferencing, the user must differentiate constants from variables and explicitly add control structures (loops and conditionals). This is performed after the demonstration is completed by editing a textual representation of the program.

Peridot extends this idea by using inferencing to avoid the need for the textual program representation--the system guesses the generalizations from the examples. Many previous systems that tried to infer from examples were not successful because the domain was too general. For example, one system [Shaw 75] tried to generate Lisp programs from examples of input/output pairs. Programming by example has been more successful in limited domains, such as generating editor macros [Nix 83] and simple transformations on pictures [Maulsby 89a].

Another important component of Peridot is the use of constraints [Leler 88], which are relationships among objects and data that must hold even when the objects are manipulated. Peridot uses two kinds of constraints. Graphical constraints relate one graphic object to another, and data constraints insure that a graphical object has a particular relationship to a data value.

Figure 1. The windows that are used when creating a procedure using Peridot. The top two windows display the parameters and active values, and example values for each. The large center window displays the user interface being created. The lower window displays the prompts and error messages from Peridot. The code window on the right is optional and is mainly used for debugging Peridot itself. The designer gives all commands using the Peridot command menu on the left. The top portion of the menu contains the most common commands. The menu and scroll bar were created using Peridot.

The idea of graphical constraints was first used in Sketchpad [Sutherland 63]. Thinglab [Borning 79] [Borning 81] and its descendants [Borning 86a] extended the ideas in Sketchpad to provide generalized programming with constraints. Juno [Nelson 85] added a textual view of the constraints and iconic techniques to define them. Constraints in Peridot, and some other systems including Apogee [Henry 88] and Grow [Barth 86], are one-directional. This means that for any property, there can be at most one formula that calculates its value. Other systems, including Sketchpad, Thinglab, and Juno use multi-way constraints, which means that there can be multiple formulas for a property or that constraint expressions can be solved for any of the objects referenced. Multi-way constraints usually require a complex constraint satisfaction algorithm that may not operate quickly enough for highly-interactive user interfaces [Borning 86b], whereas one-directional constraints as in Peridot use a much simpler, real-time algorithm.

Data constraints insure that a graphical object has a particular relationship to a data value. An example of a data constraint is that the diameter of a circle is the same as the value of a global variable, so the size of the circle will change if the variable is set. These are used in the Process Visualization System [Foley 86], which was influenced by "triggers" and "alerters" in database management systems [Buneman 79]. They are also similar to the "Control" values in GRINS [Olsen 85] except that they are programmed by example rather than textually and can be executed immediately without waiting for compilation.

The data constraints in Peridot can only be connected to active values. Active values are like parameters to a procedure except that when their value changes, the graphics that depend on them are automatically redisplayed based on the new value. Active values have been used in artificial intelligence systems and environments, such as Steamer [Stevens 83], KEE [Ramamoorthy 87] and LOOPS [Stefik 86].

Overview of Peridot

When creating an interface with Peridot, the designer must first give a name to the procedure that will be called by the application to cause the interface to appear. This procedure can have parameters, which are used for aspects of the interface that change from call to call, but do not change while the interface is running, such as the strings in a menu. Constraints will insure that the interface appropriately reflects whatever values for the parameters are supplied. The designer also specifies the active values associated with the procedure which control the parts that do change while running, such as the particular string that the mouse is over. A data constraint will insure that the feedback graphics, such as a reverse video rectangle, shows the item that the active value refers to.

Figure 1 shows a typical configuration when running Peridot. The top window shows the parameters to the procedure and an example value for each parameter. Below the parameters are the active values used by the procedure. The menu on the left is used for giving all of the Peridot commands. This menu is divided into two sections with the most common commands on the top. The center window shows the resulting user interface. The bottom window is used for printing error messages and for prompting the designer. The window on the right shows the code that Peridot is creating for this procedure. This window is mainly for debugging since it is not necessary for the designer to view or use the code to perform any operations in Peridot. The normal window configuration, in fact, does not even display this window, as shown in Figure 2-b.

While the designer is creating a user interface, Peridot is continuously trying to guess how the various pieces are related to each other and how to generalize from the examples. Because any inferencing system will occasionally guess wrong, Peridot uses three strategies to protect against incorrect inferences. First, Peridot always asks the designer if guesses are correct by printing a message in the prompt window; second, the results of the inferences can be immediately seen and executed; and third, the inferences can be undone if they are in error.

Example of Peridot
in Action

The best way to demonstrate how easy it is to create a user interface with Peridot is to work through an example. Here, a property sheet will be created. Other examples are available in other papers [Myers 86b] [Myers 87a] [Myers 88], but it is easiest to appreciate how easy Peridot is to use by seeing a videotape [Myers 87b].

To start Peridot, the designer presses a mouse button in the Peridot logo window, and the Peridot prompt window is displayed (Figure 2-a). In this window, the designer types the name of the procedure, the names for all parameters and active values for this procedure, and an example value for each. The example procedure will be called by application programs with a single argument: a list of labels.

Figure 2-a

Figure 2-b

Peridot then displays the menu and windows shown in Figure 2-b and allows the designer to begin drawing the picture. In Figure 2-b, the designer has created a gray rectangle to represent a "drop shadow" for a button. In 2-c, the designer has drawn a black rectangle to represent the background of the button, and Peridot has noticed that this rectangle seems to be the same size as the gray rectangle, offset by a constant 9 pixels. In the prompt area, it is asking the designer to confirm this constraint. (The "N" and "WRT" mentioned in the prompt area are blinking icons that point to objects. In Figure 2-c, the "N" is visible.) The designer types y for "yes," and Peridot immediately adjusts the black rectangle to be exactly the same size as the gray one (Figure 2-d). If the gray rectangle's size were now changed, the black rectangle would change size also, because a graphical constraint has been established that keeps both rectangles the same size.

Figure 2-c

Next, the designer draws a white rectangle inside the black one (Figure 2-e) and Peridot correctly infers that this rectangle should be evenly nested inside the black one. In Figure 2-f, the designer has selected the first element of the parameter Items, which is the string Bold, and used that as the string to display. Peridot infers that it is centered to the right of the white rectangle. The code that is produced for this string refers to the first element of the first parameter, whatever that is, rather than to the constant string "Bold," so that any value used for the parameter will be displayed.

Figure 2-d

Figure 2-e

Figure 2-f

Figure 2-g

Next, the designer selects all the objects created so far and specifies that they should be copied to a new position (Figure 2-g). Then the designer edits the second string to refer to the second element of the parameter (Figure 2-h). At this point, Peridot notices that the designer has used the first two elements of a list in the interface, and asks whether the rest of the elements of the list should be displayed in the same way, as part of an iteration over all the elements of the list. The designer confirms this, and the rest are immediately shown (Figure 2-i). In order to perform this conversion, Peridot had to determine what graphical objects should participate in the loop and how they should change in each cycle. Now the presentation aspects of the property sheet are finished.

Figure 2-h

Figure 2-i

Figure 2-j

Figure 2-k

Next, the designer places the iconic picture of a check-mark centered inside one of the boxes (Figure 2-j). This will be used to show which items are selected. In order to demonstrate that this should be selectable by the mouse, the "simulated mouse" icon is used (see Figure 2-k). The real mouse cannot be used since it is used for giving Peridot commands. The nose of the simulated mouse is placed over the check mark with the middle button down, and the MOUSEDependent command is given, which causes Peridot to begin guessing the dependencies on the mouse. Since there is only one active value in this example (Selected-Props), Peridot guesses that the check-mark should depend on this active value. Since the example value of that active value is a list, Peridot guesses that multiple items are allowed, and that a check mark should appear for each one that is in the list. If the example value of the active value held a single item, then Peridot would have guessed that only a single selection was allowed. The designer is asked to confirm these guesses in the prompt window. Peridot then shows the check marks displayed in the boxes next to Italic and Underline since these are the current value of Selected-Props. Finally, the designer is asked whether pressing the middle button should toggle, set, or clear the selected object, and the designer types t for toggle. The user interface is now complete and it can be tested with either the simulated mouse, or else Peridot can be put into "Run Mode" and the real mouse can be used. To set the return value of the function, the active value is selected and the Add-Return-Stmt command is executed.

The PropSheet procedure that has been created can now be used outside of Peridot as part of application programs. It is parameterized as to the list of items that are displayed, so it can be called with an entirely different list of strings, even if that list has a different number of elements (Figure 2-l), and the return value will be a list of the string selected.

Programming-by-example Aspects of Peridot

As shown in the example, Peridot requires that the designer provide examples of typical values for the parameters and active values of the procedure. These values are used to display the user interface while it is being designed. In addition, Peridot automatically guesses from the examples the appropriate graphical constraints and control structures, and how actions depend on the mouse.

Allowing the designer to work on example data while developing an interface with Peridot is a crucial component, and one of the primary contributors to its success. Otherwise, the designer could not see what the interface would look like. (What would be displayed while designing a menu if there was no list of strings?) The examples allow Peridot to use direct manipulation techniques so the design process is concrete, rather than abstract. The interface seems more like a drawing program, such as Apple MacDraw, than a programming system.

The motivation for this style is that people make fewer errors when dealing with specific examples rather than abstract ideas [Smith 75]. The programmer does not need to try to keep in mind the large and complex state of the system at each point of the computation if it is displayed on the screen. In addition, errors are usually visible immediately.

This section discusses in detail how Peridot uses Programming by Example.

How examples are used

Many PBE systems require the user to provide multiple examples in order to generate code. In some cases Peridot infers code from single examples. This is possible because the designer is required to explicitly give a command to cause Peridot to perform the inferencing. For example, the designer issues theMOUSE-

Figure 2-l

Dependent command to tell Peridot to look at the mouse position and infer the generalization for the operation. For iterations, however, the designer is required to give two examples, and Peridot can therefore usually infer the need for an iteration without an explicit command from the user. For example, in Figure 2-h, Peridot guessed that an iteration was needed since there were two sets of buttons, and two elements of the parameter Items had been used.

Peridot also allows the designer to demonstrate conditionals that display special graphics and serve as exceptions to the normal way the mouse dependencies work. For example, some items of the menu might be shown in gray if they are illegal, and horizontal lines might replace certain items (as in Figure 3). The designer can demonstrate that special actions should happen when the mouse is over these areas. For example, the reverse video feedback rectangle in Figure 3 will not appear over the items shown in gray or over the lines. These are sometimes called "negative examples" since they show the system what not to do. The conditions under which these exceptions can happen in Peridot are fairly limited, but the graphical response that is shown for exceptions is totally under the control of the designer.

How inferencing is used

In order to make Peridot easier to use, it automatically guesses certain relationships. This frees the designer from having to know when and how to specify these relationships. Peridot uses simple condition-action rules [Charniak 85] to implement these guesses. This approach is called "Plausible Inferencing" or "Abduction" in the Artificial Intelligence literature. The condition part of the rules determines whether the rule seems to apply in the current context. As shown in the example, if the condition passes, then the designer is asked whether to apply the rule or not using an English message attached to the rule. If the designer answers "yes" then the action part of the rule is applied, which changes the code of the procedure to add a graphical constraint.
Figure 3. A menu created using Peridot. The horizontal dotted lines and the gray items are exceptions to the normal way the text is displayed. The feedback black rectangle will not appear over the gray item "Enlarge" or over the dotted lines.

The rules in Peridot are simple; much simpler than those used in typical artificial intelligence systems. Furthermore, there are only about 60 rules used in all of Peridot. The goal was to see if simple mechanisms would be sufficient, which seems to be true.

Peridot uses rule-based inferencing in four ways: to infer the graphical constraints that relate one object to another, to infer when control structures are appropriate, to differentiate variables from constants, and to infer how the mouse should affect the user interface. The complete set of rules is provided in [Myers 88].

Inferring graphical constraints

Peridot infers how the various graphical objects are related to each other. Previous systems that have used constraints, such as Thinglab [Borning 81] and Juno [Nelson 85], have required that the constraints be explicitly specified by the user. This has made these systems difficult to use. Previous picture beautification systems, such as PED [Pavlidis 85], automatically infer a form of constraints, but these systems are not popular because they guess wrong too often, and because it is difficult to determine what the program has done and to repair any damage. One reason that Peridot is more successful is that it guesses correctly more frequently since it only needs to deal with the relationships that are typical in user interfaces, rather than all possible relationships that might be used in a general drawing. If the designer wants other relationships, they can be explicitly specified or, if they occur frequently, a programmer can easily add them to the rule set. Another reason for Peridot's success is that it assumes that guesses will occasionally be wrong. Therefore, it always reports to the designer the rule that it is planning to apply and allows the designer to confirm or prevent its application. This gives the designer confidence that the system is not mysteriously doing strange and possibly erroneous things. In addition, the results of the inferences are always immediately visible (the objects redraw themselves after every rule is applied), so the designer can view the results and see whether they were correct or not.

Another benefit of inferring graphical constraints is that they allow the designer to draw the picture quickly and sloppily and then have Peridot automatically "beautify" it by enforcing the constraints. This technique seems to be faster than using gridding or explicitly specifying constraints, even though Peridot occasionally guesses wrong. The designers that have used Peridot felt that this benefit outweighed the time to answer the questions and correct the occasional erroneous inferences.

The rules that Peridot applies are specific to the types of the objects drawn. For example, it is more likely for a string to be centered at the top of a rectangle than it is for another rectangle to be. Some of the rules specify all of the geometric properties of an object. Examples of these are that a rectangle is nested inside another rectangle with an even border all around, or that a string is centered vertically to the right of a rectangle. Other rules only constrain some of the properties of an object. For example, one rule might cause the width and left of a rectangle to be constrained by another rectangle, and another rule may constrain the top and height by a string. These two rules are used by the feedback black rectangle in Figure 3. Other rules that constrain only some properties of objects are that they have the same size (which does not constrain the position), or that one is centered inside another (no constraint on the sizes). In general, there are constraints for most of the simple relationships found in typical user interfaces. There are currently 50 rules for inferring graphical constraints. Of these, 16 were added based on user testing. Since most of the additional rules were added from the initial users and no new rules were needed for later users, it is expected that few new rules will be needed in the future.

The rules are expressed as simple Lisp expressions, with a test, constraint, and a message string for prompting the user. The tests naturally have thresholds built into them, so the user does not have to draw the graphics exactly. When the user draws or edits an object, Peridot goes through the rules in order trying each test. The order is determined by the types of the objects, the specificity of the rule (the rules that constrain all of the properties of the object are checked first), and which ones seemed to be the most common. [Myers 88] shows the ordering for all the rules.

The message part of the rule is the string used to prompt the user for whether to apply the rule or not, as shown in the example above. If the designer answers "yes," then the constraint is applied. If the constraint has parameters, such as how far apart the objects should be, the designer can answer "almost" and supply a new value for the parameters. If the designer answers "no," then other rules are attempted. For graphical constraints, Peridot almost always infers the correct rule within three guesses, and about 80% of the time, it is correct on its first guess.

Inferring control structures

Peridot automatically guesses when control structures such as iterations are appropriate. The criterion for these inferences in Peridot is straightforward. Iterations are inferred whenever the first two elements of a list are used, as shown in the example (Figure 2-h). To create a dependency on an active value or a parameter, the designer must explicitly select an element of these in the upper window, and then specify which property of the object depends on the selected element. Peridot automatically keeps track of these dependencies and looks for uses of consecutive elements.

Conditional control structures are automatically inferred when objects depend on the mouse, as discussed below. In addition, the designer can explicitly specify that either a conditional or an iteration is desired by executing commands from the menu.

After Peridot decides that a control structure is necessary, it must then decide what graphical objects participate in it. As shown in the property sheet of Figure 2-i, multiple graphical objects may be drawn in each cycle of an iteration (here, the black, white, and gray rectangles and the string are repeated in each cycle). To make the determination easier, Peridot requires that objects be drawn in the same order in each cycle. Clearly, the designer will have to be careful to create objects in the correct order, but the Copy command does this automatically.

There are other restrictions on inferred iterations in the current implementation. First, the two sets of objects must be created contiguously. This makes it significantly easier to determine how many objects to include in the cycle. Another requirement is that the element that comes from the list must be the last item drawn in each cycle. Again, this makes it easier on the implementation, since it can simply count the items in between the two items that come from the list. If the element is not the last item drawn (for example, in Figure 2-h, if the string had been drawn before the white rectangle), Peridot will still notice that an iteration is desired, but it will not be able to automatically identify which elements participate in the iteration. In this case, a message is printed asking the designer to select all the items that are part of the iteration. Users of Peridot have not even noticed either of these restrictions, and they would not be hard to remove if that would be appropriate.

Differentiating variables from constants

After the objects that participate in a control structure are identified, Peridot must determine which properties of the objects are constant and which change. For example, in the property sheet of Figure 2-i, Peridot determined that the text value of the string, and the y value of the rectangles and the strings should change in each cycle of the iteration, but the rest of the properties remain constant.

Other systems found that inferring variables from constants was difficult, but Peridot's simple mechanism has been successful. Again, this is due to the limited domain; graphical objects in user interfaces typically change in simple ways.

Peridot compares each pair of objects and determines which attributes change and which are constant. For the attributes that change, Peridot only detects straightforward differences, including:

Peridot can easily determine which one of these to use by looking at the values of the properties. This simple strategy makes the inferencing straightforward. If Peridot detects that a control structure is plausible but it cannot calculate how to change the objects, the designer is notified.

The major complication in generating the code for the control structures is insuring that the constraints refer to the correct objects. When the designer creates the example drawing, the inferred graphical constraints refer to the particular example objects. When these are generalized into a control structure such as an iteration, however, Peridot must insure that the constraints refer to the appropriate objects. For example, in Figure 2-i, each of the strings must be centered on the right of the correct corresponding rectangle. To solve this problem, Peridot generates the object names when the iteration is executed at run time, and then constructs the constraints based on these generated names. The later Garnet system (Chapter 10) uses variables in the constraints to solve this problem [Vander Zanden 91a].

The automatic inferencing of graphical constraints might also be classified as differentiating variables from constants, since the properties of objects that have constraints can change, and those that do not have constraints retain their original constant values.

Inferring mouse operations

Another form of control structure that Peridot infers is how the mouse should affect the graphics. This includes which objects should be affected by the mouse, and when and how they should change. When the designer gives the MOUSEDependent command, Peridot looks under the simulated mouse to determine which objects are affected and where the mouse should be for the operation to be active (as shown in Figure 2-k).

The designer specifies when the operation should happen by toggling the state of the buttons on the simulated mouse. The interaction can start after single or multiple button presses (e.g. double-clicking) and either on the down or up transition of the button.

Next, Peridot infers which object should be affected by the mouse. Usually, it is the object under the simulated mouse's nose (as in Figure 2-k), but otherwise the designer can select the appropriate object or objects which serve as the feedback graphics.

Next, Peridot infers how the objects should change with the mouse. The possibilities are:

Peridot guesses which of these is appropriate by looking at the constraints on the graphical objects that are affected by the mouse. For example, the check mark in Figure 2-j is centered inside an object that is part of an iteration, so Peridot guesses it should pick items out of the iteration. If the object is constrained to move inside another object, then Peridot guesses that it should operate as a range like a scroll bar. Otherwise, Peridot guesses that the object should either blink on and off or move freely.

Peridot also needs to know where the interaction should operate. This is usually the object under the mouse's nose (but ignoring the feedback object). If that object is a member of an iteration, however, Peridot asks the user if the operation should happen whenever the mouse is over any of the elements of the iteration. In Figure 2-k, the check-mark interaction works for any of the boxes, not just the example one it was demonstrated over. If the object for the "where" is not under the nose, the user can select it.

In order to confirm all of these inferences, Peridot asks the designer a series of questions after the MOUSEDependent command is selected. Some additional questions allow the designer to specify information that cannot be inferred from the demonstration. This includes which active value is affected by this interaction, and whether the operation should operate continuously while the mouse button is held down (i.e., have the feedback follow the mouse while the button is depressed, as in a menu) or only happen once when the button goes down. Another question determines whether the interaction adds the item under the mouse to the selected set, removes it, or adds it if it is not there and removes it if it is (toggle). In Figure 2-k, the designer used toggle, which is the default.

Including confirming the inferences, creating an interaction requires answering about eight questions, all with single-letter responses. Peridot almost always is correct in the guesses for interactions, since there is a small number of possibilities for each choice, and these are easy to identify from the demonstration.

Once all the properties have been determined, code is generated which causes a special "interactor" object to be created at run-time. This object will wait for the appropriate start event when the mouse is over the correct object, and then set the active value appropriately, which will cause the feedback to move. If desired, the designer can add application notification procedures or filters to this active value.

Editing programs

Since Peridot is designed to be a prototyping tool, it is imperative that it support easy editing of all parts of the interface. When editing the graphical aspects, the designer can simply select graphical objects and then give an editing command. These commands allow the objects to be erased, copied, or changed either using the mouse or by typing in new values for the properties. It is as easy to edit the graphical parts of the interface as it is to change a picture in a drawing program such as Apple MacDraw.

It is harder to edit control structures and mouse interactions since they do not have visual representations on the screen that can be selected. Some systems have required the user to learn a textual representation for the actions in order to allow editing (e.g. SmallStar), but this is undesirable. It was felt that a language would be too hard for non-programmers to learn. Therefore, Peridot uses more direct techniques.

For editing control structures, the designer can simply select any graphical object and give an editing command. If that object is part of a control structure, Peridot will inquire whether a modification to the control structure itself is desired, or whether there should be an exception to the normal way the control structure works.

If the designer specifies that the control structure itself should be edited, then Peridot returns the display to the original objects from which the control structure was created. For an iteration, this is the original two sets of elements, and for a conditional, it is the original one element. For example, if the designer decides to change the size of the boxes in Figure 2-i, Peridot would first return the display to look like Figure 2-h before allowing the edits. Now the designer can use all the normal editing commands to change the picture as desired. When editing is complete, then the Iteration or Conditional command is given to reinvoke the control structure.

This technique is used for three reasons. First, it is easier to insure that the designer's edits always make sense. Otherwise, if the designer changed the fourth item of a list, what would this mean? Second, if multiple items are generated by the control structure, the designer might make intermediate edits (such as deleting an object from one group) that would cause Peridot to be unable to show the control structure consistently. Third, the list controlling the iteration or conditional might happen to have only 1 or 0 items in it when the designer performed the edit, in which case there would not be two groups of objects for iterations or one for a conditional, so there would be nothing for the designer to select. Returning to the original two groups of objects allows the designer to have full freedom to edit in any way desired, using all the conventional editing commands.

It is even harder to edit mouse interactions since there is nothing to select. Peridot provides two ways to edit interactions. First, an interaction can be re-demonstrated, and Peridot will inquire if the new interaction should replace the old one or run in parallel. Running in parallel is often used since it is common to provide two different ways to produce the same effect. For example in a scroll bar, pressing on the arrows or moving the indicator up and down might affect the same active value.

The second way to edit interactions is to select an active value and give the DeleteInteractions command. Peridot then prints in the prompt window a textual description of each interaction that affects that active value, and asks if it should be deleted.

Any system that allows editing will have a certain "grain" finer than which items must be re-entered from scratch. For example, drawing packages rarely allow a rectangle to be edited into a circle (keeping some of its properties); instead, the rectangle must be erased and a circle drawn instead. Similarly, the grain chosen in Peridot for mouse interactions is one entire interaction. Since individual interactions are small (e.g. setting an active value when the left mouse button goes down over a menu item), this should not be a large burden. A complex interface is typically constructed from a number of small interactions, each of which takes only a few seconds to define. The added complexity for the designer of learning extra editing commands does not seem appropriate given the ease of re-specification. If this turns out to be obnoxious to designers in the future, however, then it would not be too difficult to allow the various parts of the interaction (the active area, the buttons, the action, the exceptions, etc.) to be edited separately. The later Lapidary system in Garnet allows the individual parts to be edited using dialog boxes (Chapter 10).

Evaluation of Peridot

It is difficult to quantify formally the range of user interfaces that Peridot can create since there is no comprehensive taxonomy of interaction techniques. Informally, it is easier to describe Peridot's range by example: it can create menus of almost any form (with single or multiple items selected), property sheets, light buttons, radio buttons, scroll bars, two-dimensional scroll boxes, percent-done progress indicators, graphical potentiometers, sliders, iconic and title line controls for windows, dynamic bar charts, and many others. Thus, Peridot can create almost all of the Apple Macintosh interface (all but parts that use text input), as well as many new interfaces, such as those which use multiple input devices concurrently. For example, my advisors asked if Peridot could make a menu where the items move back and forth as they were selected (see Figure 4-a) and a window where pressing in different areas caused it to grow from different sides (like the X/10 uwm window manager), and Peridot was able to handle both of these. Other interesting effects that Peridot can provide are a button where the body of the button appears to go into the screen in "3-D" because it moves to cover its shadow when pressed (see Figure 4-b), and animations between the initial and final forms.

(a) (b)

Figure 4. The graphical response to the mouse actions in Peridot is only limited by the creativity of the designer. In (a), text items move left and right, and in (b), number-pad buttons pretend to move in three dimensions.

Peridot does not handle any keyboard input, but the ideas in Peridot could be extended to handle the keyboard and other types of input devices. As it is currently implemented, Peridot also cannot handle objects that are created dynamically at run-time. For example, you cannot use Peridot to draw or define the behavior of circuit elements that the end user will create from a palette (although you could design the palette). Similarly, Peridot will not handle displays of trees or graphs or other application-specific data. Also, Peridot does not handle higher-level sequencing of the low level user interface elements; i.e., to get a sub-menu to appear after a menu item is selected, a Lisp procedure must be written. Most of these restrictions have been lifted in the later Lapidary system (chapter 10).

In order to evaluate how easy Peridot is to use, an informal experiment was run where 10 people used the system for about two hours each. Of these people, 5 were experienced programmers and 5 were non-programmers who had some experience using a mouse. The results of this test were encouraging. After about 1 1/2 hours of guided instruction with Peridot, the subjects were able to create a menu of their own design unassisted. This demonstrates that one basic goal of Peridot was fulfilled: non-programmers are able to create user interface elements using Peridot.

In addition, programmers will appreciate using Peridot to define graphical parts of user interfaces because it is faster and more natural than conventional programming. As a small, informal experiment, six expert programmers implemented a particular menu using their favorite hardware and software environments. Some wrote the menu by hand and others modified existing code. The results were that with Peridot, the time to create the menu ranged from 4 to 15 minutes (after the learning time), but programming took between 50 and 500 minutes [Myers 88]. Therefore, using Peridot appears to be significantly faster.


Peridot uses Constraints, Programming by Example and Visual Programming to allow non-programmers to create user interfaces by demonstration. Unlike many previous attempts to use these techniques, they have been effectively used and efficiently integrated together in Peridot. Graphical and Data Constraints are used to maintain important relationships, Programming by Example is used so that these relationships can be automatically inferred from examples, and Visual Programming allows the designer to create the user interface graphically and see the interface as it develops. As the user interface is designed, Peridot creates reasonably efficient code that can be used with actual application programs. Therefore, the user interface designer is not necessarily just prototyping; the actual end user interface code can be produced, even by a non-programmer.

The algorithms in the generated code are comparable to those that would be created by hand, and do not require any complex constraint-satisfaction techniques. The procedures that are created can have parameters and return values, just like the procedures normally found in interaction technique libraries. In addition, calls to application programs from within the user interface are appropriately parameterized. Therefore, Peridot promotes structured design and creates well-structured code (unlike many earlier UIMSs). Peridot was the first user interface management system to provide all of these capabilities.

Peridot also demonstrates that Visual Programming, Programming by Example and Constraints can be successfully integrated in practical, useful and easy-to-use systems.


I want to thank Xerox Canada, Inc. for the donation of the Xerox workstations and Interlisp environment on which Peridot was built. This research was also partially funded by the National Science and Engineering Research Council (NSERC) of Canada. I would like to also thank my thesis advisor, Bill Buxton, for his support and good ideas.


Uses and Users

Application domain: Graphical User Interface Design Environments

Tasks within the domain: Primarily creating new widgets

Intended users: Graphical user interface designers who are non-programmers

User Interaction

How does the user create, execute and modify programs?
The user creates a procedure by first typing in its name, its parameters, its active values, and an example value for each. The user then demonstrates 1) how to create the graphical presentation of the interface and 2) how the interface should respond to end-user mouse actions.
Program is represented as Lisp code.

Feedback about capabilities and inferences:
The system describes each inference in English and asks the user to verify it.


Peridot has a set of condition-action rules. If the condition applies to the current context, the user is asked if the rule should be applied. If so, the action part of the rule is applied. Inferences are used in three ways: to add graphical constraints, to detect iterations, and to determine how the mouse should affect the graphics. Note that the inferencing is based on resulting graphical objects, not on a trace of actions.

Types of examples: The user gives one example for graphical constraints and two examples for iteration over a list.

Program constructs:
Supports establishing graphical constraints between objects.
Iteration over elements in a list.
Conditionals in certain fixed cases.


Types and sources of information:
Peridot has built-in domain knowledge about typical user interface presentations (e.g. displaying all items in a list) and behaviors (e.g. selecting multiple items in a list).
Peridot has rules to infer 1) graphical constraints between objects, 2) when to add loops and conditionals, 3) whether an item is a constant or a variable, and 4) how the interface should respond to the mouse.


Machine, language, size, date: Interlisp-D, Xerox 1109 DandeTiger, 1987.

back to ... Table of Contents Watch What I Do