Representing histories of graphical applications presents a special challenge, since they deal with objects that may have no readily understood textual representation. Typically, textual applications use the spatial browsing technique to represent their histories. They list commands executed during a session, one after another in a sequence. However, commands in graphical applications may have no textual name, and may deal with graphical attributes and geometric properties that cannot be conveyed intuitively in textual terms. Thus many graphical applications instead rely on temporal reexecution to communicate their command histories. Such applications can be scrolled backwards in time, while the effects of the operations are played back on the application canvas. Essentially this provides an animation of the steps performed in a session. Temporal reexecution emphasizes the effects of the commands, while spatial browsing focuses on the commands themselves.

A disadvantage of temporal reexecution is that it provides little context. At any moment, the history represents only a single point in time. Editing such a representation is difficult, analogous to editing a textual document in a line editor. There is no structure to the presentation; all commands, no matter how trivial are represented at the same level. We have developed another technique, called editable graphical histories, that combines aspects of spatial browsing and temporal reexecution to represent sequences of graphical interactions without the above drawbacks. Editable graphical histories, like comic strips, represent a series of events by a sequence of graphical panels. Examples of editable graphical histories appear in Chapters 12 and 15. This chapter discusses how this representation has evolved from its initial implementation in the Chimera editor to its current form, the reasons for the changes, and how the histories might change in the future.

The Evolution of Editable Graphical Histories in Chimera

Figure 1a is an editor scene containing a simple drawing of four heads, generated by the steps depicted in the history shown in Figure 1b. The panels are grouped in pairs: the top panel shows the scene before the operations represented by the pair, the bottom panel shows the scene afterwards. We refer to the top panel as the prologue, and the bottom panel as the epilogue. The first panel includes those operations used to draw the outline. The second panel pair adds an eye, the third a mouth, and the fourth an ear. The fifth panel pair shows the head being selected and scaled. In the remaining panels, the head is copied, the copy is translated alongside the original, the two heads are copied, and the copies are translated above.

This history was generated in 1988, and it represents one of our first attempts at graphical histories [Kurlander 88b]. Related operations are coalesced into single panels. For example, the first panel contains several Add Line operations, as well as operations to move the cursor. This makes the history shorter, but it also makes it easier to understand because logical operations are depicted rather than physical ones. The default degree of coalescing can be adjusted according to the level of expertise of the user, or the task for which the history is used. If the component operations of a panel need to be examined in more detail, the panel pair can be split into lower-level pairs. The panels display only the portion of scene that is relevant to the operations that they represent, plus some scene context to show where the operations were performed.

As illustrations generated in Chimera became more complex, it also became more difficult to quickly differentiate between the contents of the before and after panels. We needed a method to draw one's eye to the important parts of the panel, away from the auxiliary scene context. To accomplish this, we added rendering styles to the graphical histories. During the panel composition process, objects to be included in a panel are tagged with their reason for inclusion, or equivalently, their role in the explanation. For each object in a panel, the renderer examines the object's tags to determine what style should be used to draw the object. The association between rendering styles and object tags can be easily customized. By default, contextual objects are subdued by lightening their colors, and objects that participate in the panel's operations are rendered normally. This represents the state of the histories in 1989, and is further discussed in [Kurlander 90]. (a)

(b)

This technique was employed in producing the history panels of Figure 2 to make the important objects in each panel stand out. Figure 2a shows a bar chart drawn in Chimera, and Figure 2b shows steps that added the bars to the chart, and a command that changed the stroke width of the frame. The bar created in the first panel pair is an important result of the operation, so it is rendered in its normal fashion, while the rest of the chart, which is present only to show where the bar was added, has been subdued. In this and in subsequent panels, it is clear which objects are critical to the explanation, and which provide context.

Next it became clear that we needed to abandon the two row layout, and adopt a single row of panels. There were many reasons for this decision:

• Most importantly, two rows of panels require far too much screen real estate.
• Often information in the prologue is unimportant to the user. For example, since the Set-Fill-Color operation acts on selected objects, it is not necessary to have a prologue panel to show by comparison which objects changed.
• There is a tendency for prologues to repeat information. Because there is a large amount of coherence in editing tasks, the prologue of one pair is often identical to the epilogue of the previous pair.
• Having a single row of panels results in faster drawing time.
• A single row of panels is more intuitive to read. When asked to interpret sample histories for the first time, most people try to read across the first row, then across the second. This is incorrect. (However, it has been suggested that we could turn the history on its side).
• Prologues are less important now that different rendering styles distinguish important, participatory objects from context.

(b)

Eventually, we would like the history mechanism to make a good guess about which graphical objects and widgets should be included in each type of panel, without the explicit rules that must be provided now. These would be the objects and widgets modified or referenced by the operations in the panel, and the UIMS should be able to detect this. However, the application writer needs to be able to override the default, and explicitly specify the contents for each kind of panel. To make it easier to write rules, Chimera's current rule language can refer to an object-based command class hierarchy. Since rules can refer to classes of commands, not just individual commands, application developers can write rules for large families of commands at once.

Initially the histories were editable only in the sense that the user could explicitly execute a command to undo the editor to a state represented by any panel, make changes to the editor scene, and redo the undone commands. This process often felt indirect and clumsy, so we added an option to convert the history panels into fully editable Chimera scenes. Now changes can be made directly in the panels themselves, and users can execute a command to propagate these changes into the history stream. Though we initially developed editable graphical histories with the intention of using them as a visual representation for macros by example [Kurlander 88b], we postponed implementing this until the individual panels could be made editable. Having editable panels allows us to select objects and properties directly from the panels to be used as macro arguments, and to edit the macros in place. Editable graphical histories support macros by example in many other ways as well, as is discussed in Chapter 15. Figure 3a is a postcard drawn in Chimera, and Figure 3b is the graphical history showing how the Adirondack mountains were added to the illustration. The graphical history here was recently generated by Chimera (March 1992), and reflects the changes that we made over the years. Note that panels are no longer separated into before and after rows. Also, now that the history mechanism is partially in the UIMS, it is much cleaner for the system to generate panels showing parts of the control panels, since the goal of separating applications from interfaces dictates that the editor itself not know about the appearance of these components.

(a)

(b)

Future Directions

Though picture panels can visually represent operation sequences performed in the past, they can also depict sequences that might be invoked in the future. Such futures (as opposed to histories) could be used by a system that automatically infers likely interaction paths in presenting potential steps to the user. They could be used as part of an active tutorial system for an application, to teach people how to reach probable targets. Alternatively, these futures might act as a type of "power steering" by experts to reach their goals faster.